<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Hi,<br>
<br>
You can use midPoint logging to get stats about performance of
individual midPoint components. And then you can figure out where
exactly is the problem.<br>
<br>
But I think the root of this issue is in the architecture. REST
interfaces are strictly object-oriented (or web-resource-oriented to
be precise). Each REST operation can operate only on a single web
resource. When translated to midPoint design this means (at least)
one operation for each object. Which means network latencies,
authentication (REST explicitly prohibits sessions), request
processing, authorizations, executing the operation, response
processing and latencies again. This happens for every object. The
overhead is simply too high. In other words: RESTful services are
absolutely terrible for any kind of bulk operations. And this is
more-or-less given by the principles of REST architectural style. It
is not easy to do anything about it without bending or openly
violating the REST principles.<br>
<br>
Theoretically there is a way around this: we could create a
specialized web resource for bulk operations. But it is complex,
ugly and difficult to use. And it will actually mean doing RPC and
disguising that as REST. Therefore currently we do not plan to do
this. We will do that only if there is someone explicitly sponsoring
that feature. And even then I will personally put big red stickers
all over it saying that "this may work, but it is not REST".<br>
<br>
I would suggest that the right way to do bulk operations is not to
use REST at all. It makes no sense to transport water in bottles
when you need water for entire city, does it? You should use the
right tool for the job.<br>
<br>
MidPoint has a very good built-in features that support bulk
operations. Simply use the synchronization features of midPoint.
These are designed to handle bulk data. Connect the database as
midPoint resource and pull in the data using import or
reconciliation task. If this is a one-off data load you can delete
the resource afterwards. But actually there is usually a good
benefit of keeping the resource around for a longer time in case
that the migration needs to be retried or objects need to be
updated.<br>
<br>
<pre class="moz-signature" cols="72">--
Radovan Semancik
Software Architect
evolveum.com
</pre>
<br>
<br>
<div class="moz-cite-prefix">On 07/26/2016 09:13 PM, Martin Marchese
wrote:<br>
</div>
<blockquote
cite="mid:CAG3rmdqdG3H2D0UgNE5XD_083h98TiV-1istoYG-NHJh8xKBCA@mail.gmail.com"
type="cite">
<div dir="ltr">Hi,
<div><br>
</div>
<div>We have a large database (<span class="" id=":xs.1"
tabindex="-1">aprox</span>. 400000-500000 users, most of
them linked with 2 platforms).</div>
<div><br>
</div>
<div>We are using <span class="" id=":xs.2" tabindex="-1">PostgreSQL</span>
and still loading users with some python scripts we developed
to consume data from files and execute REST services to create
and/or recompute users.</div>
<div><br>
</div>
<div>With this sizing, we are experiencing a very slow
performance in the bulk load. Is there a way to troubleshoot
this or tune the database to increase performance?</div>
<div><br>
</div>
<div>Thanks</div>
<div><br clear="all">
<div>
<div class="gmail_signature"
data-smartmail="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr"><b><span></span><span></span><span
class="" id=":xs.3" tabindex="-1">Ing</span>. <span
class="" id=":xs.4" tabindex="-1">MartÃn</span>
<span class="" id=":xs.5" tabindex="-1">Marchese</span></b><br>
<img moz-do-not-send="true"
src="http://www.identicum.com/img/favicon.ico"><span
class="" id=":xs.6" tabindex="-1">Identicum</span>
S.A.<br>
<span class="" id=":xs.7" tabindex="-1">Anchorena</span>
1357 PB<br>
Tel: +54 (11) 3526.5509<br>
<a moz-do-not-send="true"
href="mailto:mmarchese@identicum.com"
target="_blank"><span class="" id=":xs.8"
tabindex="-1">mmarchese</span>@<span class=""
id=":xs.9" tabindex="-1">identicum</span>.com</a><br>
<a moz-do-not-send="true"
href="http://www.identicum.com" target="_blank">www.<span
class="" id=":xs.10" tabindex="-1">identicum</span>.com</a></div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
midPoint mailing list
<a class="moz-txt-link-abbreviated" href="mailto:midPoint@lists.evolveum.com">midPoint@lists.evolveum.com</a>
<a class="moz-txt-link-freetext" href="http://lists.evolveum.com/mailman/listinfo/midpoint">http://lists.evolveum.com/mailman/listinfo/midpoint</a>
</pre>
</blockquote>
<br>
</body>
</html>