<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hello Arnošt,</p>
<p>this is a good observation. <br>
</p>
<p>To be honest, iterative search by paging was meant as a
workaround for databases that do not support search with
subsequent modify operations on the returned objects. But, as we
see from your message, it can be used to avoid these problems as
well :)</p>
<p>Just a slight correction: <br>
</p>
<p>
<blockquote type="cite">Midpoint in default configuration
recomputes objects by first retrieving them ALL from repository,
then passing each object to a worker thread.</blockquote>
This is not quite true. MidPoint fetches objects
one-after-another, and just after fetching each one from the
repository it passes the object to a worker thread (or processes
it directly if there are no worker threads defined). However,
because of quite strong transaction isolation setting
(serializable) the DBMS ensures that changes that occur on objects
after the transaction started (i.e. after the search was started)
are not reflected in their values. <br>
</p>
<p>I can imagine an option that would make this more optimized. E.g.
by retrieving just a list of OIDs and reading each object just
before its processing. If you have a second of free time, you
could create a jira for this.<br>
</p>
<p>Moreover, in 3.8 we loose transaction isolation a bit, from
serializable to repeatable_read. But I think this will not change
this behavior.</p>
<pre class="moz-signature" cols="72">Pavol Mederly
Software developer
evolveum.com
</pre>
<div class="moz-cite-prefix">On 29.01.2018 13:22, Arnošt Starosta -
AMI Praha a.s. wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAGPA3FJMBddCsmZH0e4u9uiuJ7P4==cCKHaF9L2CqxN4MFiKTA@mail.gmail.com">
<div dir="ltr">
<div style="color:rgb(0,0,0);font-family:"Lucida
Grande","Lucida Sans
Unicode",Arial,sans-serif;font-size:13.3333px"><b>Problem
: </b></div>
<div style="color:rgb(0,0,0);font-family:"Lucida
Grande","Lucida Sans
Unicode",Arial,sans-serif;font-size:13.3333px"><br>
</div>
<div style="color:rgb(0,0,0);font-family:"Lucida
Grande","Lucida Sans
Unicode",Arial,sans-serif;font-size:13.3333px">Midpoint
in default configuration recomputes objects by first
retrieving them ALL from repository, then passing each object
to a worker thread. If the object was updated meanwhile (e.g.
live-synced or updated from gui) before it is recomputed by
the worker thread, this update can be overwritten by the
object version retrieved when the recompute task started. It
happened on my deployment several times.</div>
<div style="color:rgb(0,0,0);font-family:"Lucida
Grande","Lucida Sans
Unicode",Arial,sans-serif;font-size:13.3333px"><br>
</div>
<div style="color:rgb(0,0,0);font-family:"Lucida
Grande","Lucida Sans
Unicode",Arial,sans-serif;font-size:13.3333px"><b
style="background-color:transparent;font-size:10pt">Is your
deployment affected? :</b></div>
<div style="color:rgb(0,0,0);font-family:"Lucida
Grande","Lucida Sans
Unicode",Arial,sans-serif;font-size:13.3333px"><br>
</div>
<div style="color:rgb(0,0,0);font-family:"Lucida
Grande","Lucida Sans
Unicode",Arial,sans-serif;font-size:13.3333px">Hard to
say, i don't see any relevant log message to check. I had to
check by debugging the running recompute task and verifying
that SqlRepositoryServiceImpl.searchObjectsIterative calls
ObjectRetriever.searchObjectsIterativeByPaging (ok) and not
ObjectRetriever.searchObjectsIterativeAttempt (can loose
updates).</div>
<div style="color:rgb(0,0,0);font-family:"Lucida
Grande","Lucida Sans
Unicode",Arial,sans-serif;font-size:13.3333px"><br>
</div>
<div style="color:rgb(0,0,0);font-family:"Lucida
Grande","Lucida Sans
Unicode",Arial,sans-serif;font-size:13.3333px">Deployments
with MySQL or H2 backend should be ok with default
configuration (check sources
SqlRepositoryConfiguration.computeDefaultIterativeSearchParameters).
Did not verify the runtime.</div>
<div style="color:rgb(0,0,0);font-family:"Lucida
Grande","Lucida Sans
Unicode",Arial,sans-serif;font-size:13.3333px"><br>
</div>
<div style="color:rgb(0,0,0);font-family:"Lucida
Grande","Lucida Sans
Unicode",Arial,sans-serif;font-size:13.3333px"><b>Solution:</b></div>
<div style="color:rgb(0,0,0);font-family:"Lucida
Grande","Lucida Sans
Unicode",Arial,sans-serif;font-size:13.3333px"><br>
</div>
<div style="color:rgb(0,0,0);font-family:"Lucida
Grande","Lucida Sans
Unicode",Arial,sans-serif;font-size:13.3333px">Configure <font
face="courier new, monospace">iterativeSearchByPaging</font> and <font
face="courier new, monospace">iterativeSearchByPagingBatchSize</font> in
config.xml <font face="courier new, monospace">midpoint/repository</font> element.
Don't know if all backends support this setting but postgres
(which i use) does.</div>
<div style="color:rgb(0,0,0);font-family:"Lucida
Grande","Lucida Sans
Unicode",Arial,sans-serif;font-size:13.3333px"><br>
</div>
<div style="color:rgb(0,0,0);font-family:"Lucida
Grande","Lucida Sans
Unicode",Arial,sans-serif;font-size:13.3333px">
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:"Courier New";color:rgb(51,51,51);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><configuration></span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:"Courier New";color:rgb(51,51,51);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> <midpoint></span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:"Courier New";color:rgb(51,51,51);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> <repository></span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:"Courier New";color:rgb(51,51,51);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> …</span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:"Courier New";color:rgb(51,51,51);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> <iterativeSearchByPaging>true</iterativeSearchByPaging></span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:"Courier New";color:rgb(51,51,51);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> <iterativeSearchByPagingBatchSize>17</iterativeSearchByPagingBatchSize></span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:"Courier New";color:rgb(51,51,51);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> …</span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:"Courier New";color:rgb(51,51,51);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> </repository></span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:"Courier New";color:rgb(51,51,51);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> </midpoint></span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:"Courier New";color:rgb(51,51,51);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"></configuration></span></p>
</div>
<div style="color:rgb(0,0,0);font-family:"Lucida
Grande","Lucida Sans
Unicode",Arial,sans-serif;font-size:13.3333px"><br>
</div>
<div style="color:rgb(0,0,0);font-family:"Lucida
Grande","Lucida Sans
Unicode",Arial,sans-serif;font-size:13.3333px">After
setting these parameters the objects to recompute are read in
'pages' and fed to worker threads until the request queue
between the reader thread and worker threads is full, then the
reader is blocked. The size of the queue is hardcoded as 2 *
number-of-worker-threads.</div>
<div style="color:rgb(0,0,0);font-family:"Lucida
Grande","Lucida Sans
Unicode",Arial,sans-serif;font-size:13.3333px"><br>
</div>
<div style="color:rgb(0,0,0);font-family:"Lucida
Grande","Lucida Sans
Unicode",Arial,sans-serif;font-size:13.3333px">By setting
the <span style="font-family:"courier
new",monospace;font-size:13.3333px">iterativeSearchByPagingBatchSize</span><span
style="font-size:13.3333px;background-color:transparent"> you
can still loose updates, but the time window when this can
happen shrinks from number-of-objects to max(page size,
2*num-of-worker-threads). Without much thought i set the
page size to (</span><span
style="font-size:13.3333px;background-color:transparent">2 *
number-of-worker-threads) + 1.</span></div>
<div><br>
</div>
<div>good luck</div>
<div>arnost</div>
<div><br>
</div>
-- <br>
<div class="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<table
style="font-family:Verdana,Arial,Helvetica,sans-serif;border-collapse:collapse;padding:0px;margin:0px;border-width:0px;border-style:solid;width:482px">
<tbody>
<tr style="padding:0px;margin:0px;border:0px solid
gray">
<td
style="color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:11px;width:160px;vertical-align:bottom;padding:0px;border:0px
solid gray">
<p><span style="font-size:14px;font-weight:bold">Arnošt
Starosta</span><br>
solution architect<br>
<br>
gsm: [+420] 603 794 932<br>
e-mail: <a
href="mailto:arnost.starosta@ami.cz"
target="_blank" moz-do-not-send="true">arnost.starosta@ami.cz</a></p>
</td>
<td
style="color:rgb(0,0,0);font-family:Verdana,Arial,Helvetica,sans-serif;font-size:10px;border-width:0px
1px 0px 0px;border-style:solid;border-color:gray
rgb(204,204,204) gray gray;padding:0px"> </td>
<td
style="color:rgb(0,0,0);font-family:Verdana,Arial,Helvetica,sans-serif;font-size:10px;padding:0px;border:0px
solid gray"> </td>
<td
style="color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:11px;vertical-align:bottom;padding:0px;width:123px;border:0px
solid gray">
<p>AMI Praha a.s.<br>
Pláničkova 11<br>
162 00 Praha 6<br>
tel.: [+420] 274 783 239<br>
web: <a href="http://www.ami.cz/"
target="_blank" moz-do-not-send="true">www.ami.cz</a></p>
</td>
<td
style="color:rgb(0,0,0);font-family:Verdana,Arial,Helvetica,sans-serif;font-size:10px;border-width:0px
1px 0px 0px;border-style:solid;border-color:gray
rgb(204,204,204) gray gray;padding:0px"> </td>
<td
style="color:rgb(0,0,0);font-family:Verdana,Arial,Helvetica,sans-serif;font-size:10px;padding:0px;border:0px
solid gray"> </td>
<td
style="color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:11px;margin:8px;width:116px;border:0px
solid gray">
<p><img
src="http://www.ami.cz/images/podpis/ami_logo.gif"
alt="AMI Praha a.s." style="border: 0px;"
moz-do-not-send="true"></p>
</td>
</tr>
<tr style="padding:0px;margin:0px;border:0px solid
gray">
<td colspan="7"
style="color:rgb(0,0,0);font-family:Verdana,Arial,Helvetica,sans-serif;font-size:10px;padding:0px;width:480px;border:0px
solid gray"><br>
<a
href="http://www.ami.cz/reseni-a-sluzby/bezpecnost-dat/identity-management"
target="_blank" moz-do-not-send="true"><img
src="http://www.ami.cz/images/podpis/AMI-podpis-IdM_1.png"
alt="AMI Praha a.s." style="border: 0px;
width: 480px; height: 82px;"
moz-do-not-send="true"></a></td>
</tr>
<tr style="padding:0px;margin:0px;border:0px solid
gray">
<td colspan="7"
style="color:rgb(128,128,128);font-family:Arial,sans-serif;font-size:11px;padding:0px;border:0px
solid gray"><br>
Textem tohoto e-mailu podepisující neslibuje
uzavřít ani neuzavírá za společnost AMI Praha
a.s.<br>
jakoukoliv smlouvu. Každá smlouva, pokud bude
uzavřena, musí mít výhradně písemnou formu.<br>
<br>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
midPoint mailing list
<a class="moz-txt-link-abbreviated" href="mailto:midPoint@lists.evolveum.com">midPoint@lists.evolveum.com</a>
<a class="moz-txt-link-freetext" href="http://lists.evolveum.com/mailman/listinfo/midpoint">http://lists.evolveum.com/mailman/listinfo/midpoint</a>
</pre>
</blockquote>
<br>
</body>
</html>