<div dir="ltr">That might be the root of the problem!<div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF"><p>The <tt>searchObjectsIterative</tt> method starts a search
operation, and each object - as soon as it's returned from the
repository - is handled by the <tt>resultHandler</tt>. (Using
ScrollableResults - as can be seen <a href="https://github.com/Evolveum/midpoint/blob/42a1a66e93347d8c8b30624a574e7dfaf3743e88/repo/repo-sql-impl/src/main/java/com/evolveum/midpoint/repo/sql/helpers/ObjectRetriever.java#L680" target="_blank">here</a>.)<br>
</p>
<p></p></div></blockquote><div>where 'here' is </div><div><br></div><div>ScrollableResults results = rQuery.scroll(ScrollMode.FORWARD_ONLY);<br></div><div><br></div><div>The jdbc spec says about TYPE_FORWARD_ONLY</div><div><br></div><div>"The rows contained in the result set depend on how the underlying database
materializes the results. That is, it contains the rows that satisfy the query at
either the time the query is executed or as the rows are retrieved"</div><div><br></div><div>If we wanted to see the changes, we would have to use TYPE_SCROLL_SENSITIVE.</div><div><br></div><div>What I don't understand is how this plays together with transaction isolation settings. Does specifying ResultSet type override them or is it the other way around? No time to read the whole spec :/</div><div><br></div><div>arnost</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF"><p>I do not know how this works internally in hibernate, JDBC driver
and DBMS itself. But I suppose that if there's any
caching/chunking/prefetching there, it does not gather all objects
before processing them.</p>
<p>Anyway, I think we can implement the OID processing. (But it's
not me who decides about the budgets :))<br>
</p><span class="gmail-">
<pre class="gmail-m_-2852671089402895732moz-signature" cols="72">Pavol Mederly
Software developer
<a href="http://evolveum.com" target="_blank">evolveum.com</a>
</pre>
</span><div><div class="gmail-h5"><div class="gmail-m_-2852671089402895732moz-cite-prefix">On 07.02.2018 13:45, Arnošt Starosta -
AMI Praha a.s. wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi Pavol,
<div><br>
</div>
<div>that unintended workaround saved my life for the moment .)<br>
<div><br>
</div>
<div>Not sure if "fetches objects one-after-another" makes the
picture clear. As i understand it the default reading
workflow goes in a single query - all objects with full
details in a single query/result set that is processed one
by one by the handlers. Don't know how fetching rows from
the result set works.</div>
<div><br>
</div>
<div>Tweaking the transaction isolation did not really help,
even with default set to 'read committed'. Thats why i think
the object 'fetching' happens in larger chunks and may not
be affected by weaker transaction isolation. Or maybe i just
misconfigured.</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">Working with oids in iterative tasks
would be great! You want the worker threads to process 'that
object' not 'this chunk of data'.</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">The jira is already there - <a href="https://jira.evolveum.com/browse/MID-4414" target="_blank">https://jira.evolveum.com/<wbr>browse/MID-4414</a></div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">arnost</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">2018-02-07 12:17 GMT+01:00 Pavol
Mederly <span dir="ltr"><<a href="mailto:mederly@evolveum.com" target="_blank">mederly@evolveum.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hello Arnošt,</p>
<p>this is a good observation. <br>
</p>
<p>To be honest, iterative search by paging was meant
as a workaround for databases that do not support
search with subsequent modify operations on the
returned objects. But, as we see from your message,
it can be used to avoid these problems as well :)</p>
<p>Just a slight correction: <br>
</p>
<p><span class="gmail-m_-2852671089402895732gmail-"> </span></p>
<blockquote type="cite">Midpoint in default
configuration recomputes objects by first retrieving
them ALL from repository, then passing each object
to a worker thread.</blockquote>
This is not quite true. MidPoint fetches objects
one-after-another, and just after fetching each one
from the repository it passes the object to a worker
thread (or processes it directly if there are no
worker threads defined). However, because of quite
strong transaction isolation setting (serializable)
the DBMS ensures that changes that occur on objects
after the transaction started (i.e. after the search
was started) are not reflected in their values. <br>
<p>I can imagine an option that would make this more
optimized. E.g. by retrieving just a list of OIDs
and reading each object just before its processing.
If you have a second of free time, you could create
a jira for this.<br>
</p>
<p>Moreover, in 3.8 we loose transaction isolation a
bit, from serializable to repeatable_read. But I
think this will not change this behavior.</p>
<pre class="gmail-m_-2852671089402895732gmail-m_-4624831061758144586moz-signature" cols="72">Pavol Mederly
Software developer
<a href="http://evolveum.com" target="_blank">evolveum.com</a>
</pre>
<div>
<div class="gmail-m_-2852671089402895732gmail-h5">
<div class="gmail-m_-2852671089402895732gmail-m_-4624831061758144586moz-cite-prefix">On
29.01.2018 13:22, Arnošt Starosta - AMI Praha
a.s. wrote:<br>
</div>
</div>
</div>
<blockquote type="cite">
<div>
<div class="gmail-m_-2852671089402895732gmail-h5">
<div dir="ltr">
<div><b>Problem : </b></div>
<div><br>
</div>
<div>Midpoint in default configuration
recomputes objects by first retrieving them
ALL from repository, then passing each
object to a worker thread. If the object was
updated meanwhile (e.g. live-synced or
updated from gui) before it is recomputed by
the worker thread, this update can be
overwritten by the object version retrieved
when the recompute task started. It happened
on my deployment several times.</div>
<div><br>
</div>
<div><b style="background-color:transparent;font-size:10pt">Is
your deployment affected? :</b></div>
<div><br>
</div>
<div>Hard to say, i don't see any relevant log
message to check. I had to check by
debugging the running recompute task and
verifying that SqlRepositoryServiceImpl.searc<wbr>hObjectsIterative
calls ObjectRetriever.searchObjectsI<wbr>terativeByPaging
(ok) and not ObjectRetriever.searchObjectsI<wbr>terativeAttempt
(can loose updates).</div>
<div><br>
</div>
<div>Deployments with MySQL or H2 backend
should be ok with default configuration
(check sources SqlRepositoryConfiguration.com<wbr>puteDefaultIterativeSearchPara<wbr>meters).
Did not verify the runtime.</div>
<div><br>
</div>
<div><b>Solution:</b></div>
<div><br>
</div>
<div>Configure <font face="courier new,
monospace">iterativeSearchByPag<wbr>ing</font> and <font face="courier new, monospace">iterativeSearchByPagin<wbr>gBatchSize</font> in
config.xml <font face="courier new,
monospace">midpoint/repository</font><wbr> element.
Don't know if all backends support this
setting but postgres (which i use) does.</div>
<div><br>
</div>
<div>
<p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:"Courier New";color:rgb(51,51,51);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><configuration></span></p>
<p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:"Courier New";color:rgb(51,51,51);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> <midpoint></span></p>
<p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:"Courier New";color:rgb(51,51,51);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> <repository></span></p>
<p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:"Courier New";color:rgb(51,51,51);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> …</span></p>
<p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:"Courier New";color:rgb(51,51,51);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> <iterativeSearchByP<wbr>aging>true</iterativeSearchByP<wbr>aging></span></p>
<p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:"Courier New";color:rgb(51,51,51);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> <iterativeSearchByPagingBatchS<wbr>ize>17</iterativeSearchByPagin<wbr>gBatchSize></span></p>
<p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:"Courier New";color:rgb(51,51,51);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> …</span></p>
<p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:"Courier New";color:rgb(51,51,51);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> </repository></span></p>
<p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:"Courier New";color:rgb(51,51,51);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> </midpoint></span></p>
<p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:"Courier New";color:rgb(51,51,51);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"></configuration></span></p>
</div>
<div><br>
</div>
<div>After setting these parameters the
objects to recompute are read in 'pages' and
fed to worker threads until the request
queue between the reader thread and worker
threads is full, then the reader is blocked.
The size of the queue is hardcoded as 2 *
number-of-worker-threads.</div>
<div><br>
</div>
<div>By setting the <span>iterativeSearchByPagingBat<wbr>chSize</span><span style="font-size:13.3333px;background-color:transparent"> you can still
loose updates, but the time window when
this can happen shrinks from
number-of-objects to max(page size,
2*num-of-worker-threads). Without much
thought i set the page size to (</span><span style="font-size:13.3333px;background-color:transparent">2 *
number-of-worker-threads) + 1.</span></div>
<div><br>
</div>
<div>good luck</div>
<div>arnost</div>
<div><br>
</div>
-- <br>
<div class="gmail-m_-2852671089402895732gmail-m_-4624831061758144586gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<table style="font-family:Verdana,Arial,Helvetica,sans-serif;border-collapse:collapse;padding:0px;margin:0px;border-width:0px;border-style:solid;width:482px">
<tbody>
<tr style="padding:0px;margin:0px;border:0px solid gray">
<td style="color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:11px;width:160px;vertical-align:bottom;padding:0px;border:0px solid gray">
<p><span style="font-size:14px;font-weight:bold">Arnošt
Starosta</span><br>
solution architect<br>
<br>
gsm: <a href="tel:+420%20603%20794%20932" value="+420603794932" target="_blank">[+420]
603 794 932</a><br>
e-mail: <a href="mailto:arnost.starosta@ami.cz" target="_blank">arnost.starosta@ami.cz</a></p>
</td>
<td style="color:rgb(0,0,0);font-family:Verdana,Arial,Helvetica,sans-serif;font-size:10px;border-width:0px 1px 0px 0px;border-style:solid;border-color:gray rgb(204,204,204) gray gray;padding:0px"> </td>
<td style="color:rgb(0,0,0);font-family:Verdana,Arial,Helvetica,sans-serif;font-size:10px;padding:0px;border:0px solid gray"> </td>
<td style="color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:11px;vertical-align:bottom;padding:0px;width:123px;border:0px solid gray">
<p>AMI Praha a.s.<br>
Pláničkova 11<br>
162 00 Praha 6<br>
tel.: <a href="tel:+420%20274%20783%20239" value="+420274783239" target="_blank">[+420]
274 783 239</a><br>
web: <a href="http://www.ami.cz/" target="_blank">www.ami.cz</a></p>
</td>
<td style="color:rgb(0,0,0);font-family:Verdana,Arial,Helvetica,sans-serif;font-size:10px;border-width:0px 1px 0px 0px;border-style:solid;border-color:gray rgb(204,204,204) gray gray;padding:0px"> </td>
<td style="color:rgb(0,0,0);font-family:Verdana,Arial,Helvetica,sans-serif;font-size:10px;padding:0px;border:0px solid gray"> </td>
<td style="color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:11px;margin:8px;width:116px;border:0px solid gray">
<p><img src="http://www.ami.cz/images/podpis/ami_logo.gif" alt="AMI Praha a.s." style="border: 0px;"></p>
</td>
</tr>
<tr style="padding:0px;margin:0px;border:0px solid gray">
<td colspan="7" style="color:rgb(0,0,0);font-family:Verdana,Arial,Helvetica,sans-serif;font-size:10px;padding:0px;width:480px;border:0px solid gray"><br>
<a href="http://www.ami.cz/reseni-a-sluzby/bezpecnost-dat/identity-management" target="_blank"><img src="http://www.ami.cz/images/podpis/AMI-podpis-IdM_1.png" alt="AMI
Praha a.s." style="border: 0px; width: 480px; height: 82px;"></a></td>
</tr>
<tr style="padding:0px;margin:0px;border:0px solid gray">
<td colspan="7" style="color:rgb(128,128,128);font-family:Arial,sans-serif;font-size:11px;padding:0px;border:0px solid gray"><br>
Textem tohoto e-mailu
podepisující neslibuje uzavřít
ani neuzavírá za společnost
AMI Praha a.s.<br>
jakoukoliv smlouvu. Každá
smlouva, pokud bude uzavřena,
musí mít výhradně písemnou
formu.<br>
<br>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset class="gmail-m_-2852671089402895732gmail-m_-4624831061758144586mimeAttachmentHeader"></fieldset>
<br>
</div>
</div>
<pre>______________________________<wbr>_________________
midPoint mailing list
<a class="gmail-m_-2852671089402895732gmail-m_-4624831061758144586moz-txt-link-abbreviated" href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>
<a class="gmail-m_-2852671089402895732gmail-m_-4624831061758144586moz-txt-link-freetext" href="http://lists.evolveum.com/mailman/listinfo/midpoint" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a>
</pre>
</blockquote>
<br>
</div>
<br>
______________________________<wbr>_________________<br>
midPoint mailing list<br>
<a href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a><br>
<a href="http://lists.evolveum.com/mailman/listinfo/midpoint" rel="noreferrer" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a><br>
<br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div class="gmail-m_-2852671089402895732gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<table style="font-family:Verdana,Arial,Helvetica,sans-serif;border-collapse:collapse;padding:0px;margin:0px;border-width:0px;border-style:solid;width:482px">
<tbody>
<tr style="padding:0px;margin:0px;border:0px solid gray">
<td style="color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:11px;width:160px;vertical-align:bottom;padding:0px;border:0px solid gray">
<p><span style="font-size:14px;font-weight:bold">Arnošt
Starosta</span><br>
solution architect<br>
<br>
gsm: <a href="tel:+420%20603%20794%20932" value="+420603794932" target="_blank">[+420] 603 794 932</a><br>
e-mail: <a href="mailto:arnost.starosta@ami.cz" target="_blank">arnost.starosta@ami.cz</a></p>
</td>
<td style="color:rgb(0,0,0);font-family:Verdana,Arial,Helvetica,sans-serif;font-size:10px;border-width:0px 1px 0px 0px;border-style:solid;border-color:gray rgb(204,204,204) gray gray;padding:0px"> </td>
<td style="color:rgb(0,0,0);font-family:Verdana,Arial,Helvetica,sans-serif;font-size:10px;padding:0px;border:0px solid gray"> </td>
<td style="color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:11px;vertical-align:bottom;padding:0px;width:123px;border:0px solid gray">
<p>AMI Praha a.s.<br>
Pláničkova 11<br>
162 00 Praha 6<br>
tel.: <a href="tel:+420%20274%20783%20239" value="+420274783239" target="_blank">[+420] 274 783 239</a><br>
web: <a href="http://www.ami.cz/" target="_blank">www.ami.cz</a></p>
</td>
<td style="color:rgb(0,0,0);font-family:Verdana,Arial,Helvetica,sans-serif;font-size:10px;border-width:0px 1px 0px 0px;border-style:solid;border-color:gray rgb(204,204,204) gray gray;padding:0px"> </td>
<td style="color:rgb(0,0,0);font-family:Verdana,Arial,Helvetica,sans-serif;font-size:10px;padding:0px;border:0px solid gray"> </td>
<td style="color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:11px;margin:8px;width:116px;border:0px solid gray">
<p><img src="http://www.ami.cz/images/podpis/ami_logo.gif" alt="AMI Praha a.s." style="border: 0px;"></p>
</td>
</tr>
<tr style="padding:0px;margin:0px;border:0px solid gray">
<td colspan="7" style="color:rgb(0,0,0);font-family:Verdana,Arial,Helvetica,sans-serif;font-size:10px;padding:0px;width:480px;border:0px solid gray"><br>
<a href="http://www.ami.cz/reseni-a-sluzby/bezpecnost-dat/identity-management" target="_blank"><img src="http://www.ami.cz/images/podpis/AMI-podpis-IdM_1.png" alt="AMI
Praha a.s." style="border: 0px; width: 480px; height: 82px;"></a></td>
</tr>
<tr style="padding:0px;margin:0px;border:0px solid gray">
<td colspan="7" style="color:rgb(128,128,128);font-family:Arial,sans-serif;font-size:11px;padding:0px;border:0px solid gray"><br>
Textem tohoto e-mailu podepisující neslibuje
uzavřít ani neuzavírá za společnost AMI
Praha a.s.<br>
jakoukoliv smlouvu. Každá smlouva, pokud
bude uzavřena, musí mít výhradně písemnou
formu.<br>
<br>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset class="gmail-m_-2852671089402895732mimeAttachmentHeader"></fieldset>
<br>
<pre>______________________________<wbr>_________________
midPoint mailing list
<a class="gmail-m_-2852671089402895732moz-txt-link-abbreviated" href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>
<a class="gmail-m_-2852671089402895732moz-txt-link-freetext" href="http://lists.evolveum.com/mailman/listinfo/midpoint" target="_blank">http://lists.evolveum.com/<wbr>mailman/listinfo/midpoint</a>
</pre>
</blockquote>
<br>
</div></div></div>
<br>______________________________<wbr>_________________<br>
midPoint mailing list<br>
<a href="mailto:midPoint@lists.evolveum.com">midPoint@lists.evolveum.com</a><br>
<a href="http://lists.evolveum.com/mailman/listinfo/midpoint" rel="noreferrer" target="_blank">http://lists.evolveum.com/<wbr>mailman/listinfo/midpoint</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><p style="color:rgb(0,0,0);font-family:Verdana,Arial,Helvetica,sans-serif;font-size:10.3846px"></p><table style="font-family:Verdana,Arial,Helvetica,sans-serif;border-collapse:collapse;padding:0px;margin:0px;border-width:0px;border-style:solid;width:482px"><tbody><tr style="padding:0px;margin:0px;border:0px solid gray"><td style="color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:11px;width:160px;vertical-align:bottom;padding:0px;border:0px solid gray"><p><span style="font-size:14px;font-weight:bold">Arnošt Starosta</span><br>solution architect<br><br>gsm: [+420] 603 794 932<br>e-mail: <a href="mailto:arnost.starosta@ami.cz" target="_blank">arnost.starosta@ami.cz</a></p></td><td style="color:rgb(0,0,0);font-family:Verdana,Arial,Helvetica,sans-serif;font-size:10px;border-width:0px 1px 0px 0px;border-style:solid;border-color:gray rgb(204,204,204) gray gray;padding:0px"> </td><td style="color:rgb(0,0,0);font-family:Verdana,Arial,Helvetica,sans-serif;font-size:10px;padding:0px;border:0px solid gray"> </td><td style="color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:11px;vertical-align:bottom;padding:0px;width:123px;border:0px solid gray"><p>AMI Praha a.s.<br>Pláničkova 11<br>162 00 Praha 6<br>tel.: [+420] 274 783 239<br>web: <a href="http://www.ami.cz/" target="_blank">www.ami.cz</a></p></td><td style="color:rgb(0,0,0);font-family:Verdana,Arial,Helvetica,sans-serif;font-size:10px;border-width:0px 1px 0px 0px;border-style:solid;border-color:gray rgb(204,204,204) gray gray;padding:0px"> </td><td style="color:rgb(0,0,0);font-family:Verdana,Arial,Helvetica,sans-serif;font-size:10px;padding:0px;border:0px solid gray"> </td><td style="color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:11px;margin:8px;width:116px;border:0px solid gray"><p><img src="http://www.ami.cz/images/podpis/ami_logo.gif" alt="AMI Praha a.s." style="border: 0px;"></p></td></tr><tr style="padding:0px;margin:0px;border:0px solid gray"><td colspan="7" style="color:rgb(0,0,0);font-family:Verdana,Arial,Helvetica,sans-serif;font-size:10px;padding:0px;width:480px;border:0px solid gray"><br><a href="http://www.ami.cz/reseni-a-sluzby/bezpecnost-dat/identity-management" target="_blank"><img src="http://www.ami.cz/images/podpis/AMI-podpis-IdM_1.png" alt="AMI Praha a.s." style="border: 0px; width: 480px; height: 82px;"></a></td></tr><tr style="padding:0px;margin:0px;border:0px solid gray"><td colspan="7" style="color:rgb(128,128,128);font-family:Arial,sans-serif;font-size:11px;padding:0px;border:0px solid gray"><br>Textem tohoto e-mailu podepisující neslibuje uzavřít ani neuzavírá za společnost AMI Praha a.s.<br>jakoukoliv smlouvu. Každá smlouva, pokud bude uzavřena, musí mít výhradně písemnou formu.<br><br></td></tr></tbody></table></div></div></div></div>
</div></div>