<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>
<blockquote type="cite">
<div>Not sure if "fetches objects one-after-another" makes the
picture clear. As i understand it the default reading workflow
goes in a single query - all objects with full details in a
single query/result set that is processed one by one by the
handlers. Don't know how fetching rows from the result set
works.</div>
</blockquote>
It is quite easy to explain. Please look <a
moz-do-not-send="true"
href="https://github.com/Evolveum/midpoint/blob/54112a0ad266f8cd3f3024111a195fb064f79ae6/repo/repo-common/src/main/java/com/evolveum/midpoint/repo/common/task/AbstractSearchIterativeTaskHandler.java#L289">here</a>:</p>
<p><tt>repositoryService.<b>searchObjectsIterative</b>((Class<O>)
type, query, resultHandler, searchOptions, false, opResult);</tt></p>
<p>The <tt>searchObjectsIterative</tt> method starts a search
operation, and each object - as soon as it's returned from the
repository - is handled by the <tt>resultHandler</tt>. (Using
ScrollableResults - as can be seen <a moz-do-not-send="true"
href="https://github.com/Evolveum/midpoint/blob/42a1a66e93347d8c8b30624a574e7dfaf3743e88/repo/repo-sql-impl/src/main/java/com/evolveum/midpoint/repo/sql/helpers/ObjectRetriever.java#L680">here</a>.)<br>
</p>
<p>I do not know how this works internally in hibernate, JDBC driver
and DBMS itself. But I suppose that if there's any
caching/chunking/prefetching there, it does not gather all objects
before processing them.</p>
<p>Anyway, I think we can implement the OID processing. (But it's
not me who decides about the budgets :))<br>
</p>
<pre class="moz-signature" cols="72">Pavol Mederly
Software developer
evolveum.com
</pre>
<div class="moz-cite-prefix">On 07.02.2018 13:45, Arnošt Starosta -
AMI Praha a.s. wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAGPA3FL3oZsBnGz1DWv9Hi=V11TP_895hOODv6up5ZyLmNXWXQ@mail.gmail.com">
<div dir="ltr">Hi Pavol,
<div><br>
</div>
<div>that unintended workaround saved my life for the moment .)<br>
<div><br>
</div>
<div>Not sure if "fetches objects one-after-another" makes the
picture clear. As i understand it the default reading
workflow goes in a single query - all objects with full
details in a single query/result set that is processed one
by one by the handlers. Don't know how fetching rows from
the result set works.</div>
<div><br>
</div>
<div>Tweaking the transaction isolation did not really help,
even with default set to 'read committed'. Thats why i think
the object 'fetching' happens in larger chunks and may not
be affected by weaker transaction isolation. Or maybe i just
misconfigured.</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">Working with oids in iterative tasks
would be great! You want the worker threads to process 'that
object' not 'this chunk of data'.</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">The jira is already there - <a
href="https://jira.evolveum.com/browse/MID-4414"
moz-do-not-send="true">https://jira.evolveum.com/browse/MID-4414</a></div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">arnost</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">2018-02-07 12:17 GMT+01:00 Pavol
Mederly <span dir="ltr"><<a
href="mailto:mederly@evolveum.com" target="_blank"
moz-do-not-send="true">mederly@evolveum.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hello Arnošt,</p>
<p>this is a good observation. <br>
</p>
<p>To be honest, iterative search by paging was meant
as a workaround for databases that do not support
search with subsequent modify operations on the
returned objects. But, as we see from your message,
it can be used to avoid these problems as well :)</p>
<p>Just a slight correction: <br>
</p>
<p><span class="gmail-"> </span></p>
<blockquote type="cite">Midpoint in default
configuration recomputes objects by first retrieving
them ALL from repository, then passing each object
to a worker thread.</blockquote>
This is not quite true. MidPoint fetches objects
one-after-another, and just after fetching each one
from the repository it passes the object to a worker
thread (or processes it directly if there are no
worker threads defined). However, because of quite
strong transaction isolation setting (serializable)
the DBMS ensures that changes that occur on objects
after the transaction started (i.e. after the search
was started) are not reflected in their values. <br>
<p>I can imagine an option that would make this more
optimized. E.g. by retrieving just a list of OIDs
and reading each object just before its processing.
If you have a second of free time, you could create
a jira for this.<br>
</p>
<p>Moreover, in 3.8 we loose transaction isolation a
bit, from serializable to repeatable_read. But I
think this will not change this behavior.</p>
<pre class="gmail-m_-4624831061758144586moz-signature" cols="72">Pavol Mederly
Software developer
<a href="http://evolveum.com" target="_blank" moz-do-not-send="true">evolveum.com</a>
</pre>
<div>
<div class="gmail-h5">
<div
class="gmail-m_-4624831061758144586moz-cite-prefix">On
29.01.2018 13:22, Arnošt Starosta - AMI Praha
a.s. wrote:<br>
</div>
</div>
</div>
<blockquote type="cite">
<div>
<div class="gmail-h5">
<div dir="ltr">
<div><b>Problem : </b></div>
<div><br>
</div>
<div>Midpoint in default configuration
recomputes objects by first retrieving them
ALL from repository, then passing each
object to a worker thread. If the object was
updated meanwhile (e.g. live-synced or
updated from gui) before it is recomputed by
the worker thread, this update can be
overwritten by the object version retrieved
when the recompute task started. It happened
on my deployment several times.</div>
<div><br>
</div>
<div><b
style="background-color:transparent;font-size:10pt">Is
your deployment affected? :</b></div>
<div><br>
</div>
<div>Hard to say, i don't see any relevant log
message to check. I had to check by
debugging the running recompute task and
verifying that SqlRepositoryServiceImpl.<wbr>searchObjectsIterative
calls ObjectRetriever.<wbr>searchObjectsIterativeByPaging
(ok) and not ObjectRetriever.<wbr>searchObjectsIterativeAttempt
(can loose updates).</div>
<div><br>
</div>
<div>Deployments with MySQL or H2 backend
should be ok with default configuration
(check sources SqlRepositoryConfiguration.<wbr>computeDefaultIterativeSearchP<wbr>arameters).
Did not verify the runtime.</div>
<div><br>
</div>
<div><b>Solution:</b></div>
<div><br>
</div>
<div>Configure <font face="courier new,
monospace">iterativeSearchByPag<wbr>ing</font> and <font
face="courier new, monospace">iterativeSearchByPagin<wbr>gBatchSize</font> in
config.xml <font face="courier new,
monospace">midpoint/repository</font><wbr> element.
Don't know if all backends support this
setting but postgres (which i use) does.</div>
<div><br>
</div>
<div>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:"Courier New";color:rgb(51,51,51);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><configuration></span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:"Courier New";color:rgb(51,51,51);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> <midpoint></span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:"Courier New";color:rgb(51,51,51);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> <repository></span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:"Courier New";color:rgb(51,51,51);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> …</span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:"Courier New";color:rgb(51,51,51);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> <<wbr>iterativeSearchByPaging>true</<wbr>iterativeSearchByPaging></span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:"Courier New";color:rgb(51,51,51);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> <<wbr>iterativeSearchByPagingBatchSi<wbr>ze>17</<wbr>iterativeSearchByPagingBatchSi<wbr>ze></span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:"Courier New";color:rgb(51,51,51);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> …</span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:"Courier New";color:rgb(51,51,51);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> </repository></span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:"Courier New";color:rgb(51,51,51);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> </midpoint></span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10pt;font-family:"Courier New";color:rgb(51,51,51);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"></configuration></span></p>
</div>
<div><br>
</div>
<div>After setting these parameters the
objects to recompute are read in 'pages' and
fed to worker threads until the request
queue between the reader thread and worker
threads is full, then the reader is blocked.
The size of the queue is hardcoded as 2 *
number-of-worker-threads.</div>
<div><br>
</div>
<div>By setting the <span>iterativeSearchByPagingBat<wbr>chSize</span><span
style="font-size:13.3333px;background-color:transparent"> you can still
loose updates, but the time window when
this can happen shrinks from
number-of-objects to max(page size,
2*num-of-worker-threads). Without much
thought i set the page size to (</span><span
style="font-size:13.3333px;background-color:transparent">2 *
number-of-worker-threads) + 1.</span></div>
<div><br>
</div>
<div>good luck</div>
<div>arnost</div>
<div><br>
</div>
-- <br>
<div
class="gmail-m_-4624831061758144586gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<table
style="font-family:Verdana,Arial,Helvetica,sans-serif;border-collapse:collapse;padding:0px;margin:0px;border-width:0px;border-style:solid;width:482px">
<tbody>
<tr
style="padding:0px;margin:0px;border:0px
solid gray">
<td
style="color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:11px;width:160px;vertical-align:bottom;padding:0px;border:0px
solid gray">
<p><span
style="font-size:14px;font-weight:bold">Arnošt
Starosta</span><br>
solution architect<br>
<br>
gsm: <a
href="tel:+420%20603%20794%20932"
value="+420603794932"
target="_blank"
moz-do-not-send="true">[+420]
603 794 932</a><br>
e-mail: <a
href="mailto:arnost.starosta@ami.cz"
target="_blank"
moz-do-not-send="true">arnost.starosta@ami.cz</a></p>
</td>
<td
style="color:rgb(0,0,0);font-family:Verdana,Arial,Helvetica,sans-serif;font-size:10px;border-width:0px
1px 0px
0px;border-style:solid;border-color:gray
rgb(204,204,204) gray
gray;padding:0px"> </td>
<td
style="color:rgb(0,0,0);font-family:Verdana,Arial,Helvetica,sans-serif;font-size:10px;padding:0px;border:0px
solid gray"> </td>
<td
style="color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:11px;vertical-align:bottom;padding:0px;width:123px;border:0px
solid gray">
<p>AMI Praha a.s.<br>
Pláničkova 11<br>
162 00 Praha 6<br>
tel.: <a
href="tel:+420%20274%20783%20239"
value="+420274783239"
target="_blank"
moz-do-not-send="true">[+420]
274 783 239</a><br>
web: <a
href="http://www.ami.cz/"
target="_blank"
moz-do-not-send="true">www.ami.cz</a></p>
</td>
<td
style="color:rgb(0,0,0);font-family:Verdana,Arial,Helvetica,sans-serif;font-size:10px;border-width:0px
1px 0px
0px;border-style:solid;border-color:gray
rgb(204,204,204) gray
gray;padding:0px"> </td>
<td
style="color:rgb(0,0,0);font-family:Verdana,Arial,Helvetica,sans-serif;font-size:10px;padding:0px;border:0px
solid gray"> </td>
<td
style="color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:11px;margin:8px;width:116px;border:0px
solid gray">
<p><img
src="http://www.ami.cz/images/podpis/ami_logo.gif"
alt="AMI Praha a.s."
style="border: 0px;"
moz-do-not-send="true"></p>
</td>
</tr>
<tr
style="padding:0px;margin:0px;border:0px
solid gray">
<td colspan="7"
style="color:rgb(0,0,0);font-family:Verdana,Arial,Helvetica,sans-serif;font-size:10px;padding:0px;width:480px;border:0px
solid gray"><br>
<a
href="http://www.ami.cz/reseni-a-sluzby/bezpecnost-dat/identity-management"
target="_blank"
moz-do-not-send="true"><img
src="http://www.ami.cz/images/podpis/AMI-podpis-IdM_1.png" alt="AMI
Praha a.s." style="border:
0px; width: 480px; height:
82px;"
moz-do-not-send="true"></a></td>
</tr>
<tr
style="padding:0px;margin:0px;border:0px
solid gray">
<td colspan="7"
style="color:rgb(128,128,128);font-family:Arial,sans-serif;font-size:11px;padding:0px;border:0px
solid gray"><br>
Textem tohoto e-mailu
podepisující neslibuje uzavřít
ani neuzavírá za společnost
AMI Praha a.s.<br>
jakoukoliv smlouvu. Každá
smlouva, pokud bude uzavřena,
musí mít výhradně písemnou
formu.<br>
<br>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset
class="gmail-m_-4624831061758144586mimeAttachmentHeader"></fieldset>
<br>
</div>
</div>
<pre>______________________________<wbr>_________________
midPoint mailing list
<a class="gmail-m_-4624831061758144586moz-txt-link-abbreviated" href="mailto:midPoint@lists.evolveum.com" target="_blank" moz-do-not-send="true">midPoint@lists.evolveum.com</a>
<a class="gmail-m_-4624831061758144586moz-txt-link-freetext" href="http://lists.evolveum.com/mailman/listinfo/midpoint" target="_blank" moz-do-not-send="true">http://lists.evolveum.com/<wbr>mailman/listinfo/midpoint</a>
</pre>
</blockquote>
<br>
</div>
<br>
______________________________<wbr>_________________<br>
midPoint mailing list<br>
<a href="mailto:midPoint@lists.evolveum.com"
moz-do-not-send="true">midPoint@lists.evolveum.com</a><br>
<a
href="http://lists.evolveum.com/mailman/listinfo/midpoint"
rel="noreferrer" target="_blank"
moz-do-not-send="true">http://lists.evolveum.com/<wbr>mailman/listinfo/midpoint</a><br>
<br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div class="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<table
style="font-family:Verdana,Arial,Helvetica,sans-serif;border-collapse:collapse;padding:0px;margin:0px;border-width:0px;border-style:solid;width:482px">
<tbody>
<tr style="padding:0px;margin:0px;border:0px
solid gray">
<td
style="color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:11px;width:160px;vertical-align:bottom;padding:0px;border:0px
solid gray">
<p><span
style="font-size:14px;font-weight:bold">Arnošt
Starosta</span><br>
solution architect<br>
<br>
gsm: [+420] 603 794 932<br>
e-mail: <a
href="mailto:arnost.starosta@ami.cz"
target="_blank" moz-do-not-send="true">arnost.starosta@ami.cz</a></p>
</td>
<td
style="color:rgb(0,0,0);font-family:Verdana,Arial,Helvetica,sans-serif;font-size:10px;border-width:0px
1px 0px
0px;border-style:solid;border-color:gray
rgb(204,204,204) gray gray;padding:0px"> </td>
<td
style="color:rgb(0,0,0);font-family:Verdana,Arial,Helvetica,sans-serif;font-size:10px;padding:0px;border:0px
solid gray"> </td>
<td
style="color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:11px;vertical-align:bottom;padding:0px;width:123px;border:0px
solid gray">
<p>AMI Praha a.s.<br>
Pláničkova 11<br>
162 00 Praha 6<br>
tel.: [+420] 274 783 239<br>
web: <a href="http://www.ami.cz/"
target="_blank" moz-do-not-send="true">www.ami.cz</a></p>
</td>
<td
style="color:rgb(0,0,0);font-family:Verdana,Arial,Helvetica,sans-serif;font-size:10px;border-width:0px
1px 0px
0px;border-style:solid;border-color:gray
rgb(204,204,204) gray gray;padding:0px"> </td>
<td
style="color:rgb(0,0,0);font-family:Verdana,Arial,Helvetica,sans-serif;font-size:10px;padding:0px;border:0px
solid gray"> </td>
<td
style="color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:11px;margin:8px;width:116px;border:0px
solid gray">
<p><img
src="http://www.ami.cz/images/podpis/ami_logo.gif"
alt="AMI Praha a.s." style="border:
0px;" moz-do-not-send="true"></p>
</td>
</tr>
<tr style="padding:0px;margin:0px;border:0px
solid gray">
<td colspan="7"
style="color:rgb(0,0,0);font-family:Verdana,Arial,Helvetica,sans-serif;font-size:10px;padding:0px;width:480px;border:0px
solid gray"><br>
<a
href="http://www.ami.cz/reseni-a-sluzby/bezpecnost-dat/identity-management"
target="_blank" moz-do-not-send="true"><img
src="http://www.ami.cz/images/podpis/AMI-podpis-IdM_1.png" alt="AMI
Praha a.s." style="border: 0px; width:
480px; height: 82px;"
moz-do-not-send="true"></a></td>
</tr>
<tr style="padding:0px;margin:0px;border:0px
solid gray">
<td colspan="7"
style="color:rgb(128,128,128);font-family:Arial,sans-serif;font-size:11px;padding:0px;border:0px
solid gray"><br>
Textem tohoto e-mailu podepisující neslibuje
uzavřít ani neuzavírá za společnost AMI
Praha a.s.<br>
jakoukoliv smlouvu. Každá smlouva, pokud
bude uzavřena, musí mít výhradně písemnou
formu.<br>
<br>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
midPoint mailing list
<a class="moz-txt-link-abbreviated" href="mailto:midPoint@lists.evolveum.com">midPoint@lists.evolveum.com</a>
<a class="moz-txt-link-freetext" href="http://lists.evolveum.com/mailman/listinfo/midpoint">http://lists.evolveum.com/mailman/listinfo/midpoint</a>
</pre>
</blockquote>
<br>
</body>
</html>