[midPoint] Updates can get lost during a running recomputation task (SOLVED)

[Arnošt Starosta - AMI Praha a.s.]

*Problem : *

Midpoint in default configuration recomputes objects by first retrieving
them ALL from repository, then passing each object to a worker thread. If
the object was updated meanwhile (e.g. live-synced or updated from gui)
before it is recomputed by the worker thread, this update can be
overwritten by the object version retrieved when the recompute task
started. It happened on my deployment several times.

*Is your deployment affected? :*

Hard to say, i don't see any relevant log message to check. I had to check
by debugging the running recompute task and verifying that
SqlRepositoryServiceImpl.searchObjectsIterative calls
ObjectRetriever.searchObjectsIterativeByPaging (ok) and not
ObjectRetriever.searchObjectsIterativeAttempt (can loose updates).

Deployments with MySQL or H2 backend should be ok with default
configuration (check sources
SqlRepositoryConfiguration.computeDefaultIterativeSearchParameters). Did
not verify the runtime.

*Solution:*

Configure iterativeSearchByPaging and iterativeSearchByPagingBatchSize in
config.xml midpoint/repository element. Don't know if all backends support
this setting but postgres (which i use) does.

<configuration>

   <midpoint>

       <repository>

           …

           <iterativeSearchByPaging>true</iterativeSearchByPaging>


<iterativeSearchByPagingBatchSize>17</iterativeSearchByPagingBatchSize>

           …

       </repository>

   </midpoint>

</configuration>

After setting these parameters the objects to recompute are read in 'pages'
and fed to worker threads until the request queue between the reader thread
and worker threads is full, then the reader is blocked. The size of the
queue is hardcoded as 2 * number-of-worker-threads.

By setting the iterativeSearchByPagingBatchSize you can still loose
updates, but the time window when this can happen shrinks from
number-of-objects to max(page size, 2*num-of-worker-threads). Without much
thought i set the page size to (2 * number-of-worker-threads) + 1.

good luck
arnost

-- 

Arnošt Starosta
solution architect

gsm: [+420] 603 794 932
e-mail: arnost.starosta at ami.cz


AMI Praha a.s.
Pláničkova 11
162 00 Praha 6
tel.: [+420] 274 783 239
web: www.ami.cz


[image: AMI Praha a.s.]

[image: AMI Praha a.s.]
<http://www.ami.cz/reseni-a-sluzby/bezpecnost-dat/identity-management>

Textem tohoto e-mailu podepisující neslibuje uzavřít ani neuzavírá za
společnost AMI Praha a.s.
jakoukoliv smlouvu. Každá smlouva, pokud bude uzavřena, musí mít výhradně
písemnou formu.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20180129/d4f0b8c5/attachment.htm>