[midPoint] imort from DB

Wed Mar 25 09:31:31 CET 2020

Hello Anton,

for "stateful" task suspension and resume you need to use buckets. It is 
still a kind of experimental functionality, but basically it should work.

See e.g. here 
https://wiki.evolveum.com/display/midPoint/Multi-node%2C+partitioned+and+stateful+tasks

However, 65000 entries is not so much. If you say 20 hours, this is 
approximately 900 milliseconds per account. This might or might not be 
adequate, depending on the runtime complexity of your configuration.

As Oskar has said, paralellizing (i.e. using worker threads) should help 
here. Using multiple nodes can help further.

But I would suggest looking at internal performance numbers as a hint 
where performance can be improved. Are mappings slow? Is repository 
slow? Is there anything else that slows down the execution, e.g. 
notifications? Could more aggresive caching or some experimental 
optimization options help?

The optimization matters are a little bit complex, so maybe our 
professional services would be most beneficial to you. It depends.

Best regards,

Pavol Mederly
Software developer
evolveum.com

On 25/03/2020 08:55, Oskar Butovič - AMI Praha a.s. wrote:
> Hello Anton,
>
> for the reconciliation task, you can use multiple worker threads. 
> https://wiki.evolveum.com/display/midPoint/Parallelized+background+tasks
>
> As long as you have enough CPU cores and your DB handles the load 
> without failures midpoint performance in reconciliation scales almost 
> linearly with the number of the worker threads.
>
> Best Regards
> Oskar Butovič
>
> st 25. 3. 2020 v 8:04 odesílatel Щенев Антон Вячеславович 
> <anton.shchenev at beeper.ru <mailto:anton.shchenev at beeper.ru>> napsal:
>
>     Hello!
>
>     Could you ask me pls, what best practices for work with big DB
>     tables(>65000 entires)(Import some attributes for midPoint’s  users)
>
>     The task processing  8 hours and this time not enough,
>     approximately need 20 hours
>
>     Can some script optimization help in synchronization? Or may be
>     there are some variants ?
>
>     I tried this setting: Configured worker threads, but it was not so
>     useful
>
>     Scheduling does not provide an opportunity to suspend a task (or
>     maybe I'm wrong?)
>
>     Every time he starts it again…
>
>     I would be grateful for some kind of answer, because this
>     functionality is very good and very flexible
>
>     Описание: Описание: Описание: cid:image004.png at 01D47D0D.3B8B0380
>
>     С уважением,
>
>     Щенев Антон Вячеславович
>
>     Ведущий инженер по ИБ
>
>     BEEPER (ООО «Инфо-контент»)
>
>     620075, г. Екатеринбург,
>
>     Улица Розы Люксембург, дом 19, 4 этаж
>
>     8-343-300-00-11 доп. 3392
>
>     anton.shchenev at beeper.ru <mailto:anton.shchenev at beeper.ru>
>
>     www.beeper.ru <http://www.beeper.ru/>
>
>     Описание: Описание: Описание: cid:image010.png at 01D47D0D.3B8B0380
>
>     Описание: Описание: Описание: ВК
>     <https://vk.com/beeperteam>Описание: Описание: Описание: FB
>     <https://www.facebook.com/beeper.ru>Описание: Описание: Описание:
>     OK <https://ok.ru/group/55521990672392>Описание: Описание:
>     Описание: In <https://www.instagram.com/beeper_team/>
>
>     _______________________________________________
>     midPoint mailing list
>     midPoint at lists.evolveum.com <mailto:midPoint at lists.evolveum.com>
>     https://lists.evolveum.com/mailman/listinfo/midpoint
>
>
>
> -- 
>
> *Oskar Butovič*
> vedoucí vývoje
>
> gsm: [+420] 774 480 101
> e‑mail: oskar.butovic at ami.cz <mailto:oskar.butovic at ami.cz>
>
> *AMI Praha a.s.*
> Pláničkova 11, 162 00 Praha 6
>
> tel.: [+420] 274 783 239 | web: www.ami.cz <https://www.ami.cz/>
>
> AMI Praha a.s.
>
> Textem tohoto e‑mailu podepisující neslibuje uzavřít ani neuzavírá 
> za společnost AMI Praha a.s.
> jakoukoliv smlouvu. Každá smlouva, pokud bude uzavřena, musí mít 
> výhradně písemnou formu.
>
> Tento e‑mail je určen výhradně pro potřeby jeho adresáta/ů a může 
> obsahovat důvěrné nebo osobní
> informace. Nejste‑li zamýšleným příjemcem, je zakázáno jakékoliv 
> zveřejňování, zprostředkování
> nebo jiné použití těchto informací. Pokud jste obdrželi e‑mail 
> neoprávněně, informujte o tom prosím
> odesílatele a vymažte neprodleně všechny kopie tohoto e‑mailu včetně 
> všech jeho příloh. Nakládáním
> s neoprávněně získanými informacemi se vystavujete riziku právního 
> postihu.
>
>
> _______________________________________________
> midPoint mailing list
> midPoint at lists.evolveum.com
> https://lists.evolveum.com/mailman/listinfo/midpoint
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20200325/cabc662b/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 1457 bytes
Desc: not available
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20200325/cabc662b/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 9061 bytes
Desc: not available
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20200325/cabc662b/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 1487 bytes
Desc: not available
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20200325/cabc662b/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 1007 bytes
Desc: not available
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20200325/cabc662b/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image005.png
Type: image/png
Size: 1897 bytes
Desc: not available
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20200325/cabc662b/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image006.png
Type: image/png
Size: 3984 bytes
Desc: not available
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20200325/cabc662b/attachment-0005.png>