[midPoint] Speeding up scriptedsql

[Jason Everling]

Thanks!! After some debugging I spotted that the orgUnit was being removed,
assigned, then updated because of case sensitivity issues. I went in and
updated all the mapping related to transform to uppercase. That was the
main time killer because that change also triggers updates to be pushed to
resources dependent on dn. There are a few more but not time spent on this
mapping.

JASON


On Mon, Jan 21, 2019 at 6:30 AM Wojciech Staszewski <
wojciech.staszewski at diagnostyka.pl> wrote:

> Hello Jason!
>
> This is also my case.
> I was trying to debug operations executed by reconciliation tasks, so I
> ran "simpleResourceObjectNotifier" with output to a log file
> and noticed much more updates/modifications on the remote databases than I
> expected.
>
> The first cause was simple in my case: matching rules in the attribute
> outbound mapping, eg. "givenName", mapped to the uppercase
> on the resource, but matching rule id midPoint was "default", so midPoint
> was updating these attributes everytime.
> There was a lot attributes like that with incorrect matching rules, after
> corrections reconciliation tasks runs a little faster.
>
> Except that, I noticed that non-tolerant entitlement associations works
> strange.
> I mean that for every user every time operation "DELETE" is executed for
> not assigned entitlements (eg. group membership),
> even if the user is NOT a group member on the target system.
>
> Log:
> The account has been successfully modified on the resource. Modified
> attributes are:
> ...
>    - Associations:
>       - DELETE:
>          - name: zabbixGroups
>          - identifiers:
>             - ConnId Name: Zabbix administrators
>          - shadowRef: Zabbix administrators (shadow) on ZABBIX  [default]
>       - DELETE:
>          - name: zabbixGroups
>          - identifiers:
>             - ConnId Name: GROUP1
>          - shadowRef: GROUP1 (shadow) on ZABBIX  [default]
>       - DELETE:
>          - name: zabbixGroups
>          - identifiers:
>             - ConnId Name: GROUP2
>          - shadowRef: GROUP2 (shadow) on ZABBIX  [default]
>       - DELETE:
>          - name: zabbixGroups
>          - identifiers:
>             - ConnId Name: GROUP3
>          - shadowRef: GROUP3 (shadow) on ZABBIX  [default]
>       - DELETE:
>          - name: zabbixGroups
>          - identifiers:
>             - ConnId Name: GROUP4
>          - shadowRef: GROUP4 (shadow) on ZABBIX  [default]
>
> ...and so on,for every not assigned group and for every account. I guess
> it slows down the reconciliation.
> But I don't know if I can do anything about that...
>
>
> Regards,
> WS
>
>
> W dniu 03.01.2019 o 20:50, Jason Everling pisze:
> > It may just be me but it seems like in 3.7 the ScriptedSQL connector is
> slower than previous versions. Script reload is also set to false.
> >
> > I am running a recon now and it is processing 1000 per 10min, previously
> I remember it only taking about 45min to process this resource is 3.4. This
> resource has 15k+ accounts so I am looking at about 2 1/2 -3 hours. This is
> a new server when I migrated it to 3.7 , running stand-alone, not container
> on Ubnt 18.04, TC 8.x, MySQL 5.7, VM resides on ssd clustered datastore,
> 12Gb ram. I am going to see what I can do on the mysql side in the meantime.
> >
> > Any have any pointers to speed it up or maybe you have done something to
> improve it?
> >
> > Thanks!
> >
> > _______________________________________________
> > midPoint mailing list
> > midPoint at lists.evolveum.com
> > http://lists.evolveum.com/mailman/listinfo/midpoint
> >
>
> --
> Wojciech Staszewski
> Administrator Systemów Sieciowych
> www.diagnostyka.pl
> Diagnostyka Sp. z o. o.
> ul. Prof. M. Życzkowskiego 16, 31-864 Kraków
> Numer KRS: 0000381559 (Sąd Rejonowy dla Krakowa-Śródmieścia w Krakowie, XI
> Wydział Gospodarczy KRS)
> NIP: 675-12-65-009; REGON: 356366975
> Kapitał zakładowy: 33 756 500 zł.
>
> Pomyśl o środowisku zanim wydrukujesz ten e-mail.
> _______________________________________________
> midPoint mailing list
> midPoint at lists.evolveum.com
> http://lists.evolveum.com/mailman/listinfo/midpoint
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20190121/b08016df/attachment.htm>