[midPoint] Large groups with OpenLDAP

[Gettes, Michael]

Double the number of users in the ldap group and what are the numbers???
Of course, maybe you need to create a test group.

I have run into this performance problem many times.

/mrg

> On May 22, 2018, at 2:57 PM, Carl Waldbieser <waldbiec at lafayette.edu> wrote:
> 
> Actually,
> 
> After double checking my test, I realized the loop was not working correctly, so the test is totally invalid.
> I guess it is nice that I discovered this after positing it to 2 communication channels rather than betting on it at Atlantic City.
> 
> Oops.
> 
> Real timings worked out to 2m21.740s for 200 requests, so the average is really 0.7087 seconds per connection (there are 2 LDAP MOD requests per connection).  So it is still better than 6 seconds per change, but not as fantastically better as I had first thought.
> 
> My comments about an optimized client are still worth exploring, though.
> 
> Thanks,
> Carl Waldbieser
> ITS Identity Management
> Lafayette College
> 
> ----- Original Message -----
> From: "Carl Waldbieser" <waldbiec at lafayette.edu>
> To: "midPoint General Discussion" <midpoint at lists.evolveum.com>
> Sent: Tuesday, May 22, 2018 2:49:49 PM
> Subject: Re: [midPoint] Large groups with OpenLDAP
> 
> Keith,
> 
> I had posted on the tier midpoint slack channel, but our largest group is for alumni, and has about 32,000 members.
> I can add and remove a member from the group fairly quickly.
> 
> On a *non-loaded* system I added and removed a member 100 times to the group, and the wall time was only 2.758s.  That meant that is single change was on average less than 30 milliseconds.  A lot of that could simply be network latency or the fact that my test opens a new TCP connection for each request.  An optimized client could make many changes simultaneously, and it wouldn't necessarily have to make a new TCP connection per request.
> 
> One noteworthy point is that we don't use the overlay that keeps user `memberOf` and group `member` attributes in sync (I think it is called the "memberOf overlay").  We keep these in sync via application code.  
> 
> Thanks,
> Carl Waldbieser
> ITS Identity Management
> Lafayette College
> 
> ----- Original Message -----
> From: "Keith Hazelton" <keith.hazelton at wisc.edu>
> To: "midPoint General Discussion" <midpoint at lists.evolveum.com>
> Sent: Tuesday, May 22, 2018 1:36:34 PM
> Subject: [midPoint] Large groups with OpenLDAP
> 
> I’d like to see midPoint/OpenLDAP experience-based reactions to this comment I saw just now: “OpenLDAP has issues with large groups. Large groups over 25-30k and you will have issues with group updates taking about 6 seconds for each change.” 
> 
> _______________________________________________
> midPoint mailing list
> midPoint at lists.evolveum.com
> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.evolveum.com_mailman_listinfo_midpoint&d=DwIGaQ&c=pZJPUDQ3SB9JplYbifm4nt2lEVG5pWx2KikqINpWlZM&r=EUBHI54mtQDlbcqo5rUTdQ&m=L5F_71rBSl-qfVIlSy87F8qwtqykVvdJeDHkxKCi1YY&s=FkoFozgGrgzyI63qaoJeX0Dp2bIjF9-6gZj99ZrAi-w&e=
> _______________________________________________
> midPoint mailing list
> midPoint at lists.evolveum.com
> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.evolveum.com_mailman_listinfo_midpoint&d=DwIGaQ&c=pZJPUDQ3SB9JplYbifm4nt2lEVG5pWx2KikqINpWlZM&r=EUBHI54mtQDlbcqo5rUTdQ&m=L5F_71rBSl-qfVIlSy87F8qwtqykVvdJeDHkxKCi1YY&s=FkoFozgGrgzyI63qaoJeX0Dp2bIjF9-6gZj99ZrAi-w&e=