[midPoint] Large groups with OpenLDAP

[Carl Waldbieser]

Actually,

After double checking my test, I realized the loop was not working correctly, so the test is totally invalid.
I guess it is nice that I discovered this after positing it to 2 communication channels rather than betting on it at Atlantic City.

Oops.

Real timings worked out to 2m21.740s for 200 requests, so the average is really 0.7087 seconds per connection (there are 2 LDAP MOD requests per connection).  So it is still better than 6 seconds per change, but not as fantastically better as I had first thought.

My comments about an optimized client are still worth exploring, though.

Thanks,
Carl Waldbieser
ITS Identity Management
Lafayette College

----- Original Message -----
From: "Carl Waldbieser" <waldbiec at lafayette.edu>
To: "midPoint General Discussion" <midpoint at lists.evolveum.com>
Sent: Tuesday, May 22, 2018 2:49:49 PM
Subject: Re: [midPoint] Large groups with OpenLDAP

Keith,

I had posted on the tier midpoint slack channel, but our largest group is for alumni, and has about 32,000 members.
I can add and remove a member from the group fairly quickly.

On a *non-loaded* system I added and removed a member 100 times to the group, and the wall time was only 2.758s.  That meant that is single change was on average less than 30 milliseconds.  A lot of that could simply be network latency or the fact that my test opens a new TCP connection for each request.  An optimized client could make many changes simultaneously, and it wouldn't necessarily have to make a new TCP connection per request.

One noteworthy point is that we don't use the overlay that keeps user `memberOf` and group `member` attributes in sync (I think it is called the "memberOf overlay").  We keep these in sync via application code.  

Thanks,
Carl Waldbieser
ITS Identity Management
Lafayette College

----- Original Message -----
From: "Keith Hazelton" <keith.hazelton at wisc.edu>
To: "midPoint General Discussion" <midpoint at lists.evolveum.com>
Sent: Tuesday, May 22, 2018 1:36:34 PM
Subject: [midPoint] Large groups with OpenLDAP

I’d like to see midPoint/OpenLDAP experience-based reactions to this comment I saw just now: “OpenLDAP has issues with large groups. Large groups over 25-30k and you will have issues with group updates taking about 6 seconds for each change.” 

_______________________________________________
midPoint mailing list
midPoint at lists.evolveum.com
http://lists.evolveum.com/mailman/listinfo/midpoint