[midPoint] LDAP role/group inducement
Ivan Noris
ivan.noris at evolveum.com
Tue Jan 28 14:37:21 CET 2020
Hi Jan,
> @Ivan Noris & @Pavol Mederly
> Ivan, your suggestion that I should remove the native-capabilities tag
> from the opendj resource did the trick (uniqueMember attribute is
> cleared from the group when deleting the entitlement). I think that I
> took that part from an xml snippet from the mailing list.
It was not just the capabilities, but also the ICF result handlers (see
https://wiki.evolveum.com/display/midPoint/ICF+Issues#ICFIssues-ResultHandlers
- one of the trickiest ICF legacy). Anyway, native capabilities should
be always fetched by the connector.
> Together with the fact that I did not read up on that part in
> Confluence: it was a disaster in the making.
> My sanity has returned along with my confidence in the fact that
> midPoint is the right tool for our use-case.
Having the confidence for midPoint is always good thing and we are glad
for that. Especially when there were issues that are now resolved. And
especially when the resolution is by a configuration, not a bugfix.
Yes there could be some issue also with the documentation and anyone is
welcome to report such issues to be solved or even contribute the
fix(es). In parallel we are considering how to improve our documentation
in reasonable time.
> Pavol, thank you for your detailed feedback on my clumsy attempt at
> bypassing my mis-configuration with hacks :) As I said, I knew I was
> doing iffy stuff by commenting that line. But one does strange things
> when facing a seemingly insurmountable problem.
> I am greatly indebted for your help and I must say I am impressed with
> the state of your project (well documented Confluence, readable code,
> helpful community).
> FYI: We are seeking professional help from DAASI for the follow-up
> verification of config/usage/deployment. I wanted to verify the basic
> usage in a PoC myself to see if midPoint was a fit.
Good to hear!
Good luck & best regards
Ivan
>
> Op di 28 jan. 2020 om 08:40 schreef Jan Lievens
> <jan.lievens at biggerfish.be <mailto:jan.lievens at biggerfish.be>>:
>
> So now my conclusion is that the NPE in the admin points to the
> fact that together with name and identifiers there should also be
> a shadowRef saved in the shadow (PostgreSQL account) repo.
> We see that only the name and identifiers are saved in the
> beginning (first import task). I suppose the shadowRef is at that
> point not resolvable (perhaps the entitlements are not imported
> yet so the shadow of an entitlement is not yet available).
> Now we also see that if we perform a null safe check on the NPE
> site our use-case works (uniqueMember is cleared of the LDAP group
> when we remove entitlement from PostgreSQL) but we see that the
> shadow of the entitlement is lingering in the repo (and indeed
> when we again add the entitlement again in PostgreSQL an exception
> is thrown saying that there are conflicting shadows in the import
> task run).
> So my hunch is that the shadowRef should be saved on the
> association when possible. I see that during the reconciliation
> the shardowRef is correctly added beside the name and identifiers
> tags (but alas never saved to the repo).
> I will pursue this line of thinking. My best guess is to save this
> shadowRef in the association someplace in the ShadowManager.
>
> Op ma 27 jan. 2020 om 15:15 schreef Jan Lievens
> <jan.lievens at biggerfish.be <mailto:jan.lievens at biggerfish.be>>:
>
> So I figured out what the NullPointerException is all about:
> The code at line ShadowAssociationWrapperFactoryImpl.java:193
> "shadowAss.add(associationValue.findReference(ShadowAssociationType.F_SHADOW_REF).getValue().clone());"
> expects to have a shadowRef in the association which it does
> not have (it only has name and identifiers tags, like can be
> seen from my previous post). Now I would say that this missing
> shadowRef needs to be fixed next because when I null check
> this (and thus this line is not performed) the
> lingering-"uniqueMember" bug in LDAP is resolved and the GUI
> admin does not crash with a NPE but the shadow (the shadow of
> the entitlement that is, for which the shadowRef should exist)
> is not cleaned up and can still be seen in the repo. So when I
> again add the entitlement in PostgreSQL and perform an import
> task on that resource it crashes on the fact that it sees
> conflicting shadows (no surprise there).
>
> Op za 25 jan. 2020 om 15:20 schreef Jan Lievens
> <jan.lievens at biggerfish.be <mailto:jan.lievens at biggerfish.be>>:
>
> So I debugged this problem with the lacking "association"
> item in the account shadow of the PostgreSQL resource. So
> again a tl;dr for the people with no time to follow my
> lengthy explanation: the lingering uniqueMember reference
> gets cleaned up in LDAP when I remove
> line com.evolveum.midpoint.repo.sql.helpers.ObjectRetriever.java:557
> "prismObject.removeContainer(ShadowType.F_ASSOCIATION);"
>
> So what I see is that the shadow that gets written (xml in
> fullObject column) the first time into the repo and it
> contains the "association" item.
>
> <shadow
> xmlns="http://midpoint.evolveum.com/xml/ns/public/common/common-3"
> xmlns:c="http://midpoint.evolveum.com/xml/ns/public/common/common-3"
> xmlns:icfs="http://midpoint.evolveum.com/xml/ns/public/connector/icf-1/resource-schema-3"
> xmlns:org="http://midpoint.evolveum.com/xml/ns/public/common/org-3"
> xmlns:q="http://prism.evolveum.com/xml/ns/public/query-3"
> xmlns:ri="http://midpoint.evolveum.com/xml/ns/public/resource/instance-3"
> xmlns:t="http://prism.evolveum.com/xml/ns/public/types-3"
> oid="954b8cfe-5c84-4f82-946f-0196dcec92c0" version="0">
> <name>810710333333</name>
> <resourceRef
> oid="dafe0482-faef-47b7-ae08-66b150929bb7"
> relation="org:default" type="c:ResourceType">
> <!-- Scripted SQL -->
> </resourceRef>
>
> <objectClass>ri:AccountObjectClass</objectClass>
>
> <primaryIdentifierValue>3fd83cd4-d1bb-4d7f-9a1a-12a02ed85a95</primaryIdentifierValue>
> <kind>account</kind>
> <exists>true</exists>
> <attributes>
> <icfs:name>810710333333</icfs:name>
>
> <icfs:uid>3fd83cd4-d1bb-4d7f-9a1a-12a02ed85a95</icfs:uid>
> </attributes>
> <association id="1">
> <name>ri:ents</name>
> <identifiers>
>
> <icfs:name>20001-Milieumedewerker-01</icfs:name>
> </identifiers>
> </association>
> <activation/>
> </shadow>
>
> It is afterwards (in a following reconciliation wave)
> that this gets overwritten with a version that has no
> association. Poof gone...
>
> <shadow
> xmlns="http://midpoint.evolveum.com/xml/ns/public/common/common-3"
> xmlns:c="http://midpoint.evolveum.com/xml/ns/public/common/common-3"
> xmlns:icfs="http://midpoint.evolveum.com/xml/ns/public/connector/icf-1/resource-schema-3"
> xmlns:org="http://midpoint.evolveum.com/xml/ns/public/common/org-3"
> xmlns:q="http://prism.evolveum.com/xml/ns/public/query-3"
> xmlns:ri="http://midpoint.evolveum.com/xml/ns/public/resource/instance-3"
> xmlns:t="http://prism.evolveum.com/xml/ns/public/types-3"
> oid="954b8cfe-5c84-4f82-946f-0196dcec92c0" version="1">
> <name>810710333333</name>
> <resourceRef
> oid="dafe0482-faef-47b7-ae08-66b150929bb7"
> relation="org:default" type="c:ResourceType"/>
>
> <synchronizationTimestamp>2020-01-24T13:30:38.108Z</synchronizationTimestamp>
>
> <fullSynchronizationTimestamp>2020-01-24T13:30:38.108Z</fullSynchronizationTimestamp>
>
> <objectClass>ri:AccountObjectClass</objectClass>
>
> <primaryIdentifierValue>3fd83cd4-d1bb-4d7f-9a1a-12a02ed85a95</primaryIdentifierValue>
> <kind>account</kind>
> <intent>webidm</intent>
> <exists>true</exists>
> <attributes
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xmlns:c="http://midpoint.evolveum.com/xml/ns/public/common/common-3"
> xsi:type="c:ShadowAttributesType">
> <icfs:name>810710333333</icfs:name>
>
> <icfs:uid>3fd83cd4-d1bb-4d7f-9a1a-12a02ed85a95</icfs:uid>
> </attributes>
> <activation/>
> </shadow>
>
> Immediately we see that the following modifications made
> to the just created shadow: REPLACE intent, kind,
> synchronizationTimestamp and fullSynchronizationTimestamp.
> It is in this replace that the "association"-part is
> removed and written into the repo. It is on position
> com.evolveum.midpoint.repo.sql.helpers.ObjectRetriever.java:557
> that the association from the shadow gets removed in a
> very explicit fashion with the ominous comment: "we store
> it because provisioning now sends it to repo, but it
> should be transient".
> When we remove this line that removes the association we
> get a fat NPE in the admin tool when opening the user, but
> our bug is fixed 😎.
> But somehow I can't shake the feeling that removing this
> line is hacky (I suppose other code might also depend on
> this line being there) and that the association should get
> fetched from the inbound source (i.e. PostgreSQL) at the
> right moment. But again I'm too novice to really know how
> to configure that. Something of the likes of
> <searchStrategy>onResourceIfNeeded</searchStrategy> maybe?
> If someone could shime in here to actually verify that
> this is a bug then I could send in a pull request with the
> fix. I will look at the NPE in de admin GUI this monday so
> then I will know more on how unorthodox this fix is.
>
>
>
> Op vr 24 jan. 2020 om 10:14 schreef Jan Lievens
> <jan.lievens at biggerfish.be
> <mailto:jan.lievens at biggerfish.be>>:
>
> Before diving into the more technical description of
> why I came to my conclusion a short tl;dr for the
> impatient: something is probably not correct in my
> schemaHandling part of the Scripted SQL resource with
> respect to the entitlement association. What that is,
> I have no clue about.
>
> So I did a bit of code spelunking in midPoint to
> better understand the problem. Here is what I came up
> with:
>
> - To see what happened in the code with respect to
> the LDAP association I put logger
> com.evolveum.midpoint.provisioning into the debug
> level which provides me with some fine log statement
> when the association (user->group: uniqueMember on the
> group) is made in LDAP:
> 2020-01-23 08:58:22,737 [PROVISIONING]
> [midPointScheduler_Worker-2] DEBUG
> (com.evolveum.midpoint.provisioning.impl.ResourceObjectConverter):
> PROVISIONING MODIFY operation on
> resource:d0811790-6420-11e4-86b2-3c9755567874(OpenDJ)
> MODIFY object, object class Technical Role, identified by:
> {
> entryUUID: 88972d7b-6deb-4d94-b343-d039099d23c4
> dn: cn=jirauser,ou=groups,dc=didm,dc=be
> }
> changes: [
> PropertyModificationOperation:
> delta:
> attributes/uniqueMember
> ADD: uid=81071022511,ou=people,dc=didm,dc=be
> matchingRule:
> {http://prism.evolveum.com/xml/ns/public/matching-rule-3}stringIgnoreCase
> <http://prism.evolveum.com/xml/ns/public/matching-rule-3%7DstringIgnoreCase>
> ]
> - So basically I had to find out why the inverse is
> not happening when the association is severed (when we
> delete the entitlement in the PostgreSQL database)
> - The conclusion I get from the code is that this
> should happen in the
> ReconciliationProcessor.reconcileProjectionAssociations
> method. More specifically at the end of that method it
> is decided to tolerate the association or not
> (tolerant=false on the association is a required thing
> after all 😀).
> - In this decideIfTolerateAssociation method I see
> this beautiful call that is going to solve my world of
> pain:
> recordAssociationDelta(valueMatcher, accCtx, assocDef,
> ModificationType.DELETE, isCValue.getValue(), null,
> "it is not given by any mapping and the association is
> not tolerant");
> - In my configuration this statement is not evaluated
> because it requires that there is a
> PrismContainerValue<ShadowAssociationType> provided in
> the arguments of this method.
> - And why is there no ShadowAssociationType in this
> case I hear you ask? Well because in
> method com.evolveum.midpoint.prism.impl.PrismContainerValueImpl#findItemByQName
> it cannot find an "association" item in the shadow of
> the PostgreSQL account (see screenshot in attachment).
> - And indeed when I look at that specific shadow's
> repository object xml, I see nothing that looks like
> an association.
> - And why is there no "association" item on the
> PostgreSQL account shadow? That I do not know. Maybe
> somebody can put me out of my misery?
> - So basically I assume something is fishy about my
> schemaHandling part of the Scripted SQL resource with
> respect to the entitlement association.
>
> Sorry if this is somewhat too technical, I realise
> there is a dev mailing list but I wanted to add it
> here so that people find the solution and problem in
> the same space.
>
> Op di 21 jan. 2020 om 15:50 schreef Jan Lievens
> <jan.lievens at biggerfish.be
> <mailto:jan.lievens at biggerfish.be>>:
>
> Hi,
>
> May I also add that I tried this setup with
> midPoint versions 4.0.1, 4.0 and 3.9 all with the
> same result (lingering uniqueMember attributes in
> LDAP).
>
>
> Op di 21 jan. 2020 om 14:47 schreef Jan Lievens
> <jan.lievens at biggerfish.be
> <mailto:jan.lievens at biggerfish.be>>:
>
> Hi all,
>
> I have reduced the complexity of my use case
> and made it easy to reproduce. Starting from
> the unix story
> (https://wiki.evolveum.com/display/midPoint/Unix+Story+Test)
> Now I have the following situation:
> - Sync accounts from CSV to OpenDJ (works great)
> - I have one meta-role that is assigned to a
> test role which results in an LDAP group
> (works great)
> - Then I assign this test role to a user
> which results in a uniqueMember attribute on
> the LDAP group (nice)
> - Then I unassign the test role from said
> user and the uniqueMember attribute on the
> LDAP group is not getting removed (😭)
> - The uniqueMember attribute cannot be
> removed even after running reconcile/recompute
> on accounts and entitlements
>
> More succinct description to reproduce:
> - git checkout
> https://github.com/jalieven/midpoint-docker
> - cd midpoint-docker
> - git checkout simple-ldap-group
> - 'make restart' (starts docker with
> postgres/opendj/midpoint)
> - go to admin GUI http://localhost:8080/midpoint
> - login 'administrator' pwd '5ecr3t'
> - create a new role 'TestGroup' with an
> assignment to role 'LDAP Group Metarole'
> [- in OpenDJ: be.didm.group.TestGroup is
> created] => OK (checking can be done with
> 'make check_ldap' in root project)
> - in admin GUI: add 'TestGroup' to 'user01'
> assignments
> [- in OpenDJ: the be.didm.group.TestGroup gets
> the attribute uniqueMember set to
> 'uid=user01,ou=people,dc=didm,dc=be'] => OK
> - in admin GUI: remove 'TestGroup' from
> 'user01' assignments (minus button and SAVE)
> [- in OpenDJ: nothing happens: uniqueMember is
> not removed from be.didm.group.TestGroup] =>
> Not OK!
> - to verify this stale uniqueMember attribute:
> in root of project do 'make check_ldap'
>
> Config files can be found in
> demo/simple/midpoint_server/container_files/mp-home/post-initial-objects
>
> As you can see reproducing this
> bug/miss-config takes 5min of work. If this is
> a miss-configuration you might want to check
> because the same is happening in the unix
> story test. If this is a bug in midPoint all
> the more so to look at it. It is hard for me
> as a novice to say which one it is.
>
> It seems to me that this is a common use-case.
> I have come across this issue a few times in
> this mailing list, none of which provide a
> sufficient solution:
> http://lists.evolveum.com/pipermail/midpoint/2016-April/001720.html
> (the 'it-works-on-my-machine'-stance is kind
> of lame in this case and it doesn't help that
> the referred zip files does not resolve)
> http://lists.evolveum.com/pipermail/midpoint/2016-November/002807.html
> (a lot of tolerant/strength stuff here)
> http://lists.evolveum.com/pipermail/midpoint/2016-November/002843.html
> (suggests association shortcuts and
> multivalued schema fix in scripted sql
> connector, the latter of which does not apply
> in my described use-case)
>
> The midPoint Confluence suggests also to set
> various packages to trace log level to debug
> what is going on but I must say that in so
> much logging I have no idea what to look for.
> When I set the projector/clockwork to debug
> all I see is that the remove of the
> association is not happening. More
> specifically this package on
> trace org.identityconnectors.framework.spi.operations
> points to the fact that the delete is not
> happening in the connector.
>
> May I also add that the
> page https://wiki.evolveum.com/display/midPoint/Initial+Logging+Setup+HOWTO is
> not up to date. Setting the
> skipRepositoryLoggingSettings does not work.
> So there is currently no way to add specific
> loggers at start up through the
> logback(-extra).xml. It always gets
> overwritten by what is inside the repository.
> It appears that the
> skipRepositoryLoggingSettings was removed from
> the code some time ago.
>
> Kind Regards,
> Jan Lievens
>
> Op wo 15 jan. 2020 om 16:19 schreef Jan
> Lievens <jan.lievens at biggerfish.be
> <mailto:jan.lievens at biggerfish.be>>:
>
> Hi,
>
> I have a question related to inducement
> construction on a role toward an LDAP
> resource.
> I have the following situation in midPoint
> (see the attachment for all the xml files)
> - Accounts and Entitlements (intent:
> privileges) are imported from a
> ScriptedSQL connector.
> - These imported entitlements have
> multiple privileges which are created as
> roles with an "assignmentTargetSearch"
> (post-initial-objects/210-entitlement-object-template.xml)
> - I also have these privilege/roles
> defined with an assignment to (multiple)
> fixed technical roles
> (kind:entitlement/intent:group).
> - Lastly I would like for these technical
> roles (groups) and the associated accounts
> to get synced to LDAP in an
> objectToSubject fashion. I do this with
> inducements and construction tags in
> post-initial-objects/100-profiles-webidm.xml.
> - The result I get in LDAP is that
> accounts are synced correctly but the
> group names are not what I expect (I
> expect technical role names eg. JiraUser)
> but get the names of the Entitlements
> (intent:privileges) defined in the DB.
> - Additionally the associations are not
> synced (no uniqueMember refs are synced on
> the LDAP groups).
>
> Surely I am missing something here. I
> think this use case is quiet standard (a
> hierarchy of roles and only the leafs get
> synced to LDAP).
> I have experimented with the order of the
> inducement but these came out even more
> negative (no accounts or groups were created).
> I also have experimented with the tolerant
> setting on the association (since I find a
> lot of answers in this mailing list
> suggesting this) but to no avail.
> It is quiet frustrating to be so close to
> having this use-case implemented
> DB->mP->LDAP but failing in the sync to LDAP.
>
> Kind regards,
> --
> Jan Lievens
> IT Consultant
>
>
>
> --
> Jan Lievens
> IT Consultant
> Bigger Fish
> Hoogstraat 35
> 9450 Haaltert
> Belgium
> VAT: BE 0734.993.447
> Mob: +32 494 84 66 97
>
>
>
> --
> Jan Lievens
> IT Consultant
> Bigger Fish
> Hoogstraat 35
> 9450 Haaltert
> Belgium
> VAT: BE 0734.993.447
> Mob: +32 494 84 66 97
>
>
>
> --
> Jan Lievens
> IT Consultant
> Bigger Fish
> Hoogstraat 35
> 9450 Haaltert
> Belgium
> VAT: BE 0734.993.447
> Mob: +32 494 84 66 97
>
>
>
> --
> Jan Lievens
> IT Consultant
> Bigger Fish
> Hoogstraat 35
> 9450 Haaltert
> Belgium
> VAT: BE 0734.993.447
> Mob: +32 494 84 66 97
>
>
>
> --
> Jan Lievens
> IT Consultant
> Bigger Fish
> Hoogstraat 35
> 9450 Haaltert
> Belgium
> VAT: BE 0734.993.447
> Mob: +32 494 84 66 97
>
>
>
> --
> Jan Lievens
> IT Consultant
> Bigger Fish
> Hoogstraat 35
> 9450 Haaltert
> Belgium
> VAT: BE 0734.993.447
> Mob: +32 494 84 66 97
>
>
>
> --
> Jan Lievens
> IT Consultant
> Bigger Fish
> Hoogstraat 35
> 9450 Haaltert
> Belgium
> VAT: BE 0734.993.447
> Mob: +32 494 84 66 97
>
> _______________________________________________
> midPoint mailing list
> midPoint at lists.evolveum.com
> http://lists.evolveum.com/mailman/listinfo/midpoint
--
Ivan Noris
Senior Identity Engineer
evolveum.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20200128/e6337bdc/attachment.htm>
More information about the midPoint
mailing list