[midPoint] LDAP role/group inducement
Jan Lievens
jan.lievens at biggerfish.be
Tue Jan 28 14:22:12 CET 2020
@Ivan Noris & @Pavol Mederly
Ivan, your suggestion that I should remove the native-capabilities tag from
the opendj resource did the trick (uniqueMember attribute is cleared from
the group when deleting the entitlement). I think that I took that part
from an xml snippet from the mailing list.
Together with the fact that I did not read up on that part in Confluence:
it was a disaster in the making.
My sanity has returned along with my confidence in the fact that midPoint
is the right tool for our use-case.
Pavol, thank you for your detailed feedback on my clumsy attempt at
bypassing my mis-configuration with hacks :) As I said, I knew I was doing
iffy stuff by commenting that line. But one does strange things when facing
a seemingly insurmountable problem.
I am greatly indebted for your help and I must say I am impressed with the
state of your project (well documented Confluence, readable code, helpful
community).
FYI: We are seeking professional help from DAASI for the follow-up
verification of config/usage/deployment. I wanted to verify the basic usage
in a PoC myself to see if midPoint was a fit.
Op di 28 jan. 2020 om 08:40 schreef Jan Lievens <jan.lievens at biggerfish.be>:
> So now my conclusion is that the NPE in the admin points to the fact that
> together with name and identifiers there should also be a shadowRef saved
> in the shadow (PostgreSQL account) repo.
> We see that only the name and identifiers are saved in the beginning
> (first import task). I suppose the shadowRef is at that point not
> resolvable (perhaps the entitlements are not imported yet so the shadow of
> an entitlement is not yet available).
> Now we also see that if we perform a null safe check on the NPE site our
> use-case works (uniqueMember is cleared of the LDAP group when we remove
> entitlement from PostgreSQL) but we see that the shadow of the entitlement
> is lingering in the repo (and indeed when we again add the entitlement
> again in PostgreSQL an exception is thrown saying that there are
> conflicting shadows in the import task run).
> So my hunch is that the shadowRef should be saved on the association when
> possible. I see that during the reconciliation the shardowRef is correctly
> added beside the name and identifiers tags (but alas never saved to the
> repo).
> I will pursue this line of thinking. My best guess is to save this
> shadowRef in the association someplace in the ShadowManager.
>
> Op ma 27 jan. 2020 om 15:15 schreef Jan Lievens <jan.lievens at biggerfish.be
> >:
>
>> So I figured out what the NullPointerException is all about: The code at
>> line ShadowAssociationWrapperFactoryImpl.java:193
>> "shadowAss.add(associationValue.findReference(ShadowAssociationType.F_SHADOW_REF).getValue().clone());"
>> expects to have a shadowRef in the association which it does not have (it
>> only has name and identifiers tags, like can be seen from my previous
>> post). Now I would say that this missing shadowRef needs to be fixed next
>> because when I null check this (and thus this line is not performed) the
>> lingering-"uniqueMember" bug in LDAP is resolved and the GUI admin does not
>> crash with a NPE but the shadow (the shadow of the entitlement that is, for
>> which the shadowRef should exist) is not cleaned up and can still be seen
>> in the repo. So when I again add the entitlement in PostgreSQL and perform
>> an import task on that resource it crashes on the fact that it sees
>> conflicting shadows (no surprise there).
>>
>> Op za 25 jan. 2020 om 15:20 schreef Jan Lievens <
>> jan.lievens at biggerfish.be>:
>>
>>> So I debugged this problem with the lacking "association" item in the
>>> account shadow of the PostgreSQL resource. So again a tl;dr for the people
>>> with no time to follow my lengthy explanation: the lingering uniqueMember
>>> reference gets cleaned up in LDAP when I remove
>>> line com.evolveum.midpoint.repo.sql.helpers.ObjectRetriever.java:557
>>> "prismObject.removeContainer(ShadowType.F_ASSOCIATION);"
>>>
>>> So what I see is that the shadow that gets written (xml in fullObject
>>> column) the first time into the repo and it contains the "association"
>>> item.
>>>
>>> <shadow xmlns="
>>> http://midpoint.evolveum.com/xml/ns/public/common/common-3" xmlns:c="
>>> http://midpoint.evolveum.com/xml/ns/public/common/common-3" xmlns:icfs="
>>> http://midpoint.evolveum.com/xml/ns/public/connector/icf-1/resource-schema-3"
>>> xmlns:org="http://midpoint.evolveum.com/xml/ns/public/common/org-3"
>>> xmlns:q="http://prism.evolveum.com/xml/ns/public/query-3" xmlns:ri="
>>> http://midpoint.evolveum.com/xml/ns/public/resource/instance-3"
>>> xmlns:t="http://prism.evolveum.com/xml/ns/public/types-3"
>>> oid="954b8cfe-5c84-4f82-946f-0196dcec92c0" version="0">
>>> <name>810710333333</name>
>>> <resourceRef
>>> oid="dafe0482-faef-47b7-ae08-66b150929bb7" relation="org:default"
>>> type="c:ResourceType">
>>> <!-- Scripted SQL -->
>>> </resourceRef>
>>> <objectClass>ri:AccountObjectClass</objectClass>
>>>
>>> <primaryIdentifierValue>3fd83cd4-d1bb-4d7f-9a1a-12a02ed85a95</primaryIdentifierValue>
>>> <kind>account</kind>
>>> <exists>true</exists>
>>> <attributes>
>>> <icfs:name>810710333333</icfs:name>
>>>
>>> <icfs:uid>3fd83cd4-d1bb-4d7f-9a1a-12a02ed85a95</icfs:uid>
>>> </attributes>
>>> <association id="1">
>>> <name>ri:ents</name>
>>> <identifiers>
>>>
>>> <icfs:name>20001-Milieumedewerker-01</icfs:name>
>>> </identifiers>
>>> </association>
>>> <activation/>
>>> </shadow>
>>>
>>> It is afterwards (in a following reconciliation wave) that this gets
>>> overwritten with a version that has no association. Poof gone...
>>>
>>> <shadow xmlns="
>>> http://midpoint.evolveum.com/xml/ns/public/common/common-3" xmlns:c="
>>> http://midpoint.evolveum.com/xml/ns/public/common/common-3" xmlns:icfs="
>>> http://midpoint.evolveum.com/xml/ns/public/connector/icf-1/resource-schema-3"
>>> xmlns:org="http://midpoint.evolveum.com/xml/ns/public/common/org-3"
>>> xmlns:q="http://prism.evolveum.com/xml/ns/public/query-3" xmlns:ri="
>>> http://midpoint.evolveum.com/xml/ns/public/resource/instance-3"
>>> xmlns:t="http://prism.evolveum.com/xml/ns/public/types-3"
>>> oid="954b8cfe-5c84-4f82-946f-0196dcec92c0" version="1">
>>> <name>810710333333</name>
>>> <resourceRef
>>> oid="dafe0482-faef-47b7-ae08-66b150929bb7" relation="org:default"
>>> type="c:ResourceType"/>
>>>
>>> <synchronizationTimestamp>2020-01-24T13:30:38.108Z</synchronizationTimestamp>
>>>
>>> <fullSynchronizationTimestamp>2020-01-24T13:30:38.108Z</fullSynchronizationTimestamp>
>>> <objectClass>ri:AccountObjectClass</objectClass>
>>>
>>> <primaryIdentifierValue>3fd83cd4-d1bb-4d7f-9a1a-12a02ed85a95</primaryIdentifierValue>
>>> <kind>account</kind>
>>> <intent>webidm</intent>
>>> <exists>true</exists>
>>> <attributes xmlns:xsi="
>>> http://www.w3.org/2001/XMLSchema-instance" xmlns:c="
>>> http://midpoint.evolveum.com/xml/ns/public/common/common-3"
>>> xsi:type="c:ShadowAttributesType">
>>> <icfs:name>810710333333</icfs:name>
>>>
>>> <icfs:uid>3fd83cd4-d1bb-4d7f-9a1a-12a02ed85a95</icfs:uid>
>>> </attributes>
>>> <activation/>
>>> </shadow>
>>>
>>> Immediately we see that the following modifications made to the just
>>> created shadow: REPLACE intent, kind, synchronizationTimestamp and
>>> fullSynchronizationTimestamp. It is in this replace that the
>>> "association"-part is removed and written into the repo. It is on position
>>> com.evolveum.midpoint.repo.sql.helpers.ObjectRetriever.java:557 that the
>>> association from the shadow gets removed in a very explicit fashion with
>>> the ominous comment: "we store it because provisioning now sends it to
>>> repo, but it should be transient".
>>> When we remove this line that removes the association we get a fat NPE
>>> in the admin tool when opening the user, but our bug is fixed 😎.
>>> But somehow I can't shake the feeling that removing this line is hacky
>>> (I suppose other code might also depend on this line being there) and that
>>> the association should get fetched from the inbound source (i.e.
>>> PostgreSQL) at the right moment. But again I'm too novice to really know
>>> how to configure that. Something of the likes of
>>> <searchStrategy>onResourceIfNeeded</searchStrategy> maybe?
>>> If someone could shime in here to actually verify that this is a bug
>>> then I could send in a pull request with the fix. I will look at the NPE in
>>> de admin GUI this monday so then I will know more on how unorthodox this
>>> fix is.
>>>
>>>
>>>
>>> Op vr 24 jan. 2020 om 10:14 schreef Jan Lievens <
>>> jan.lievens at biggerfish.be>:
>>>
>>>> Before diving into the more technical description of why I came to my
>>>> conclusion a short tl;dr for the impatient: something is probably not
>>>> correct in my schemaHandling part of the Scripted SQL resource with respect
>>>> to the entitlement association. What that is, I have no clue about.
>>>>
>>>> So I did a bit of code spelunking in midPoint to better understand the
>>>> problem. Here is what I came up with:
>>>>
>>>> - To see what happened in the code with respect to the LDAP
>>>> association I put logger com.evolveum.midpoint.provisioning into the debug
>>>> level which provides me with some fine log statement when the association
>>>> (user->group: uniqueMember on the group) is made in LDAP:
>>>> 2020-01-23 08:58:22,737 [PROVISIONING] [midPointScheduler_Worker-2]
>>>> DEBUG (com.evolveum.midpoint.provisioning.impl.ResourceObjectConverter):
>>>> PROVISIONING MODIFY operation on
>>>> resource:d0811790-6420-11e4-86b2-3c9755567874(OpenDJ)
>>>> MODIFY object, object class Technical Role, identified by:
>>>> {
>>>> entryUUID: 88972d7b-6deb-4d94-b343-d039099d23c4
>>>> dn: cn=jirauser,ou=groups,dc=didm,dc=be
>>>> }
>>>> changes: [
>>>> PropertyModificationOperation:
>>>> delta:
>>>> attributes/uniqueMember
>>>> ADD: uid=81071022511,ou=people,dc=didm,dc=be
>>>> matchingRule: {
>>>> http://prism.evolveum.com/xml/ns/public/matching-rule-3}stringIgnoreCase
>>>> ]
>>>> - So basically I had to find out why the inverse is not happening when
>>>> the association is severed (when we delete the entitlement in the
>>>> PostgreSQL database)
>>>> - The conclusion I get from the code is that this should happen in the
>>>> ReconciliationProcessor.reconcileProjectionAssociations method. More
>>>> specifically at the end of that method it is decided to tolerate the
>>>> association or not (tolerant=false on the association is a required thing
>>>> after all 😀).
>>>> - In this decideIfTolerateAssociation method I see this beautiful call
>>>> that is going to solve my world of pain:
>>>> recordAssociationDelta(valueMatcher, accCtx, assocDef,
>>>> ModificationType.DELETE, isCValue.getValue(), null, "it is not given by any
>>>> mapping and the association is not tolerant");
>>>> - In my configuration this statement is not evaluated because it
>>>> requires that there is a PrismContainerValue<ShadowAssociationType>
>>>> provided in the arguments of this method.
>>>> - And why is there no ShadowAssociationType in this case I hear you
>>>> ask? Well because in
>>>> method com.evolveum.midpoint.prism.impl.PrismContainerValueImpl#findItemByQName
>>>> it cannot find an "association" item in the shadow of the PostgreSQL
>>>> account (see screenshot in attachment).
>>>> - And indeed when I look at that specific shadow's repository object
>>>> xml, I see nothing that looks like an association.
>>>> - And why is there no "association" item on the PostgreSQL account
>>>> shadow? That I do not know. Maybe somebody can put me out of my misery?
>>>> - So basically I assume something is fishy about my schemaHandling
>>>> part of the Scripted SQL resource with respect to the entitlement
>>>> association.
>>>>
>>>> Sorry if this is somewhat too technical, I realise there is a dev
>>>> mailing list but I wanted to add it here so that people find the solution
>>>> and problem in the same space.
>>>>
>>>> Op di 21 jan. 2020 om 15:50 schreef Jan Lievens <
>>>> jan.lievens at biggerfish.be>:
>>>>
>>>>> Hi,
>>>>>
>>>>> May I also add that I tried this setup with midPoint versions 4.0.1,
>>>>> 4.0 and 3.9 all with the same result (lingering uniqueMember attributes in
>>>>> LDAP).
>>>>>
>>>>>
>>>>> Op di 21 jan. 2020 om 14:47 schreef Jan Lievens <
>>>>> jan.lievens at biggerfish.be>:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I have reduced the complexity of my use case and made it easy to
>>>>>> reproduce. Starting from the unix story (
>>>>>> https://wiki.evolveum.com/display/midPoint/Unix+Story+Test)
>>>>>> Now I have the following situation:
>>>>>> - Sync accounts from CSV to OpenDJ (works great)
>>>>>> - I have one meta-role that is assigned to a test role which results
>>>>>> in an LDAP group (works great)
>>>>>> - Then I assign this test role to a user which results in a
>>>>>> uniqueMember attribute on the LDAP group (nice)
>>>>>> - Then I unassign the test role from said user and the uniqueMember
>>>>>> attribute on the LDAP group is not getting removed (ðŸ˜)
>>>>>> - The uniqueMember attribute cannot be removed even after running
>>>>>> reconcile/recompute on accounts and entitlements
>>>>>>
>>>>>> More succinct description to reproduce:
>>>>>> - git checkout https://github.com/jalieven/midpoint-docker
>>>>>> - cd midpoint-docker
>>>>>> - git checkout simple-ldap-group
>>>>>> - 'make restart' (starts docker with postgres/opendj/midpoint)
>>>>>> - go to admin GUI http://localhost:8080/midpoint
>>>>>> - login 'administrator' pwd '5ecr3t'
>>>>>> - create a new role 'TestGroup' with an assignment to role 'LDAP
>>>>>> Group Metarole'
>>>>>> [- in OpenDJ: be.didm.group.TestGroup is created] => OK (checking can
>>>>>> be done with 'make check_ldap' in root project)
>>>>>> - in admin GUI: add 'TestGroup' to 'user01' assignments
>>>>>> [- in OpenDJ: the be.didm.group.TestGroup gets the attribute
>>>>>> uniqueMember set to 'uid=user01,ou=people,dc=didm,dc=be'] => OK
>>>>>> - in admin GUI: remove 'TestGroup' from 'user01' assignments (minus
>>>>>> button and SAVE)
>>>>>> [- in OpenDJ: nothing happens: uniqueMember is not removed from
>>>>>> be.didm.group.TestGroup] => Not OK!
>>>>>> - to verify this stale uniqueMember attribute: in root of project do
>>>>>> 'make check_ldap'
>>>>>>
>>>>>> Config files can be found in
>>>>>> demo/simple/midpoint_server/container_files/mp-home/post-initial-objects
>>>>>>
>>>>>> As you can see reproducing this bug/miss-config takes 5min of work.
>>>>>> If this is a miss-configuration you might want to check because the same is
>>>>>> happening in the unix story test. If this is a bug in midPoint all the more
>>>>>> so to look at it. It is hard for me as a novice to say which one it is.
>>>>>>
>>>>>> It seems to me that this is a common use-case. I have come
>>>>>> across this issue a few times in this mailing list, none of which provide a
>>>>>> sufficient solution:
>>>>>> http://lists.evolveum.com/pipermail/midpoint/2016-April/001720.html
>>>>>> (the 'it-works-on-my-machine'-stance is kind of lame in this case and it
>>>>>> doesn't help that the referred zip files does not resolve)
>>>>>> http://lists.evolveum.com/pipermail/midpoint/2016-November/002807.html
>>>>>> (a lot of tolerant/strength stuff here)
>>>>>> http://lists.evolveum.com/pipermail/midpoint/2016-November/002843.html
>>>>>> (suggests association shortcuts and multivalued schema fix in scripted sql
>>>>>> connector, the latter of which does not apply in my described use-case)
>>>>>>
>>>>>> The midPoint Confluence suggests also to set various packages to
>>>>>> trace log level to debug what is going on but I must say that in so much
>>>>>> logging I have no idea what to look for.
>>>>>> When I set the projector/clockwork to debug all I see is that the
>>>>>> remove of the association is not happening. More specifically this package
>>>>>> on trace org.identityconnectors.framework.spi.operations points to the fact
>>>>>> that the delete is not happening in the connector.
>>>>>>
>>>>>> May I also add that the page
>>>>>> https://wiki.evolveum.com/display/midPoint/Initial+Logging+Setup+HOWTO is
>>>>>> not up to date. Setting the skipRepositoryLoggingSettings does not work. So
>>>>>> there is currently no way to add specific loggers at start up through the
>>>>>> logback(-extra).xml. It always gets overwritten by what is inside the
>>>>>> repository.
>>>>>> It appears that the skipRepositoryLoggingSettings was removed from
>>>>>> the code some time ago.
>>>>>>
>>>>>> Kind Regards,
>>>>>> Jan Lievens
>>>>>>
>>>>>> Op wo 15 jan. 2020 om 16:19 schreef Jan Lievens <
>>>>>> jan.lievens at biggerfish.be>:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I have a question related to inducement construction on a role
>>>>>>> toward an LDAP resource.
>>>>>>> I have the following situation in midPoint (see the attachment for
>>>>>>> all the xml files)
>>>>>>> - Accounts and Entitlements (intent: privileges) are imported from a
>>>>>>> ScriptedSQL connector.
>>>>>>> - These imported entitlements have multiple privileges which are
>>>>>>> created as roles with an "assignmentTargetSearch"
>>>>>>> (post-initial-objects/210-entitlement-object-template.xml)
>>>>>>> - I also have these privilege/roles defined with an assignment to
>>>>>>> (multiple) fixed technical roles (kind:entitlement/intent:group).
>>>>>>> - Lastly I would like for these technical roles (groups) and the
>>>>>>> associated accounts to get synced to LDAP in an objectToSubject fashion. I
>>>>>>> do this with inducements and construction tags in
>>>>>>> post-initial-objects/100-profiles-webidm.xml.
>>>>>>> - The result I get in LDAP is that accounts are synced correctly but
>>>>>>> the group names are not what I expect (I expect technical role names
>>>>>>> eg. JiraUser) but get the names of the Entitlements (intent:privileges)
>>>>>>> defined in the DB.
>>>>>>> - Additionally the associations are not synced (no uniqueMember refs
>>>>>>> are synced on the LDAP groups).
>>>>>>>
>>>>>>> Surely I am missing something here. I think this use case is quiet
>>>>>>> standard (a hierarchy of roles and only the leafs get synced to LDAP).
>>>>>>> I have experimented with the order of the inducement but these came
>>>>>>> out even more negative (no accounts or groups were created).
>>>>>>> I also have experimented with the tolerant setting on the
>>>>>>> association (since I find a lot of answers in this mailing list suggesting
>>>>>>> this) but to no avail.
>>>>>>> It is quiet frustrating to be so close to having this use-case
>>>>>>> implemented DB->mP->LDAP but failing in the sync to LDAP.
>>>>>>>
>>>>>>> Kind regards,
>>>>>>> --
>>>>>>> Jan Lievens
>>>>>>> IT Consultant
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Jan Lievens
>>>>>> IT Consultant
>>>>>> Bigger Fish
>>>>>> Hoogstraat 35
>>>>>> 9450 Haaltert
>>>>>> Belgium
>>>>>> VAT: BE 0734.993.447
>>>>>> Mob: +32 494 84 66 97
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Jan Lievens
>>>>> IT Consultant
>>>>> Bigger Fish
>>>>> Hoogstraat 35
>>>>> 9450 Haaltert
>>>>> Belgium
>>>>> VAT: BE 0734.993.447
>>>>> Mob: +32 494 84 66 97
>>>>>
>>>>
>>>>
>>>> --
>>>> Jan Lievens
>>>> IT Consultant
>>>> Bigger Fish
>>>> Hoogstraat 35
>>>> 9450 Haaltert
>>>> Belgium
>>>> VAT: BE 0734.993.447
>>>> Mob: +32 494 84 66 97
>>>>
>>>
>>>
>>> --
>>> Jan Lievens
>>> IT Consultant
>>> Bigger Fish
>>> Hoogstraat 35
>>> 9450 Haaltert
>>> Belgium
>>> VAT: BE 0734.993.447
>>> Mob: +32 494 84 66 97
>>>
>>
>>
>> --
>> Jan Lievens
>> IT Consultant
>> Bigger Fish
>> Hoogstraat 35
>> 9450 Haaltert
>> Belgium
>> VAT: BE 0734.993.447
>> Mob: +32 494 84 66 97
>>
>
>
> --
> Jan Lievens
> IT Consultant
> Bigger Fish
> Hoogstraat 35
> 9450 Haaltert
> Belgium
> VAT: BE 0734.993.447
> Mob: +32 494 84 66 97
>
--
Jan Lievens
IT Consultant
Bigger Fish
Hoogstraat 35
9450 Haaltert
Belgium
VAT: BE 0734.993.447
Mob: +32 494 84 66 97
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20200128/df09bd7e/attachment.htm>
More information about the midPoint
mailing list