[midPoint] LDAP role/group inducement
Jan Lievens
jan.lievens at biggerfish.be
Tue Jan 28 08:40:57 CET 2020
So now my conclusion is that the NPE in the admin points to the fact that
together with name and identifiers there should also be a shadowRef saved
in the shadow (PostgreSQL account) repo.
We see that only the name and identifiers are saved in the beginning (first
import task). I suppose the shadowRef is at that point not resolvable
(perhaps the entitlements are not imported yet so the shadow of an
entitlement is not yet available).
Now we also see that if we perform a null safe check on the NPE site our
use-case works (uniqueMember is cleared of the LDAP group when we remove
entitlement from PostgreSQL) but we see that the shadow of the entitlement
is lingering in the repo (and indeed when we again add the entitlement
again in PostgreSQL an exception is thrown saying that there are
conflicting shadows in the import task run).
So my hunch is that the shadowRef should be saved on the association when
possible. I see that during the reconciliation the shardowRef is correctly
added beside the name and identifiers tags (but alas never saved to the
repo).
I will pursue this line of thinking. My best guess is to save this
shadowRef in the association someplace in the ShadowManager.
Op ma 27 jan. 2020 om 15:15 schreef Jan Lievens <jan.lievens at biggerfish.be>:
> So I figured out what the NullPointerException is all about: The code at
> line ShadowAssociationWrapperFactoryImpl.java:193
> "shadowAss.add(associationValue.findReference(ShadowAssociationType.F_SHADOW_REF).getValue().clone());"
> expects to have a shadowRef in the association which it does not have (it
> only has name and identifiers tags, like can be seen from my previous
> post). Now I would say that this missing shadowRef needs to be fixed next
> because when I null check this (and thus this line is not performed) the
> lingering-"uniqueMember" bug in LDAP is resolved and the GUI admin does not
> crash with a NPE but the shadow (the shadow of the entitlement that is, for
> which the shadowRef should exist) is not cleaned up and can still be seen
> in the repo. So when I again add the entitlement in PostgreSQL and perform
> an import task on that resource it crashes on the fact that it sees
> conflicting shadows (no surprise there).
>
> Op za 25 jan. 2020 om 15:20 schreef Jan Lievens <jan.lievens at biggerfish.be
> >:
>
>> So I debugged this problem with the lacking "association" item in the
>> account shadow of the PostgreSQL resource. So again a tl;dr for the people
>> with no time to follow my lengthy explanation: the lingering uniqueMember
>> reference gets cleaned up in LDAP when I remove
>> line com.evolveum.midpoint.repo.sql.helpers.ObjectRetriever.java:557
>> "prismObject.removeContainer(ShadowType.F_ASSOCIATION);"
>>
>> So what I see is that the shadow that gets written (xml in fullObject
>> column) the first time into the repo and it contains the "association"
>> item.
>>
>> <shadow xmlns="
>> http://midpoint.evolveum.com/xml/ns/public/common/common-3" xmlns:c="
>> http://midpoint.evolveum.com/xml/ns/public/common/common-3" xmlns:icfs="
>> http://midpoint.evolveum.com/xml/ns/public/connector/icf-1/resource-schema-3"
>> xmlns:org="http://midpoint.evolveum.com/xml/ns/public/common/org-3"
>> xmlns:q="http://prism.evolveum.com/xml/ns/public/query-3" xmlns:ri="
>> http://midpoint.evolveum.com/xml/ns/public/resource/instance-3" xmlns:t="
>> http://prism.evolveum.com/xml/ns/public/types-3"
>> oid="954b8cfe-5c84-4f82-946f-0196dcec92c0" version="0">
>> <name>810710333333</name>
>> <resourceRef
>> oid="dafe0482-faef-47b7-ae08-66b150929bb7" relation="org:default"
>> type="c:ResourceType">
>> <!-- Scripted SQL -->
>> </resourceRef>
>> <objectClass>ri:AccountObjectClass</objectClass>
>>
>> <primaryIdentifierValue>3fd83cd4-d1bb-4d7f-9a1a-12a02ed85a95</primaryIdentifierValue>
>> <kind>account</kind>
>> <exists>true</exists>
>> <attributes>
>> <icfs:name>810710333333</icfs:name>
>>
>> <icfs:uid>3fd83cd4-d1bb-4d7f-9a1a-12a02ed85a95</icfs:uid>
>> </attributes>
>> <association id="1">
>> <name>ri:ents</name>
>> <identifiers>
>>
>> <icfs:name>20001-Milieumedewerker-01</icfs:name>
>> </identifiers>
>> </association>
>> <activation/>
>> </shadow>
>>
>> It is afterwards (in a following reconciliation wave) that this gets
>> overwritten with a version that has no association. Poof gone...
>>
>> <shadow xmlns="
>> http://midpoint.evolveum.com/xml/ns/public/common/common-3" xmlns:c="
>> http://midpoint.evolveum.com/xml/ns/public/common/common-3" xmlns:icfs="
>> http://midpoint.evolveum.com/xml/ns/public/connector/icf-1/resource-schema-3"
>> xmlns:org="http://midpoint.evolveum.com/xml/ns/public/common/org-3"
>> xmlns:q="http://prism.evolveum.com/xml/ns/public/query-3" xmlns:ri="
>> http://midpoint.evolveum.com/xml/ns/public/resource/instance-3" xmlns:t="
>> http://prism.evolveum.com/xml/ns/public/types-3"
>> oid="954b8cfe-5c84-4f82-946f-0196dcec92c0" version="1">
>> <name>810710333333</name>
>> <resourceRef
>> oid="dafe0482-faef-47b7-ae08-66b150929bb7" relation="org:default"
>> type="c:ResourceType"/>
>>
>> <synchronizationTimestamp>2020-01-24T13:30:38.108Z</synchronizationTimestamp>
>>
>> <fullSynchronizationTimestamp>2020-01-24T13:30:38.108Z</fullSynchronizationTimestamp>
>> <objectClass>ri:AccountObjectClass</objectClass>
>>
>> <primaryIdentifierValue>3fd83cd4-d1bb-4d7f-9a1a-12a02ed85a95</primaryIdentifierValue>
>> <kind>account</kind>
>> <intent>webidm</intent>
>> <exists>true</exists>
>> <attributes xmlns:xsi="
>> http://www.w3.org/2001/XMLSchema-instance" xmlns:c="
>> http://midpoint.evolveum.com/xml/ns/public/common/common-3"
>> xsi:type="c:ShadowAttributesType">
>> <icfs:name>810710333333</icfs:name>
>>
>> <icfs:uid>3fd83cd4-d1bb-4d7f-9a1a-12a02ed85a95</icfs:uid>
>> </attributes>
>> <activation/>
>> </shadow>
>>
>> Immediately we see that the following modifications made to the just
>> created shadow: REPLACE intent, kind, synchronizationTimestamp and
>> fullSynchronizationTimestamp. It is in this replace that the
>> "association"-part is removed and written into the repo. It is on position
>> com.evolveum.midpoint.repo.sql.helpers.ObjectRetriever.java:557 that the
>> association from the shadow gets removed in a very explicit fashion with
>> the ominous comment: "we store it because provisioning now sends it to
>> repo, but it should be transient".
>> When we remove this line that removes the association we get a fat NPE in
>> the admin tool when opening the user, but our bug is fixed 😎.
>> But somehow I can't shake the feeling that removing this line is hacky (I
>> suppose other code might also depend on this line being there) and that the
>> association should get fetched from the inbound source (i.e. PostgreSQL) at
>> the right moment. But again I'm too novice to really know how to configure
>> that. Something of the likes of
>> <searchStrategy>onResourceIfNeeded</searchStrategy> maybe?
>> If someone could shime in here to actually verify that this is a bug then
>> I could send in a pull request with the fix. I will look at the NPE in de
>> admin GUI this monday so then I will know more on how unorthodox this fix
>> is.
>>
>>
>>
>> Op vr 24 jan. 2020 om 10:14 schreef Jan Lievens <
>> jan.lievens at biggerfish.be>:
>>
>>> Before diving into the more technical description of why I came to my
>>> conclusion a short tl;dr for the impatient: something is probably not
>>> correct in my schemaHandling part of the Scripted SQL resource with respect
>>> to the entitlement association. What that is, I have no clue about.
>>>
>>> So I did a bit of code spelunking in midPoint to better understand the
>>> problem. Here is what I came up with:
>>>
>>> - To see what happened in the code with respect to the LDAP association
>>> I put logger com.evolveum.midpoint.provisioning into the debug level which
>>> provides me with some fine log statement when the association (user->group:
>>> uniqueMember on the group) is made in LDAP:
>>> 2020-01-23 08:58:22,737 [PROVISIONING] [midPointScheduler_Worker-2]
>>> DEBUG (com.evolveum.midpoint.provisioning.impl.ResourceObjectConverter):
>>> PROVISIONING MODIFY operation on
>>> resource:d0811790-6420-11e4-86b2-3c9755567874(OpenDJ)
>>> MODIFY object, object class Technical Role, identified by:
>>> {
>>> entryUUID: 88972d7b-6deb-4d94-b343-d039099d23c4
>>> dn: cn=jirauser,ou=groups,dc=didm,dc=be
>>> }
>>> changes: [
>>> PropertyModificationOperation:
>>> delta:
>>> attributes/uniqueMember
>>> ADD: uid=81071022511,ou=people,dc=didm,dc=be
>>> matchingRule: {
>>> http://prism.evolveum.com/xml/ns/public/matching-rule-3}stringIgnoreCase
>>> ]
>>> - So basically I had to find out why the inverse is not happening when
>>> the association is severed (when we delete the entitlement in the
>>> PostgreSQL database)
>>> - The conclusion I get from the code is that this should happen in the
>>> ReconciliationProcessor.reconcileProjectionAssociations method. More
>>> specifically at the end of that method it is decided to tolerate the
>>> association or not (tolerant=false on the association is a required thing
>>> after all 😀).
>>> - In this decideIfTolerateAssociation method I see this beautiful call
>>> that is going to solve my world of pain:
>>> recordAssociationDelta(valueMatcher, accCtx, assocDef,
>>> ModificationType.DELETE, isCValue.getValue(), null, "it is not given by any
>>> mapping and the association is not tolerant");
>>> - In my configuration this statement is not evaluated because it
>>> requires that there is a PrismContainerValue<ShadowAssociationType>
>>> provided in the arguments of this method.
>>> - And why is there no ShadowAssociationType in this case I hear you
>>> ask? Well because in
>>> method com.evolveum.midpoint.prism.impl.PrismContainerValueImpl#findItemByQName
>>> it cannot find an "association" item in the shadow of the PostgreSQL
>>> account (see screenshot in attachment).
>>> - And indeed when I look at that specific shadow's repository object
>>> xml, I see nothing that looks like an association.
>>> - And why is there no "association" item on the PostgreSQL account
>>> shadow? That I do not know. Maybe somebody can put me out of my misery?
>>> - So basically I assume something is fishy about my schemaHandling part
>>> of the Scripted SQL resource with respect to the entitlement association.
>>>
>>> Sorry if this is somewhat too technical, I realise there is a dev
>>> mailing list but I wanted to add it here so that people find the solution
>>> and problem in the same space.
>>>
>>> Op di 21 jan. 2020 om 15:50 schreef Jan Lievens <
>>> jan.lievens at biggerfish.be>:
>>>
>>>> Hi,
>>>>
>>>> May I also add that I tried this setup with midPoint versions 4.0.1,
>>>> 4.0 and 3.9 all with the same result (lingering uniqueMember attributes in
>>>> LDAP).
>>>>
>>>>
>>>> Op di 21 jan. 2020 om 14:47 schreef Jan Lievens <
>>>> jan.lievens at biggerfish.be>:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I have reduced the complexity of my use case and made it easy to
>>>>> reproduce. Starting from the unix story (
>>>>> https://wiki.evolveum.com/display/midPoint/Unix+Story+Test)
>>>>> Now I have the following situation:
>>>>> - Sync accounts from CSV to OpenDJ (works great)
>>>>> - I have one meta-role that is assigned to a test role which results
>>>>> in an LDAP group (works great)
>>>>> - Then I assign this test role to a user which results in a
>>>>> uniqueMember attribute on the LDAP group (nice)
>>>>> - Then I unassign the test role from said user and the uniqueMember
>>>>> attribute on the LDAP group is not getting removed (ðŸ˜)
>>>>> - The uniqueMember attribute cannot be removed even after running
>>>>> reconcile/recompute on accounts and entitlements
>>>>>
>>>>> More succinct description to reproduce:
>>>>> - git checkout https://github.com/jalieven/midpoint-docker
>>>>> - cd midpoint-docker
>>>>> - git checkout simple-ldap-group
>>>>> - 'make restart' (starts docker with postgres/opendj/midpoint)
>>>>> - go to admin GUI http://localhost:8080/midpoint
>>>>> - login 'administrator' pwd '5ecr3t'
>>>>> - create a new role 'TestGroup' with an assignment to role 'LDAP Group
>>>>> Metarole'
>>>>> [- in OpenDJ: be.didm.group.TestGroup is created] => OK (checking can
>>>>> be done with 'make check_ldap' in root project)
>>>>> - in admin GUI: add 'TestGroup' to 'user01' assignments
>>>>> [- in OpenDJ: the be.didm.group.TestGroup gets the attribute
>>>>> uniqueMember set to 'uid=user01,ou=people,dc=didm,dc=be'] => OK
>>>>> - in admin GUI: remove 'TestGroup' from 'user01' assignments (minus
>>>>> button and SAVE)
>>>>> [- in OpenDJ: nothing happens: uniqueMember is not removed from
>>>>> be.didm.group.TestGroup] => Not OK!
>>>>> - to verify this stale uniqueMember attribute: in root of project do
>>>>> 'make check_ldap'
>>>>>
>>>>> Config files can be found in
>>>>> demo/simple/midpoint_server/container_files/mp-home/post-initial-objects
>>>>>
>>>>> As you can see reproducing this bug/miss-config takes 5min of work. If
>>>>> this is a miss-configuration you might want to check because the same is
>>>>> happening in the unix story test. If this is a bug in midPoint all the more
>>>>> so to look at it. It is hard for me as a novice to say which one it is.
>>>>>
>>>>> It seems to me that this is a common use-case. I have come across this
>>>>> issue a few times in this mailing list, none of which provide a sufficient
>>>>> solution:
>>>>> http://lists.evolveum.com/pipermail/midpoint/2016-April/001720.html
>>>>> (the 'it-works-on-my-machine'-stance is kind of lame in this case and it
>>>>> doesn't help that the referred zip files does not resolve)
>>>>> http://lists.evolveum.com/pipermail/midpoint/2016-November/002807.html
>>>>> (a lot of tolerant/strength stuff here)
>>>>> http://lists.evolveum.com/pipermail/midpoint/2016-November/002843.html
>>>>> (suggests association shortcuts and multivalued schema fix in scripted sql
>>>>> connector, the latter of which does not apply in my described use-case)
>>>>>
>>>>> The midPoint Confluence suggests also to set various packages to trace
>>>>> log level to debug what is going on but I must say that in so much logging
>>>>> I have no idea what to look for.
>>>>> When I set the projector/clockwork to debug all I see is that the
>>>>> remove of the association is not happening. More specifically this package
>>>>> on trace org.identityconnectors.framework.spi.operations points to the fact
>>>>> that the delete is not happening in the connector.
>>>>>
>>>>> May I also add that the page
>>>>> https://wiki.evolveum.com/display/midPoint/Initial+Logging+Setup+HOWTO is
>>>>> not up to date. Setting the skipRepositoryLoggingSettings does not work. So
>>>>> there is currently no way to add specific loggers at start up through the
>>>>> logback(-extra).xml. It always gets overwritten by what is inside the
>>>>> repository.
>>>>> It appears that the skipRepositoryLoggingSettings was removed from the
>>>>> code some time ago.
>>>>>
>>>>> Kind Regards,
>>>>> Jan Lievens
>>>>>
>>>>> Op wo 15 jan. 2020 om 16:19 schreef Jan Lievens <
>>>>> jan.lievens at biggerfish.be>:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I have a question related to inducement construction on a role toward
>>>>>> an LDAP resource.
>>>>>> I have the following situation in midPoint (see the attachment for
>>>>>> all the xml files)
>>>>>> - Accounts and Entitlements (intent: privileges) are imported from a
>>>>>> ScriptedSQL connector.
>>>>>> - These imported entitlements have multiple privileges which are
>>>>>> created as roles with an "assignmentTargetSearch"
>>>>>> (post-initial-objects/210-entitlement-object-template.xml)
>>>>>> - I also have these privilege/roles defined with an assignment to
>>>>>> (multiple) fixed technical roles (kind:entitlement/intent:group).
>>>>>> - Lastly I would like for these technical roles (groups) and the
>>>>>> associated accounts to get synced to LDAP in an objectToSubject fashion. I
>>>>>> do this with inducements and construction tags in
>>>>>> post-initial-objects/100-profiles-webidm.xml.
>>>>>> - The result I get in LDAP is that accounts are synced correctly but
>>>>>> the group names are not what I expect (I expect technical role names
>>>>>> eg. JiraUser) but get the names of the Entitlements (intent:privileges)
>>>>>> defined in the DB.
>>>>>> - Additionally the associations are not synced (no uniqueMember refs
>>>>>> are synced on the LDAP groups).
>>>>>>
>>>>>> Surely I am missing something here. I think this use case is quiet
>>>>>> standard (a hierarchy of roles and only the leafs get synced to LDAP).
>>>>>> I have experimented with the order of the inducement but these came
>>>>>> out even more negative (no accounts or groups were created).
>>>>>> I also have experimented with the tolerant setting on the association
>>>>>> (since I find a lot of answers in this mailing list suggesting this) but to
>>>>>> no avail.
>>>>>> It is quiet frustrating to be so close to having this use-case
>>>>>> implemented DB->mP->LDAP but failing in the sync to LDAP.
>>>>>>
>>>>>> Kind regards,
>>>>>> --
>>>>>> Jan Lievens
>>>>>> IT Consultant
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Jan Lievens
>>>>> IT Consultant
>>>>> Bigger Fish
>>>>> Hoogstraat 35
>>>>> 9450 Haaltert
>>>>> Belgium
>>>>> VAT: BE 0734.993.447
>>>>> Mob: +32 494 84 66 97
>>>>>
>>>>
>>>>
>>>> --
>>>> Jan Lievens
>>>> IT Consultant
>>>> Bigger Fish
>>>> Hoogstraat 35
>>>> 9450 Haaltert
>>>> Belgium
>>>> VAT: BE 0734.993.447
>>>> Mob: +32 494 84 66 97
>>>>
>>>
>>>
>>> --
>>> Jan Lievens
>>> IT Consultant
>>> Bigger Fish
>>> Hoogstraat 35
>>> 9450 Haaltert
>>> Belgium
>>> VAT: BE 0734.993.447
>>> Mob: +32 494 84 66 97
>>>
>>
>>
>> --
>> Jan Lievens
>> IT Consultant
>> Bigger Fish
>> Hoogstraat 35
>> 9450 Haaltert
>> Belgium
>> VAT: BE 0734.993.447
>> Mob: +32 494 84 66 97
>>
>
>
> --
> Jan Lievens
> IT Consultant
> Bigger Fish
> Hoogstraat 35
> 9450 Haaltert
> Belgium
> VAT: BE 0734.993.447
> Mob: +32 494 84 66 97
>
--
Jan Lievens
IT Consultant
Bigger Fish
Hoogstraat 35
9450 Haaltert
Belgium
VAT: BE 0734.993.447
Mob: +32 494 84 66 97
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20200128/bbf767a1/attachment.htm>
More information about the midPoint
mailing list