<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hello Jan,</p>
<p>you attempted a really deep dive into midPoint code. You have my
respect. Although I am not able to give you a definite answer
because my time is very constrained, let me say a few notes:</p>
<p>
<blockquote type="cite">
<div> - The conclusion I get from the code is that this should
happen in the
ReconciliationProcessor.reconcileProjectionAssociations
method. More specifically at the end of that method it is
decided to tolerate the association or not (tolerant=false on
the association is a required thing after all 😀).</div>
<div> - In this decideIfTolerateAssociation method I see this
beautiful call that is going to solve my world of pain:</div>
<div>recordAssociationDelta(valueMatcher, accCtx, assocDef,
ModificationType.DELETE, isCValue.getValue(), null, "it is not
given by any mapping and the association is not tolerant");<br>
</div>
</blockquote>
</p>
<p>MidPoint has two ways how it decides whether to make a change (in
your case, deleting an association or not):</p>
<ol>
<li>It computes the change (we call it "delta") directly - e.g.
from some mapping.</li>
<li>Or it computes a required object state and - by comparing it
with the current state of the object - it computes the necessary
changes ("deltas").</li>
</ol>
<p>The latter case requires the setting of tolerant=false. The
former one should apply regardless of the tolerant flag setting.</p>
<p>This is the general approach as I see it. I think that in your
case both ways should work - midPoint should know that it has to
create the delta ("1") and, if for some reason it knows that not,
it should compute that the association value is to be deleted
("2").</p>
<p>Why this does not work ... this requires more deep analysis.</p>
<p>But let me continue with the notes:</p>
<p>
<blockquote type="cite">
<div> - And why is there no ShadowAssociationType in this case I
hear you ask? Well because in
method com.evolveum.midpoint.prism.impl.PrismContainerValueImpl#findItemByQName
it cannot find an "association" item in the shadow of the
PostgreSQL account (see screenshot in attachment).</div>
<div> - And indeed when I look at that specific shadow's
repository object xml, I see nothing that looks like an
association.</div>
<div> - And why is there no "association" item on the PostgreSQL
account shadow? That I do not know. Maybe somebody can put me
out of my misery?</div>
</blockquote>
This is quite simple. :) Unless so called attribute caching (an
experimental feature) is enabled, midPoint does not store
attributes (and associations, which are basically the same as
attributes) in the repository. The exception are identifiers. And
I think that associations are not stored in repo even with
attribute caching enabled.<br>
</p>
<p>But your use case should work even without the caching.</p>
<p>From your second mail:</p>
<p>
<blockquote type="cite">So I debugged this problem with the
lacking "association" item in the account shadow of the
PostgreSQL resource. So again a tl;dr for the people with no
time to follow my lengthy explanation: the lingering
uniqueMember reference gets cleaned up in LDAP when I remove
line com.evolveum.midpoint.repo.sql.helpers.ObjectRetriever.java:557
"prismObject.removeContainer(ShadowType.F_ASSOCIATION);"<br>
</blockquote>
This is an interesting attempt, but definitely not the way to go
(stems from what I wrote above). <br>
</p>
<p>
<blockquote type="cite">But somehow I can't shake the feeling that
removing this line is hacky (I suppose other code might also
depend on this line being there) and that the association should
get fetched from the inbound source (i.e. PostgreSQL) at the
right moment.</blockquote>
Exactly!<br>
<blockquote type="cite"> But again I'm too novice to really know
how to configure that. Something of the likes of
<searchStrategy>onResourceIfNeeded</searchStrategy>
maybe?</blockquote>
I am not sure without looking deeper.</p>
<p>
<blockquote type="cite">So my hunch is that the shadowRef should
be saved on the association when possible. I see that during the
reconciliation the shardowRef is correctly added beside the name
and identifiers tags (but alas never saved to the repo).<br>
I will pursue this line of thinking. My best guess is to save
this shadowRef in the association someplace in the
ShadowManager.</blockquote>
Again, I would strongly advise against going this way. I am
writing this to spare your time and effort.</p>
<p>To wrap all this up: Although I really sympathize with your
efforts and devotion, I cannot help you much besides this.
Generally, I think that midPoint should know it has to delete the
association when the assignment goes away (case "1" in my list at
the beginning). And if not, it should somehow determine this from
the current state of the object. The information about the
association should not definitely come from the repository. It
must come from the resource (i.e. from provisioning module). <br>
</p>
<p>I would strongly recommend you to obtain professional services
from Evolveum. E.g. now I have to finish writing this and devote
my time to clients that have bought such services, as they have
equally urgent issues... :-)<br>
</p>
<p>Best regards,<br>
</p>
<pre class="moz-signature" cols="72">Pavol Mederly
Software developer
evolveum.com
</pre>
<div class="moz-cite-prefix">On 24/01/2020 10:14, Jan Lievens wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CALKj1oaOvSODA893uAi3cLQhb+gigHHQ0xg4=0ru+muYEWOQ5w@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div>Before diving into the more technical description of why I
came to my conclusion a short tl;dr for the impatient:
something is probably not correct in my schemaHandling part of
the Scripted SQL resource with respect to the entitlement
association. What that is, I have no clue about.</div>
<div><br>
</div>
<div>So I did a bit of code spelunking in midPoint to better
understand the problem. Here is what I came up with:</div>
<div><br>
</div>
<div> - To see what happened in the code with respect to the
LDAP association I put logger
com.evolveum.midpoint.provisioning into the debug level which
provides me with some fine log statement when the association
(user->group: uniqueMember on the group) is made in LDAP:</div>
<div>2020-01-23 08:58:22,737 [PROVISIONING]
[midPointScheduler_Worker-2] DEBUG
(com.evolveum.midpoint.provisioning.impl.ResourceObjectConverter):
PROVISIONING MODIFY operation on
<a class="moz-txt-link-freetext" href="resource:d0811790-6420-11e4-86b2-3c9755567874(OpenDJ)">resource:d0811790-6420-11e4-86b2-3c9755567874(OpenDJ)</a><br>
MODIFY object, object class Technical Role, identified by:<br>
{<br>
entryUUID: 88972d7b-6deb-4d94-b343-d039099d23c4<br>
dn: cn=jirauser,ou=groups,dc=didm,dc=be<br>
}<br>
changes: [<br>
PropertyModificationOperation:<br>
delta:<br>
attributes/uniqueMember<br>
ADD: uid=81071022511,ou=people,dc=didm,dc=be<br>
matchingRule: {<a
href="http://prism.evolveum.com/xml/ns/public/matching-rule-3%7DstringIgnoreCase"
target="_blank" moz-do-not-send="true">http://prism.evolveum.com/xml/ns/public/matching-rule-3}stringIgnoreCase</a><br>
]<br>
</div>
<div> - So basically I had to find out why the inverse is not
happening when the association is severed (when we delete the
entitlement in the PostgreSQL database)</div>
<div> - The conclusion I get from the code is that this should
happen in the
ReconciliationProcessor.reconcileProjectionAssociations
method. More specifically at the end of that method it is
decided to tolerate the association or not (tolerant=false on
the association is a required thing after all 😀).</div>
<div> - In this decideIfTolerateAssociation method I see this
beautiful call that is going to solve my world of pain:</div>
<div>recordAssociationDelta(valueMatcher, accCtx, assocDef,
ModificationType.DELETE, isCValue.getValue(), null, "it is not
given by any mapping and the association is not tolerant");<br>
</div>
<div> - In my configuration this statement is not evaluated
because it requires that there is a
PrismContainerValue<ShadowAssociationType> provided in
the arguments of this method.</div>
<div> - And why is there no ShadowAssociationType in this case I
hear you ask? Well because in
method com.evolveum.midpoint.prism.impl.PrismContainerValueImpl#findItemByQName
it cannot find an "association" item in the shadow of the
PostgreSQL account (see screenshot in attachment).</div>
<div> - And indeed when I look at that specific shadow's
repository object xml, I see nothing that looks like an
association.</div>
<div> - And why is there no "association" item on the PostgreSQL
account shadow? That I do not know. Maybe somebody can put me
out of my misery?</div>
<div> - So basically I assume something is fishy about my
schemaHandling part of the Scripted SQL resource with respect
to the entitlement association.</div>
<div><br>
</div>
<div>Sorry if this is somewhat too technical, I realise there is
a dev mailing list but I wanted to add it here so that people
find the solution and problem in the same space.</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">Op di 21 jan. 2020 om 15:50
schreef Jan Lievens <<a
href="mailto:jan.lievens@biggerfish.be"
moz-do-not-send="true">jan.lievens@biggerfish.be</a>>:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Hi,
<div><br>
</div>
<div>May I also add that I tried this setup with midPoint
versions 4.0.1, 4.0 and 3.9 all with the same result
(lingering uniqueMember attributes in LDAP).</div>
<div><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">Op di 21 jan. 2020 om
14:47 schreef Jan Lievens <<a
href="mailto:jan.lievens@biggerfish.be" target="_blank"
moz-do-not-send="true">jan.lievens@biggerfish.be</a>>:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Hi all,
<div><br>
</div>
<div>I have reduced the complexity of my use case and
made it easy to reproduce. Starting from the unix
story (<a
href="https://wiki.evolveum.com/display/midPoint/Unix+Story+Test"
target="_blank" moz-do-not-send="true">https://wiki.evolveum.com/display/midPoint/Unix+Story+Test</a>)</div>
<div>Now I have the following situation:</div>
<div> - Sync accounts from CSV to OpenDJ (works great)</div>
<div> - I have one meta-role that is assigned to a test
role which results in an LDAP group (works great)</div>
<div> - Then I assign this test role to a user which
results in a uniqueMember attribute on the LDAP group
(nice)</div>
<div> - Then I unassign the test role from said user and
the uniqueMember attribute on the LDAP group is not
getting removed (😭)</div>
<div> - The uniqueMember attribute cannot be removed
even after running reconcile/recompute on accounts and
entitlements</div>
<div><br>
</div>
<div>More succinct description to reproduce:</div>
<div>- git checkout <a
href="https://github.com/jalieven/midpoint-docker"
target="_blank" moz-do-not-send="true">https://github.com/jalieven/midpoint-docker</a><br>
- cd midpoint-docker<br>
- git checkout simple-ldap-group<br>
- 'make restart' (starts docker with
postgres/opendj/midpoint)<br>
- go to admin GUI <a
href="http://localhost:8080/midpoint"
target="_blank" moz-do-not-send="true">http://localhost:8080/midpoint</a>
<br>
- login 'administrator' pwd '5ecr3t'<br>
- create a new role 'TestGroup' with an assignment to
role 'LDAP Group Metarole'<br>
[- in OpenDJ: be.didm.group.TestGroup is created]
=> OK (checking can be done with 'make check_ldap'
in root project) <br>
- in admin GUI: add 'TestGroup' to 'user01'
assignments<br>
[- in OpenDJ: the be.didm.group.TestGroup gets the
attribute uniqueMember set to
'uid=user01,ou=people,dc=didm,dc=be'] => OK<br>
- in admin GUI: remove 'TestGroup' from 'user01'
assignments (minus button and SAVE)<br>
[- in OpenDJ: nothing happens: uniqueMember is not
removed from be.didm.group.TestGroup] => Not OK!</div>
<div>- to verify this stale uniqueMember attribute: in
root of project do 'make check_ldap'<br>
<br>
Config files can be found in
demo/simple/midpoint_server/container_files/mp-home/post-initial-objects<br>
</div>
<div><br>
</div>
<div>As you can see reproducing this bug/miss-config
takes 5min of work. If this is a miss-configuration
you might want to check because the same is happening
in the unix story test. If this is a bug in midPoint
all the more so to look at it. It is hard for me as a
novice to say which one it is. </div>
<div><br>
</div>
<div>
<div>It seems to me that this is a common use-case. I
have come across this issue a few times in this
mailing list, none of which provide a sufficient
solution:</div>
<div><a
href="http://lists.evolveum.com/pipermail/midpoint/2016-April/001720.html"
target="_blank" moz-do-not-send="true">http://lists.evolveum.com/pipermail/midpoint/2016-April/001720.html</a>
(the 'it-works-on-my-machine'-stance is kind of lame
in this case and it doesn't help that the referred
zip files does not resolve)<br>
</div>
<div><a
href="http://lists.evolveum.com/pipermail/midpoint/2016-November/002807.html"
target="_blank" moz-do-not-send="true">http://lists.evolveum.com/pipermail/midpoint/2016-November/002807.html</a>
(a lot of tolerant/strength stuff here)<br>
</div>
<div><a
href="http://lists.evolveum.com/pipermail/midpoint/2016-November/002843.html"
target="_blank" moz-do-not-send="true">http://lists.evolveum.com/pipermail/midpoint/2016-November/002843.html</a>
(suggests association shortcuts and multivalued
schema fix in scripted sql connector, the latter of
which does not apply in my described use-case)</div>
</div>
<div><br>
</div>
<div>The midPoint Confluence suggests also to set
various packages to trace log level to debug what is
going on but I must say that in so much logging I have
no idea what to look for.</div>
<div>When I set the projector/clockwork to debug all I
see is that the remove of the association is not
happening. More specifically this package on
trace org.identityconnectors.framework.spi.operations
points to the fact that the delete is not happening in
the connector.</div>
<div><br>
</div>
<div>May I also add that the page <a
href="https://wiki.evolveum.com/display/midPoint/Initial+Logging+Setup+HOWTO"
target="_blank" moz-do-not-send="true">https://wiki.evolveum.com/display/midPoint/Initial+Logging+Setup+HOWTO</a> is
not up to date. Setting the
skipRepositoryLoggingSettings does not work. So there
is currently no way to add specific loggers at start
up through the logback(-extra).xml. It always gets
overwritten by what is inside the repository.<br>
</div>
<div>It appears that the
skipRepositoryLoggingSettings was removed from the
code some time ago. </div>
<div><br>
</div>
<div>Kind Regards,</div>
<div>Jan Lievens</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">Op wo 15 jan. 2020 om
16:19 schreef Jan Lievens <<a
href="mailto:jan.lievens@biggerfish.be"
target="_blank" moz-do-not-send="true">jan.lievens@biggerfish.be</a>>:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px
0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Hi,
<div><br>
</div>
<div>I have a question related to inducement
construction on a role toward an LDAP resource.</div>
<div>I have the following situation in midPoint (see
the attachment for all the xml files)</div>
<div>- Accounts and Entitlements (intent:
privileges) are imported from a ScriptedSQL
connector.</div>
<div>- These imported entitlements have multiple
privileges which are created as roles with an
"assignmentTargetSearch"
(post-initial-objects/210-entitlement-object-template.xml)</div>
<div>- I also have these privilege/roles defined
with an assignment to (multiple) fixed technical
roles (kind:entitlement/intent:group).</div>
<div>- Lastly I would like for these technical roles
(groups) and the associated accounts to get synced
to LDAP in an objectToSubject fashion. I do this
with inducements and construction tags in
post-initial-objects/100-profiles-webidm.xml.</div>
<div>- The result I get in LDAP is that accounts are
synced correctly but the group names are not what
I expect (I expect technical role names
eg. JiraUser) but get the names of the
Entitlements (intent:privileges) defined in the
DB.</div>
<div>- Additionally the associations are not synced
(no uniqueMember refs are synced on the LDAP
groups).</div>
<div><br>
</div>
<div>Surely I am missing something here. I think
this use case is quiet standard (a hierarchy of
roles and only the leafs get synced to LDAP).</div>
<div>I have experimented with the order of the
inducement but these came out even more negative
(no accounts or groups were created).</div>
<div>I also have experimented with the tolerant
setting on the association (since I find a lot of
answers in this mailing list suggesting this) but
to no avail.</div>
<div>It is quiet frustrating to be so close to
having this use-case implemented
DB->mP->LDAP but failing in the sync to
LDAP.</div>
<div><br clear="all">
<div>Kind regards,</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr"><span>
<div dir="ltr">
<div dir="ltr">Jan Lievens
<div dir="ltr"><span>
<div dir="ltr">
<div dir="ltr">
<div
style="font-size:12.8px;letter-spacing:0.2px">IT
Consultant</div>
<div
style="letter-spacing:0.2px"><br>
</div>
</div>
</div>
</span></div>
</div>
</div>
</span></div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr"><span>
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>Jan Lievens
<div dir="ltr"><span>
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div
style="font-size:12.8px;letter-spacing:0.2px">IT
Consultant</div>
<div
style="letter-spacing:0.2px"><font
size="1">Bigger
Fish</font></div>
<div
style="letter-spacing:0.2px"><font
size="1">Hoogstraat
35</font></div>
<div
style="letter-spacing:0.2px"><font
size="1">9450
Haaltert</font></div>
<div
style="letter-spacing:0.2px"><font
size="1">Belgium<br>
</font></div>
<div
style="letter-spacing:0.2px"><font
size="1">VAT: BE
0734.993.447</font></div>
<div
style="font-size:12.8px;letter-spacing:0.2px">
<div><font
size="1">Mob:
+32 494 84 66
97</font></div>
</div>
</div>
</div>
</div>
</div>
</div>
</span></div>
</div>
</div>
</div>
</div>
</div>
</span></div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr"><span>
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>Jan Lievens
<div dir="ltr"><span>
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div
style="font-size:12.8px;letter-spacing:0.2px">IT
Consultant</div>
<div
style="letter-spacing:0.2px"><font
size="1">Bigger Fish</font></div>
<div
style="letter-spacing:0.2px"><font
size="1">Hoogstraat
35</font></div>
<div
style="letter-spacing:0.2px"><font
size="1">9450
Haaltert</font></div>
<div
style="letter-spacing:0.2px"><font
size="1">Belgium<br>
</font></div>
<div
style="letter-spacing:0.2px"><font
size="1">VAT: BE
0734.993.447</font></div>
<div
style="font-size:12.8px;letter-spacing:0.2px">
<div><font size="1">Mob:
+32 494 84 66 97</font></div>
</div>
</div>
</div>
</div>
</div>
</div>
</span></div>
</div>
</div>
</div>
</div>
</div>
</span></div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr"><span>
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>Jan Lievens
<div dir="ltr"><span>
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div
style="font-size:12.8px;letter-spacing:0.2px">IT
Consultant</div>
<div
style="letter-spacing:0.2px"><font
size="1">Bigger Fish</font></div>
<div
style="letter-spacing:0.2px"><font
size="1">Hoogstraat 35</font></div>
<div
style="letter-spacing:0.2px"><font
size="1">9450 Haaltert</font></div>
<div
style="letter-spacing:0.2px"><font
size="1">Belgium<br>
</font></div>
<div
style="letter-spacing:0.2px"><font
size="1">VAT: BE
0734.993.447</font></div>
<div
style="font-size:12.8px;letter-spacing:0.2px">
<div><font size="1">Mob:
+32 494 84 66 97</font></div>
</div>
</div>
</div>
</div>
</div>
</div>
</span></div>
</div>
</div>
</div>
</div>
</div>
</span></div>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
midPoint mailing list
<a class="moz-txt-link-abbreviated" href="mailto:midPoint@lists.evolveum.com">midPoint@lists.evolveum.com</a>
<a class="moz-txt-link-freetext" href="http://lists.evolveum.com/mailman/listinfo/midpoint">http://lists.evolveum.com/mailman/listinfo/midpoint</a>
</pre>
</blockquote>
</body>
</html>