<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hello Andrea,</p>
<p>actually - after looking better at your task - you really invoke
your bulk action as iterative (indicated by the handler URI), but
using the "search" inside it defeats the whole purpose of
iterativity of the task.<br>
</p>
<p>Because, what it does, is invoking operation over all the
matching users for <b>each single user</b> that is found in the
repository. :-) You could check e.g. in the log files, what the
task really does. BTW, I strongly suggest to do so, just for
educational purposes. (And, of course, I can be mistaken. But I am
almost sure here.)</p>
<p>--<br>
</p>
<p>Unfortunately, this kind of helping/troubleshooting/explaining,
is better provided by our training and/or professional services
team, so I have to stop here. Maybe other helpful community
members could explain this in more detail, if needed.</p>
<p>--<br>
</p>
<p>A final note: I am well aware that the creation of multi-node
tasks in 4.3 and before is really ugly, hard to learn, and hard to
configure. (I created it long time ago, not having enough time and
experience to think it through. It was an experimental feature
then, after all.) In 4.4, we took hard efforts to re-work it, to
make it simpler and much easier to use. I strongly advise to
upgrade.<br>
</p>
<p>Best regards,<br>
</p>
<pre class="moz-signature" cols="72">--
Pavol Mederly
Software developer
evolveum.com</pre>
<div class="moz-cite-prefix">On 06/09/2022 12:21, Andrea Picconi via
midPoint wrote:<br>
</div>
<blockquote type="cite"
cite="mid:DU2PR09MB5470D10A2A9310FE78AEE4B4F77E9@DU2PR09MB5470.eurprd09.prod.outlook.com">
<pre class="moz-quote-pre" wrap="">Hi Pavol,
is my bulk iterative or is there something wrong? I created it from the example you posted on that page, the only difference is the presence of "search" in my XML, but it shouldn't be a problem I think.
Everything else is virtually identical.
Thank you and regards,
Andrea
-----Original Message-----
From: midPoint <a class="moz-txt-link-rfc2396E" href="mailto:midpoint-bounces@lists.evolveum.com"><midpoint-bounces@lists.evolveum.com></a> On Behalf Of Pavol Mederly via midPoint
Sent: Tuesday, September 6, 2022 10:25 AM
To: <a class="moz-txt-link-abbreviated" href="mailto:midpoint@lists.evolveum.com">midpoint@lists.evolveum.com</a>
Cc: Pavol Mederly <a class="moz-txt-link-rfc2396E" href="mailto:mederly@evolveum.com"><mederly@evolveum.com></a>
Subject: Re: [midPoint] JAVA Heap Space Problem
Hello,
using non-iterative bulk actions is not recommended, exactly because of the memory consumption requirements. Please see the docs.
<a class="moz-txt-link-freetext" href="https://docs.evolveum.com/midpoint/reference/misc/bulk/scripting-tasks/">https://docs.evolveum.com/midpoint/reference/misc/bulk/scripting-tasks/</a>
(the examples are pre-4.4 but can be adapted easily)
Regards,
--
Pavol Mederly
Software developer
evolveum.com
On 31/08/2022 18:02, Andrea Picconi via midPoint wrote:
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">Our version is 4.3
Is the same for us, usage grows over the time.
For now, we will try to use your technique if we can find something similar to your numbers, thank you Richard.
Andrea
-----Original Message-----
From: midPoint <a class="moz-txt-link-rfc2396E" href="mailto:midpoint-bounces@lists.evolveum.com"><midpoint-bounces@lists.evolveum.com></a> On Behalf Of
Richard Frovarp via midPoint
Sent: Wednesday, August 31, 2022 5:41 PM
To: <a class="moz-txt-link-abbreviated" href="mailto:midpoint@lists.evolveum.com">midpoint@lists.evolveum.com</a>
Cc: Richard Frovarp <a class="moz-txt-link-rfc2396E" href="mailto:richard.frovarp@ndsu.edu"><richard.frovarp@ndsu.edu></a>
Subject: Re: [midPoint] JAVA Heap Space Problem
Might depend on the version you are on. I've seen it not release memory after it processes an account. So usage grows over time.
How far does it get before it fails? My usual technique when I have to do something like this is to only operate on a subset of my population.
So, in your searchFilter do an AND requirement and then add a filter
on something. I have a number that is evenly distributed over 0-9 at
the end of the number on my users. So I do an ends with condition and
run
10 of them for each number. That's get me 10% of my population at once, which easily fits in my heap. Can also do a starts with filter if that works better.
On Wed, 2022-08-31 at 14:54 +0000, Andrea Picconi via midPoint wrote:
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">Hi all,
we have a problem of Java heap space with a bulk action: after 1-2
hours from the start of the bulk action, Midpoint presents us a
problem of full memory, preventing the task from moving forward and,
above all, almost completely blocking the possibility of using the
entire system.
We need to delete an attribute on, more or less, 160k account.
We have already add RAM on our machines (our system works on two
nodes, in parallel) to 24GB and followed the instructions for
anIterative bulk action multi-node (below our XML).
An idea why we have always the same problem with the memory?
Something of change inside the XML?
-----
<task
xmlns=<a class="moz-txt-link-freetext" href="http://midpoint.evolveum.com/xml/ns/public/common/common-3">http://midpoint.evolveum.com/xml/ns/public/common/common-3</a>
xmlns:c=<a class="moz-txt-link-freetext" href="http://midpoint.evolveum.com/xml/ns/public/common/common-3">http://midpoint.evolveum.com/xml/ns/public/common/common-3</a>
xmlns:q=<a class="moz-txt-link-freetext" href="http://prism.evolveum.com/xml/ns/public/query-3">http://prism.evolveum.com/xml/ns/public/query-3</a>
xmlns:t=<a class="moz-txt-link-freetext" href="http://prism.evolveum.com/xml/ns/public/types-3">http://prism.evolveum.com/xml/ns/public/types-3</a>
xmlns:s=<a class="moz-txt-link-freetext" href="http://midpoint.evolveum.com/xml/ns/public/model/scripting-3">http://midpoint.evolveum.com/xml/ns/public/model/scripting-3</a>
xmlns:xsi=<a class="moz-txt-link-freetext" href="http://www.w3.org/2001/XMLSchema-instance">http://www.w3.org/2001/XMLSchema-instance</a>>
<name>Bulk Delete(multinode)</name>
<extension>
<scext:executeScript
xmlns:scext=
<a class="moz-txt-link-freetext" href="http://midpoint.evolveum.com/xml/ns/public/model/scripting/extension">http://midpoint.evolveum.com/xml/ns/public/model/scripting/extension</a>-
3
<s:search>
<s:type>c:UserType</s:type>
<s:searchFilter>
<q:substring>
<q:matching>polyStringNorm</q:matching>
<q:path>c:extension/stuActivAdminStatus</q:path>
<q:value>enabled</q:value>
<q:anchorStart>true</q:anchorStart>
</q:substring>
</s:searchFilter>
<s:action>
<s:type>modify</s:type>
<s:parameter>
<s:name>delta</s:name>
<value xsi:type="t:ObjectDeltaType">
<t:changeType>modify</t:changeType>
<t:itemDelta>
<t:modificationType>replace</t:modifi
cationType>
<t:path>c:extension/stuPersonalEmail</t:path>
</t:itemDelta>
</value>
</s:parameter>
</s:action>
</s:search>
</scext:executeScript>
<mext:objectType
xmlns:mext=
<a class="moz-txt-link-freetext" href="http://midpoint.evolveum.com/xml/ns/public/model/extension-3">http://midpoint.evolveum.com/xml/ns/public/model/extension-3</a>
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">UserType</mext:objectType>
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap=""> </extension>
<ownerRef oid="00000000-0000-0000-0000-000000000002"/>
<executionStatus>suspended</executionStatus>
<handlerUri>
<a class="moz-txt-link-freetext" href="http://midpoint.evolveum.com/xml/ns/public/task/workers-creation/hand">http://midpoint.evolveum.com/xml/ns/public/task/workers-creation/hand</a>
l
er-3</handlerUri
<workManagement>
<buckets>
<oidSegmentation>
<depth>3</depth>
</oidSegmentation>
</buckets>
<workers>
<handlerUri>
<a class="moz-txt-link-freetext" href="http://midpoint.evolveum.com/xml/ns/public/model/iterative-scripting/">http://midpoint.evolveum.com/xml/ns/public/model/iterative-scripting/</a>
h
andler-3</handlerUri
<workersPerNode>
<count>2</count>
</workersPerNode>
</workers>
</workManagement>
<recurrence>single</recurrence>
</task>
---
Thank you in advance,
Andrea
Andrea Picconi
IAM Engineer
<a class="moz-txt-link-abbreviated" href="mailto:MAIL:ANDREA.PICCONI@INNOVERY.NET">MAIL:ANDREA.PICCONI@INNOVERY.NET</a>
PHONE: +39 06 51963439 (ext. 196)
MOBILE:-
<a class="moz-txt-link-abbreviated" href="http://WWW.INNOVERY.NET">WWW.INNOVERY.NET</a>
STRADA QUATTRO SNC, PAL A6
CENTRO DIREZIONALE MILANOFIORI | 20057 ASSAGO (MI)
_______________________________________________
midPoint mailing list
<a class="moz-txt-link-abbreviated" href="mailto:midPoint@lists.evolveum.com">midPoint@lists.evolveum.com</a>
<a class="moz-txt-link-freetext" href="https://lists.evolveum.com/mailman/listinfo/midpoint">https://lists.evolveum.com/mailman/listinfo/midpoint</a>
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">_______________________________________________
midPoint mailing list
<a class="moz-txt-link-abbreviated" href="mailto:midPoint@lists.evolveum.com">midPoint@lists.evolveum.com</a>
<a class="moz-txt-link-freetext" href="https://lists.evolveum.com/mailman/listinfo/midpoint">https://lists.evolveum.com/mailman/listinfo/midpoint</a>
_______________________________________________
midPoint mailing list
<a class="moz-txt-link-abbreviated" href="mailto:midPoint@lists.evolveum.com">midPoint@lists.evolveum.com</a>
<a class="moz-txt-link-freetext" href="https://lists.evolveum.com/mailman/listinfo/midpoint">https://lists.evolveum.com/mailman/listinfo/midpoint</a>
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">_______________________________________________
midPoint mailing list
<a class="moz-txt-link-abbreviated" href="mailto:midPoint@lists.evolveum.com">midPoint@lists.evolveum.com</a>
<a class="moz-txt-link-freetext" href="https://lists.evolveum.com/mailman/listinfo/midpoint">https://lists.evolveum.com/mailman/listinfo/midpoint</a>
_______________________________________________
midPoint mailing list
<a class="moz-txt-link-abbreviated" href="mailto:midPoint@lists.evolveum.com">midPoint@lists.evolveum.com</a>
<a class="moz-txt-link-freetext" href="https://lists.evolveum.com/mailman/listinfo/midpoint">https://lists.evolveum.com/mailman/listinfo/midpoint</a>
</pre>
</blockquote>
</body>
</html>