[midPoint] JAVA Heap Space Problem
Pavol Mederly
mederly at evolveum.com
Tue Sep 6 12:35:43 CEST 2022
Hello Andrea,
actually - after looking better at your task - you really invoke your
bulk action as iterative (indicated by the handler URI), but using the
"search" inside it defeats the whole purpose of iterativity of the task.
Because, what it does, is invoking operation over all the matching users
for *each single user* that is found in the repository. :-) You could
check e.g. in the log files, what the task really does. BTW, I strongly
suggest to do so, just for educational purposes. (And, of course, I can
be mistaken. But I am almost sure here.)
--
Unfortunately, this kind of helping/troubleshooting/explaining, is
better provided by our training and/or professional services team, so I
have to stop here. Maybe other helpful community members could explain
this in more detail, if needed.
--
A final note: I am well aware that the creation of multi-node tasks in
4.3 and before is really ugly, hard to learn, and hard to configure. (I
created it long time ago, not having enough time and experience to think
it through. It was an experimental feature then, after all.) In 4.4, we
took hard efforts to re-work it, to make it simpler and much easier to
use. I strongly advise to upgrade.
Best regards,
--
Pavol Mederly
Software developer
evolveum.com
On 06/09/2022 12:21, Andrea Picconi via midPoint wrote:
> Hi Pavol,
>
> is my bulk iterative or is there something wrong? I created it from the example you posted on that page, the only difference is the presence of "search" in my XML, but it shouldn't be a problem I think.
> Everything else is virtually identical.
>
> Thank you and regards,
> Andrea
>
>
> -----Original Message-----
> From: midPoint<midpoint-bounces at lists.evolveum.com> On Behalf Of Pavol Mederly via midPoint
> Sent: Tuesday, September 6, 2022 10:25 AM
> To:midpoint at lists.evolveum.com
> Cc: Pavol Mederly<mederly at evolveum.com>
> Subject: Re: [midPoint] JAVA Heap Space Problem
>
> Hello,
>
> using non-iterative bulk actions is not recommended, exactly because of the memory consumption requirements. Please see the docs.
>
> https://docs.evolveum.com/midpoint/reference/misc/bulk/scripting-tasks/
>
> (the examples are pre-4.4 but can be adapted easily)
>
> Regards,
>
> --
> Pavol Mederly
> Software developer
> evolveum.com
>
> On 31/08/2022 18:02, Andrea Picconi via midPoint wrote:
>> Our version is 4.3
>> Is the same for us, usage grows over the time.
>>
>> For now, we will try to use your technique if we can find something similar to your numbers, thank you Richard.
>>
>> Andrea
>>
>> -----Original Message-----
>> From: midPoint<midpoint-bounces at lists.evolveum.com> On Behalf Of
>> Richard Frovarp via midPoint
>> Sent: Wednesday, August 31, 2022 5:41 PM
>> To:midpoint at lists.evolveum.com
>> Cc: Richard Frovarp<richard.frovarp at ndsu.edu>
>> Subject: Re: [midPoint] JAVA Heap Space Problem
>>
>> Might depend on the version you are on. I've seen it not release memory after it processes an account. So usage grows over time.
>>
>> How far does it get before it fails? My usual technique when I have to do something like this is to only operate on a subset of my population.
>> So, in your searchFilter do an AND requirement and then add a filter
>> on something. I have a number that is evenly distributed over 0-9 at
>> the end of the number on my users. So I do an ends with condition and
>> run
>> 10 of them for each number. That's get me 10% of my population at once, which easily fits in my heap. Can also do a starts with filter if that works better.
>>
>> On Wed, 2022-08-31 at 14:54 +0000, Andrea Picconi via midPoint wrote:
>>> Hi all,
>>>
>>> we have a problem of Java heap space with a bulk action: after 1-2
>>> hours from the start of the bulk action, Midpoint presents us a
>>> problem of full memory, preventing the task from moving forward and,
>>> above all, almost completely blocking the possibility of using the
>>> entire system.
>>> We need to delete an attribute on, more or less, 160k account.
>>> We have already add RAM on our machines (our system works on two
>>> nodes, in parallel) to 24GB and followed the instructions for
>>> anIterative bulk action multi-node (below our XML).
>>>
>>> An idea why we have always the same problem with the memory?
>>> Something of change inside the XML?
>>>
>>> -----
>>> <task
>>> xmlns=http://midpoint.evolveum.com/xml/ns/public/common/common-3
>>>
>>> xmlns:c=http://midpoint.evolveum.com/xml/ns/public/common/common-3
>>> xmlns:q=http://prism.evolveum.com/xml/ns/public/query-3
>>> xmlns:t=http://prism.evolveum.com/xml/ns/public/types-3
>>>
>>> xmlns:s=http://midpoint.evolveum.com/xml/ns/public/model/scripting-3
>>> xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance>
>>> <name>Bulk Delete(multinode)</name>
>>> <extension>
>>> <scext:executeScript
>>> xmlns:scext=
>>> http://midpoint.evolveum.com/xml/ns/public/model/scripting/extension-
>>> 3
>>> <s:search>
>>> <s:type>c:UserType</s:type>
>>> <s:searchFilter>
>>> <q:substring>
>>> <q:matching>polyStringNorm</q:matching>
>>>
>>> <q:path>c:extension/stuActivAdminStatus</q:path>
>>> <q:value>enabled</q:value>
>>> <q:anchorStart>true</q:anchorStart>
>>> </q:substring>
>>> </s:searchFilter>
>>> <s:action>
>>> <s:type>modify</s:type>
>>> <s:parameter>
>>> <s:name>delta</s:name>
>>> <value xsi:type="t:ObjectDeltaType">
>>> <t:changeType>modify</t:changeType>
>>> <t:itemDelta>
>>>
>>> <t:modificationType>replace</t:modifi
>>> cationType>
>>>
>>> <t:path>c:extension/stuPersonalEmail</t:path>
>>> </t:itemDelta>
>>> </value>
>>> </s:parameter>
>>> </s:action>
>>> </s:search>
>>> </scext:executeScript>
>>> <mext:objectType
>>> xmlns:mext=
>>> http://midpoint.evolveum.com/xml/ns/public/model/extension-3
>>>> UserType</mext:objectType>
>>> </extension>
>>> <ownerRef oid="00000000-0000-0000-0000-000000000002"/>
>>> <executionStatus>suspended</executionStatus>
>>>
>>> <handlerUri>
>>> http://midpoint.evolveum.com/xml/ns/public/task/workers-creation/hand
>>> l
>>> er-3</handlerUri
>>> <workManagement>
>>> <buckets>
>>> <oidSegmentation>
>>> <depth>3</depth>
>>> </oidSegmentation>
>>> </buckets>
>>> <workers>
>>>
>>> <handlerUri>
>>> http://midpoint.evolveum.com/xml/ns/public/model/iterative-scripting/
>>> h
>>> andler-3</handlerUri
>>> <workersPerNode>
>>> <count>2</count>
>>> </workersPerNode>
>>> </workers>
>>> </workManagement>
>>> <recurrence>single</recurrence>
>>> </task>
>>> ---
>>>
>>> Thank you in advance,
>>> Andrea
>>>
>>> Andrea Picconi
>>> IAM Engineer
>>> MAIL:ANDREA.PICCONI at INNOVERY.NET
>>> PHONE: +39 06 51963439 (ext. 196)
>>> MOBILE:-
>>>
>>> WWW.INNOVERY.NET
>>> STRADA QUATTRO SNC, PAL A6
>>> CENTRO DIREZIONALE MILANOFIORI | 20057 ASSAGO (MI)
>>>
>>>
>>>
>>> _______________________________________________
>>> midPoint mailing list
>>> midPoint at lists.evolveum.com
>>> https://lists.evolveum.com/mailman/listinfo/midpoint
>> _______________________________________________
>> midPoint mailing list
>> midPoint at lists.evolveum.com
>> https://lists.evolveum.com/mailman/listinfo/midpoint
>> _______________________________________________
>> midPoint mailing list
>> midPoint at lists.evolveum.com
>> https://lists.evolveum.com/mailman/listinfo/midpoint
> _______________________________________________
> midPoint mailing list
> midPoint at lists.evolveum.com
> https://lists.evolveum.com/mailman/listinfo/midpoint
> _______________________________________________
> midPoint mailing list
> midPoint at lists.evolveum.com
> https://lists.evolveum.com/mailman/listinfo/midpoint
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20220906/186df9a6/attachment-0001.htm>
More information about the midPoint
mailing list