[midPoint] JAVA Heap Space Problem
Andrea Picconi
andrea.picconi at innovery.net
Tue Sep 6 12:21:04 CEST 2022
Hi Pavol,
is my bulk iterative or is there something wrong? I created it from the example you posted on that page, the only difference is the presence of "search" in my XML, but it shouldn't be a problem I think.
Everything else is virtually identical.
Thank you and regards,
Andrea
-----Original Message-----
From: midPoint <midpoint-bounces at lists.evolveum.com> On Behalf Of Pavol Mederly via midPoint
Sent: Tuesday, September 6, 2022 10:25 AM
To: midpoint at lists.evolveum.com
Cc: Pavol Mederly <mederly at evolveum.com>
Subject: Re: [midPoint] JAVA Heap Space Problem
Hello,
using non-iterative bulk actions is not recommended, exactly because of the memory consumption requirements. Please see the docs.
https://docs.evolveum.com/midpoint/reference/misc/bulk/scripting-tasks/
(the examples are pre-4.4 but can be adapted easily)
Regards,
--
Pavol Mederly
Software developer
evolveum.com
On 31/08/2022 18:02, Andrea Picconi via midPoint wrote:
> Our version is 4.3
> Is the same for us, usage grows over the time.
>
> For now, we will try to use your technique if we can find something similar to your numbers, thank you Richard.
>
> Andrea
>
> -----Original Message-----
> From: midPoint <midpoint-bounces at lists.evolveum.com> On Behalf Of
> Richard Frovarp via midPoint
> Sent: Wednesday, August 31, 2022 5:41 PM
> To: midpoint at lists.evolveum.com
> Cc: Richard Frovarp <richard.frovarp at ndsu.edu>
> Subject: Re: [midPoint] JAVA Heap Space Problem
>
> Might depend on the version you are on. I've seen it not release memory after it processes an account. So usage grows over time.
>
> How far does it get before it fails? My usual technique when I have to do something like this is to only operate on a subset of my population.
> So, in your searchFilter do an AND requirement and then add a filter
> on something. I have a number that is evenly distributed over 0-9 at
> the end of the number on my users. So I do an ends with condition and
> run
> 10 of them for each number. That's get me 10% of my population at once, which easily fits in my heap. Can also do a starts with filter if that works better.
>
> On Wed, 2022-08-31 at 14:54 +0000, Andrea Picconi via midPoint wrote:
>> Hi all,
>>
>> we have a problem of Java heap space with a bulk action: after 1-2
>> hours from the start of the bulk action, Midpoint presents us a
>> problem of full memory, preventing the task from moving forward and,
>> above all, almost completely blocking the possibility of using the
>> entire system.
>> We need to delete an attribute on, more or less, 160k account.
>> We have already add RAM on our machines (our system works on two
>> nodes, in parallel) to 24GB and followed the instructions for
>> anIterative bulk action multi-node (below our XML).
>>
>> An idea why we have always the same problem with the memory?
>> Something of change inside the XML?
>>
>> -----
>> <task
>> xmlns=http://midpoint.evolveum.com/xml/ns/public/common/common-3
>>
>> xmlns:c=http://midpoint.evolveum.com/xml/ns/public/common/common-3
>> xmlns:q=http://prism.evolveum.com/xml/ns/public/query-3
>> xmlns:t=http://prism.evolveum.com/xml/ns/public/types-3
>>
>> xmlns:s=http://midpoint.evolveum.com/xml/ns/public/model/scripting-3
>> xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance>
>> <name>Bulk Delete(multinode)</name>
>> <extension>
>> <scext:executeScript
>> xmlns:scext=
>> http://midpoint.evolveum.com/xml/ns/public/model/scripting/extension-
>> 3
>> <s:search>
>> <s:type>c:UserType</s:type>
>> <s:searchFilter>
>> <q:substring>
>> <q:matching>polyStringNorm</q:matching>
>>
>> <q:path>c:extension/stuActivAdminStatus</q:path>
>> <q:value>enabled</q:value>
>> <q:anchorStart>true</q:anchorStart>
>> </q:substring>
>> </s:searchFilter>
>> <s:action>
>> <s:type>modify</s:type>
>> <s:parameter>
>> <s:name>delta</s:name>
>> <value xsi:type="t:ObjectDeltaType">
>> <t:changeType>modify</t:changeType>
>> <t:itemDelta>
>>
>> <t:modificationType>replace</t:modifi
>> cationType>
>>
>> <t:path>c:extension/stuPersonalEmail</t:path>
>> </t:itemDelta>
>> </value>
>> </s:parameter>
>> </s:action>
>> </s:search>
>> </scext:executeScript>
>> <mext:objectType
>> xmlns:mext=
>> http://midpoint.evolveum.com/xml/ns/public/model/extension-3
>>> UserType</mext:objectType>
>> </extension>
>> <ownerRef oid="00000000-0000-0000-0000-000000000002"/>
>> <executionStatus>suspended</executionStatus>
>>
>> <handlerUri>
>> http://midpoint.evolveum.com/xml/ns/public/task/workers-creation/hand
>> l
>> er-3</handlerUri
>> <workManagement>
>> <buckets>
>> <oidSegmentation>
>> <depth>3</depth>
>> </oidSegmentation>
>> </buckets>
>> <workers>
>>
>> <handlerUri>
>> http://midpoint.evolveum.com/xml/ns/public/model/iterative-scripting/
>> h
>> andler-3</handlerUri
>> <workersPerNode>
>> <count>2</count>
>> </workersPerNode>
>> </workers>
>> </workManagement>
>> <recurrence>single</recurrence>
>> </task>
>> ---
>>
>> Thank you in advance,
>> Andrea
>>
>> Andrea Picconi
>> IAM Engineer
>> MAIL:ANDREA.PICCONI at INNOVERY.NET
>> PHONE: +39 06 51963439 (ext. 196)
>> MOBILE:-
>>
>> WWW.INNOVERY.NET
>> STRADA QUATTRO SNC, PAL A6
>> CENTRO DIREZIONALE MILANOFIORI | 20057 ASSAGO (MI)
>>
>>
>>
>> _______________________________________________
>> midPoint mailing list
>> midPoint at lists.evolveum.com
>> https://lists.evolveum.com/mailman/listinfo/midpoint
> _______________________________________________
> midPoint mailing list
> midPoint at lists.evolveum.com
> https://lists.evolveum.com/mailman/listinfo/midpoint
> _______________________________________________
> midPoint mailing list
> midPoint at lists.evolveum.com
> https://lists.evolveum.com/mailman/listinfo/midpoint
_______________________________________________
midPoint mailing list
midPoint at lists.evolveum.com
https://lists.evolveum.com/mailman/listinfo/midpoint
More information about the midPoint
mailing list