[midPoint] JAVA Heap Space Problem

[Pavol Mederly]

Hello,

using non-iterative bulk actions is not recommended, exactly because of 
the memory consumption requirements. Please see the docs.

https://docs.evolveum.com/midpoint/reference/misc/bulk/scripting-tasks/

(the examples are pre-4.4 but can be adapted easily)

Regards,

-- 
Pavol Mederly
Software developer
evolveum.com

On 31/08/2022 18:02, Andrea Picconi via midPoint wrote:
> Our version is 4.3
> Is the same for us, usage grows over the time.
>
> For now, we will try to use your technique if we can find something similar to your numbers, thank you Richard.
>
> Andrea
>
> -----Original Message-----
> From: midPoint <midpoint-bounces at lists.evolveum.com> On Behalf Of Richard Frovarp via midPoint
> Sent: Wednesday, August 31, 2022 5:41 PM
> To: midpoint at lists.evolveum.com
> Cc: Richard Frovarp <richard.frovarp at ndsu.edu>
> Subject: Re: [midPoint] JAVA Heap Space Problem
>
> Might depend on the version you are on. I've seen it not release memory after it processes an account. So usage grows over time.
>
> How far does it get before it fails? My usual technique when I have to do something like this is to only operate on a subset of my population.
> So, in your searchFilter do an AND requirement and then add a filter on something. I have a number that is evenly distributed over 0-9 at the end of the number on my users. So I do an ends with condition and run
> 10 of them for each number. That's get me 10% of my population at once, which easily fits in my heap. Can also do a starts with filter if that works better.
>
> On Wed, 2022-08-31 at 14:54 +0000, Andrea Picconi via midPoint wrote:
>> Hi all,
>>   
>> we have a problem of Java heap space with a bulk action: after 1-2
>> hours from the start of the bulk action, Midpoint presents us a
>> problem of full memory, preventing the task from moving forward and,
>> above all, almost completely blocking the possibility of using the
>> entire system.
>> We need to delete an attribute on, more or less, 160k account.
>> We have already add RAM on our machines (our system works on two
>> nodes, in parallel) to 24GB and followed the instructions for
>> anIterative bulk action multi-node (below our XML).
>>   
>> An idea why we have always the same problem with the memory?
>> Something of change inside the XML?
>>   
>> -----
>> <task
>> xmlns=http://midpoint.evolveum.com/xml/ns/public/common/common-3
>>       
>> xmlns:c=http://midpoint.evolveum.com/xml/ns/public/common/common-3
>>        xmlns:q=http://prism.evolveum.com/xml/ns/public/query-3
>>        xmlns:t=http://prism.evolveum.com/xml/ns/public/types-3
>>       
>> xmlns:s=http://midpoint.evolveum.com/xml/ns/public/model/scripting-3
>>        xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance>
>>      <name>Bulk Delete(multinode)</name>
>>      <extension>
>>          <scext:executeScript
>> xmlns:scext=
>> http://midpoint.evolveum.com/xml/ns/public/model/scripting/extension-3
>>              <s:search>
>>                  <s:type>c:UserType</s:type>
>>                  <s:searchFilter>
>>                      <q:substring>
>>                          <q:matching>polyStringNorm</q:matching>
>>                         
>> <q:path>c:extension/stuActivAdminStatus</q:path>
>>                          <q:value>enabled</q:value>
>>                          <q:anchorStart>true</q:anchorStart>
>>                      </q:substring>
>>                  </s:searchFilter>
>>                  <s:action>
>>                      <s:type>modify</s:type>
>>                      <s:parameter>
>>                          <s:name>delta</s:name>
>>                          <value xsi:type="t:ObjectDeltaType">
>>                              <t:changeType>modify</t:changeType>
>>                              <t:itemDelta>
>>                                  <t:modificationType>replace</t:modifi
>> cationType>
>>                        
>>           <t:path>c:extension/stuPersonalEmail</t:path>
>>                              </t:itemDelta>
>>                          </value>
>>                      </s:parameter>
>>                  </s:action>
>>              </s:search>
>>          </scext:executeScript>
>>          <mext:objectType
>> xmlns:mext=
>> http://midpoint.evolveum.com/xml/ns/public/model/extension-3
>>> UserType</mext:objectType>
>>      </extension>
>>      <ownerRef oid="00000000-0000-0000-0000-000000000002"/>
>>      <executionStatus>suspended</executionStatus>
>>     
>> <handlerUri>
>> http://midpoint.evolveum.com/xml/ns/public/task/workers-creation/handl
>> er-3</handlerUri
>>      <workManagement>
>>          <buckets>
>>              <oidSegmentation>
>>                 <depth>3</depth>
>>              </oidSegmentation>
>>          </buckets>
>>          <workers>
>>             
>> <handlerUri>
>> http://midpoint.evolveum.com/xml/ns/public/model/iterative-scripting/h
>> andler-3</handlerUri
>>              <workersPerNode>
>>                  <count>2</count>
>>              </workersPerNode>
>>          </workers>
>>      </workManagement>
>>      <recurrence>single</recurrence>
>> </task>
>> ---
>>   
>> Thank you in advance,
>> Andrea
>>   
>> Andrea Picconi
>> IAM Engineer
>> MAIL:ANDREA.PICCONI at INNOVERY.NET
>> PHONE: +39  06 51963439  (ext. 196)
>> MOBILE:-
>>   
>> WWW.INNOVERY.NET
>> STRADA QUATTRO SNC, PAL A6
>> CENTRO DIREZIONALE MILANOFIORI | 20057 ASSAGO (MI)
>>   
>>   
>>   
>> _______________________________________________
>> midPoint mailing list
>> midPoint at lists.evolveum.com
>> https://lists.evolveum.com/mailman/listinfo/midpoint
> _______________________________________________
> midPoint mailing list
> midPoint at lists.evolveum.com
> https://lists.evolveum.com/mailman/listinfo/midpoint
> _______________________________________________
> midPoint mailing list
> midPoint at lists.evolveum.com
> https://lists.evolveum.com/mailman/listinfo/midpoint