[midPoint] Speeding up Ninja

Viliam Repan Viliam.Repan at evolveum.com
Sat Feb 17 17:17:56 CET 2024 

Hi Steven,

you can’t run ninja 4.8.x against midPoint 4.4.* because midPoint repository database schema doesn’t match.
However, it should be ok to run ninja from midPoint 4.4.7 against midPoint 4.4.1 (sql server repository) - because DB schema didn’t change between these versions of midPoint for sql server.
I’ve quickly tested this agains H2 database, seems ok. Please test this one correctly before running it in production environment.

As for the second idea - divide would be a bit complicated since there are cases where objects don’t have create/modify timestamp in metadata available, therefore such query wouldn’t work (was fixed in midPoint 4.8)
You can check whether this is your case by querying database directly with query such as:

select count(*) from m_object where createTimestamp is null or modifyTimestamp is null;

This could potentially work with exporting everything up to 2/14/24 (for objects which have create/modify timestamp is available) and then export/import everything since 2/14/24 + everything with null create/modify timestamp…

Examples of ninja exports with custom filter searching by create/modify timestamp (or it’s existence) using midPoint query language:

./bin/ninja.sh export -f '%metadata/modifyTimestamp > "2020-01-01”'

./bin/ninja.sh export -f '%metadata/modifyTimestamp not exists’

./bin/ninja.sh export -f '%metadata/createTimestamp > "2020-01-01”'

These can be combined with other options like —output (-o), —type (-t), —zip (-z), —multi-thread (-l). XML query filter can be used with -f option or can be read from file as well.

Best regards,


viliam


> On 15 Feb 2024, at 15:22, Ashwill, Steven L via midPoint <midpoint at lists.evolveum.com> wrote:
> 
> If we must use a later version of Ninja to activate multiple threads, as recommended below, can I use the ninja from 4.8.1, but only upgrade to 4.4.3.  We are currently running on 4.4.1 with a SQL Server and we are trying to go to 4.4.3 Postgres.(Smallest software change possible). The main concern is that with over 3 million user objects our downtime window for production is too long.
> If this is possible, can you provide both the export command and the import command that would be the most efficient.  
> 
> Another idea would be to divide up the work.  Is it possible to export everything up to 2/14/24, then import everything.  Then export everything since 2/14/24 and then import just a few days of changes during the cut over. If this is possible how would I do that?
> 
> Thanks
> 
> -----Original Message-----
> From: midPoint <midpoint-bounces at lists.evolveum.com> On Behalf Of midpoint-request at lists.evolveum.com
> Sent: Monday, February 5, 2024 1:38 PM
> To: midpoint at lists.evolveum.com
> Subject: midPoint Digest, Vol 142, Issue 3
> 
> Send midPoint mailing list submissions to
> 	midpoint at lists.evolveum.com
> 
> To subscribe or unsubscribe via the World Wide Web, visit
> 	https://urldefense.com/v3/__https://lists.evolveum.com/mailman/listinfo/midpoint__;!!DZ3fjg!4FmunPgtH3lNRHTdz604Ya4KL4bNJB5bWok58lwWmLH1-TEXEkq4wsqLnjFeBCmLqZFAQ57dFVQGLmiTxON4DVHRo_98wBfEIYNK5Q$
> or, via email, send a message with subject or body 'help' to
> 	midpoint-request at lists.evolveum.com
> 
> You can reach the person managing the list at
> 	midpoint-owner at lists.evolveum.com
> 
> When replying, please edit your Subject line so it is more specific than "Re: Contents of midPoint digest..."
> 
> 
> Today's Topics:
> 
>   1. Re: Speed up Ninja (Fabian Noll-Dukiewicz)
>   2. Re: Speed up Ninja (Viliam Repan)
> 
> 
> ----------------------------------------------------------------------
> 
> Message: 1
> Date: Mon, 5 Feb 2024 19:23:27 +0000
> From: Fabian Noll-Dukiewicz <fabian.noll-dukiewicz at veryfy.gmbh>
> To: midPoint General Discussion <midpoint at lists.evolveum.com>, Martin
> 	Lízner <martin.lizner at ami.cz>
> Cc: "Ashwill, Steven L" <sashwill at uillinois.edu>
> Subject: Re: [midPoint] Speed up Ninja
> Message-ID:
> 	<BE1P281MB1939794B5D06E2CD169BE88EA3472 at BE1P281MB1939.DEUP281.PROD.OUTLOOK.COM>
> 	
> Content-Type: text/plain; charset="iso-8859-2"
> 
> I think you have to use the ninja tool provided by the 4.8.1 release.
> 
> Kind regards,
> Fabian
> 
> Fabian Noll-Dukiewicz
> Spezialist Identity & Access Management | Geschäftsführer
> Tel.: +49 152 244 63 211
> Email: fabian.noll-dukiewicz at veryfy.gmbh
> Web: https://urldefense.com/v3/__https://veryfy.gmbh__;!!DZ3fjg!4FmunPgtH3lNRHTdz604Ya4KL4bNJB5bWok58lwWmLH1-TEXEkq4wsqLnjFeBCmLqZFAQ57dFVQGLmiTxON4DVHRo_98wBdkSBh3Ag$ 
> 
> 
> Von: midPoint <midpoint-bounces at lists.evolveum.com> im Auftrag von Ashwill, Steven L via midPoint <midpoint at lists.evolveum.com>
> Datum: Montag, 5. Februar 2024 um 16:55
> An: Martin Lízner <martin.lizner at ami.cz>, midpoint at lists.evolveum.com <midpoint at lists.evolveum.com>
> Cc: Ashwill, Steven L <sashwill at uillinois.edu>
> Betreff: Re: [midPoint] Speed up Ninja
> I saw the -l switch but no documentation on the website.  I tried 8 threads to 16 threads but there was no change in the time it took. We have 16 cores on the application server. We are using version 4.4.3.  Is thread perhaps a feature introduced later?
> 
> From: Martin Lízner <martin.lizner at ami.cz>
> Sent: Monday, February 5, 2024 8:04 AM
> To: midpoint at lists.evolveum.com
> Cc: Ashwill, Steven L <sashwill at uillinois.edu>
> Subject: Re: Speed up Ninja
> 
> Hi, use threads... e.g. 32
> 
> java -jar ninja.jar export -O export.zip -z -l 32 ________________________________
> Od: midPoint <midpoint-bounces at lists.evolveum.com<mailto:midpoint-bounces at lists.evolveum.com>> za uživatele Ashwill, Steven L via midPoint <midpoint at lists.evolveum.com<mailto:midpoint at lists.evolveum.com>>
> Odesláno: pondělí 5. února 2024 14:22
> Komu: midpoint at lists.evolveum.com<mailto:midpoint at lists.evolveum.com> <midpoint at lists.evolveum.com<mailto:midpoint at lists.evolveum.com>>
> Kopie: Ashwill, Steven L <sashwill at uillinois.edu<mailto:sashwill at uillinois.edu>>
> Předmět: [midPoint] Speed up Ninja
> 
> 
> We are working on migrating to Postgres using the ninja tool(4.4.1 now, then 4.8.1 this summer). We have more than 3 million user objects and the exports are taking a long time.   What options are there to speed up the export and the import?
> 
> 
> 
> STEVEN L ASHWILL
> University of Illinois Urbana-Champaign
> 
> 
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <https://urldefense.com/v3/__https://lists.evolveum.com/pipermail/midpoint/attachments/20240205/757ee92f/attachment-0001.htm__;!!DZ3fjg!4FmunPgtH3lNRHTdz604Ya4KL4bNJB5bWok58lwWmLH1-TEXEkq4wsqLnjFeBCmLqZFAQ57dFVQGLmiTxON4DVHRo_98wBf0uT3NDA$ >
> 
> ------------------------------
> 
> Message: 2
> Date: Mon, 5 Feb 2024 20:37:47 +0100
> From: Viliam Repan <viliam.repan at evolveum.com>
> To: midPoint General Discussion <midpoint at lists.evolveum.com>
> Cc: Martin Lízner <martin.lizner at ami.cz>, "Ashwill, Steven L"
> 	<sashwill at uillinois.edu>
> Subject: Re: [midPoint] Speed up Ninja
> Message-ID: <217D9554-A1D5-45D5-8894-7D3922B828EE at evolveum.com>
> Content-Type: text/plain; charset="utf-8"
> 
> Hi Ashwill,
> 
> where do you start ninja from? Directly from application server?
> 
> Ninja connects separately to midpoint repository (DB) using it’s own JDBC connection(s), so unless it’s running on application server, it doesn’t matter how many CPU app server has.
> Mostly only speed of DB matters (CPU, disk I/O).
> 
> Can you please check the following?
> 
> 1/ Is ninja running in the same VPN/network as database? Or at least the same network as application server is located?
> 2/ When you start ninja with -l option, can you check database stats (cpu, disk i/o). Also what is cpu/disk usage on machine running ninja?
> 3/ Is ninja writing export file to local disk or NFS share?
> 4/ Do you use -r (raw) export?
> 
> What I’m trying to find is where are the current bottlenecks during export.
> 
> Can you please post speeds with different number of threads used during export?
> 
> Thank you very much.
> 
> 
> Best regards,
> 
> 
> viliam
> 
>> On 5 Feb 2024, at 16:55, Ashwill, Steven L via midPoint <midpoint at lists.evolveum.com> wrote:
>> 
>> I saw the -l switch but no documentation on the website.  I tried 8 threads to 16 threads but there was no change in the time it took. We have 16 cores on the application server. We are using version 4.4.3.  Is thread perhaps a feature introduced later? 
>> 
>> From: Martin Lízner <martin.lizner at ami.cz>
>> Sent: Monday, February 5, 2024 8:04 AM
>> To: midpoint at lists.evolveum.com
>> Cc: Ashwill, Steven L <sashwill at uillinois.edu>
>> Subject: Re: Speed up Ninja
>> 
>> Hi, use threads... e.g. 32
>> 
>> java -jar ninja.jar export -O export.zip -z -l 32
>> Od: midPoint <midpoint-bounces at lists.evolveum.com 
>> <mailto:midpoint-bounces at lists.evolveum.com>> za uživatele Ashwill, 
>> Steven L via midPoint <midpoint at lists.evolveum.com 
>> <mailto:midpoint at lists.evolveum.com>>
>> Odesláno: pondělí 5. února 2024 14:22
>> Komu: midpoint at lists.evolveum.com <mailto:midpoint at lists.evolveum.com> 
>> <midpoint at lists.evolveum.com <mailto:midpoint at lists.evolveum.com>>
>> Kopie: Ashwill, Steven L <sashwill at uillinois.edu 
>> <mailto:sashwill at uillinois.edu>>
>> Předmět: [midPoint] Speed up Ninja
>> 
>> We are working on migrating to Postgres using the ninja tool(4.4.1 now, then 4.8.1 this summer). We have more than 3 million user objects and the exports are taking a long time.   What options are there to speed up the export and the import?
>> 
>> STEVEN L ASHWILL
>> University of Illinois Urbana-Champaign
>> 
>> 
>> _______________________________________________
>> midPoint mailing list
>> midPoint at lists.evolveum.com
>> https://urldefense.com/v3/__https://lists.evolveum.com/mailman/listinf
>> o/midpoint__;!!DZ3fjg!4FmunPgtH3lNRHTdz604Ya4KL4bNJB5bWok58lwWmLH1-TEX
>> Ekq4wsqLnjFeBCmLqZFAQ57dFVQGLmiTxON4DVHRo_98wBfEIYNK5Q$
> 
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <https://urldefense.com/v3/__https://lists.evolveum.com/pipermail/midpoint/attachments/20240205/041b83ee/attachment.htm__;!!DZ3fjg!4FmunPgtH3lNRHTdz604Ya4KL4bNJB5bWok58lwWmLH1-TEXEkq4wsqLnjFeBCmLqZFAQ57dFVQGLmiTxON4DVHRo_98wBfWYrapUA$ >
> 
> ------------------------------
> 
> Subject: Digest Footer
> 
> _______________________________________________
> midPoint mailing list
> midPoint at lists.evolveum.com
> https://urldefense.com/v3/__https://lists.evolveum.com/mailman/listinfo/midpoint__;!!DZ3fjg!4FmunPgtH3lNRHTdz604Ya4KL4bNJB5bWok58lwWmLH1-TEXEkq4wsqLnjFeBCmLqZFAQ57dFVQGLmiTxON4DVHRo_98wBfEIYNK5Q$ 
> 
> 
> ------------------------------
> 
> End of midPoint Digest, Vol 142, Issue 3
> ****************************************
> _______________________________________________
> midPoint mailing list
> midPoint at lists.evolveum.com
> https://lists.evolveum.com/mailman/listinfo/midpoint

More information about the midPoint mailing list