[midPoint] Speeding up Ninja

Thu Feb 15 15:22:39 CET 2024

If we must use a later version of Ninja to activate multiple threads, as recommended below, can I use the ninja from 4.8.1, but only upgrade to 4.4.3.  We are currently running on 4.4.1 with a SQL Server and we are trying to go to 4.4.3 Postgres.(Smallest software change possible). The main concern is that with over 3 million user objects our downtime window for production is too long.
If this is possible, can you provide both the export command and the import command that would be the most efficient.  

Another idea would be to divide up the work.  Is it possible to export everything up to 2/14/24, then import everything.  Then export everything since 2/14/24 and then import just a few days of changes during the cut over. If this is possible how would I do that?

Thanks

-----Original Message-----
From: midPoint <midpoint-bounces at lists.evolveum.com> On Behalf Of midpoint-request at lists.evolveum.com
Sent: Monday, February 5, 2024 1:38 PM
To: midpoint at lists.evolveum.com
Subject: midPoint Digest, Vol 142, Issue 3

Send midPoint mailing list submissions to
	midpoint at lists.evolveum.com

To subscribe or unsubscribe via the World Wide Web, visit
	https://urldefense.com/v3/__https://lists.evolveum.com/mailman/listinfo/midpoint__;!!DZ3fjg!4FmunPgtH3lNRHTdz604Ya4KL4bNJB5bWok58lwWmLH1-TEXEkq4wsqLnjFeBCmLqZFAQ57dFVQGLmiTxON4DVHRo_98wBfEIYNK5Q$
or, via email, send a message with subject or body 'help' to
	midpoint-request at lists.evolveum.com

You can reach the person managing the list at
	midpoint-owner at lists.evolveum.com

When replying, please edit your Subject line so it is more specific than "Re: Contents of midPoint digest..."

Today's Topics:

   1. Re: Speed up Ninja (Fabian Noll-Dukiewicz)
   2. Re: Speed up Ninja (Viliam Repan)

----------------------------------------------------------------------

Message: 1
Date: Mon, 5 Feb 2024 19:23:27 +0000
From: Fabian Noll-Dukiewicz <fabian.noll-dukiewicz at veryfy.gmbh>
To: midPoint General Discussion <midpoint at lists.evolveum.com>, Martin
	Lízner <martin.lizner at ami.cz>
Cc: "Ashwill, Steven L" <sashwill at uillinois.edu>
Subject: Re: [midPoint] Speed up Ninja
Message-ID:
	<BE1P281MB1939794B5D06E2CD169BE88EA3472 at BE1P281MB1939.DEUP281.PROD.OUTLOOK.COM>

Content-Type: text/plain; charset="iso-8859-2"

I think you have to use the ninja tool provided by the 4.8.1 release.

Kind regards,
Fabian

Fabian Noll-Dukiewicz
Spezialist Identity & Access Management | Geschäftsführer
Tel.: +49 152 244 63 211
Email: fabian.noll-dukiewicz at veryfy.gmbh
Web: https://urldefense.com/v3/__https://veryfy.gmbh__;!!DZ3fjg!4FmunPgtH3lNRHTdz604Ya4KL4bNJB5bWok58lwWmLH1-TEXEkq4wsqLnjFeBCmLqZFAQ57dFVQGLmiTxON4DVHRo_98wBdkSBh3Ag$ 

Von: midPoint <midpoint-bounces at lists.evolveum.com> im Auftrag von Ashwill, Steven L via midPoint <midpoint at lists.evolveum.com>
Datum: Montag, 5. Februar 2024 um 16:55
An: Martin Lízner <martin.lizner at ami.cz>, midpoint at lists.evolveum.com <midpoint at lists.evolveum.com>
Cc: Ashwill, Steven L <sashwill at uillinois.edu>
Betreff: Re: [midPoint] Speed up Ninja
I saw the -l switch but no documentation on the website.  I tried 8 threads to 16 threads but there was no change in the time it took. We have 16 cores on the application server. We are using version 4.4.3.  Is thread perhaps a feature introduced later?

From: Martin Lízner <martin.lizner at ami.cz>
Sent: Monday, February 5, 2024 8:04 AM
To: midpoint at lists.evolveum.com
Cc: Ashwill, Steven L <sashwill at uillinois.edu>
Subject: Re: Speed up Ninja

Hi, use threads... e.g. 32

java -jar ninja.jar export -O export.zip -z -l 32 ________________________________
Od: midPoint <midpoint-bounces at lists.evolveum.com<mailto:midpoint-bounces at lists.evolveum.com>> za uživatele Ashwill, Steven L via midPoint <midpoint at lists.evolveum.com<mailto:midpoint at lists.evolveum.com>>
Odesláno: pondělí 5. února 2024 14:22
Komu: midpoint at lists.evolveum.com<mailto:midpoint at lists.evolveum.com> <midpoint at lists.evolveum.com<mailto:midpoint at lists.evolveum.com>>
Kopie: Ashwill, Steven L <sashwill at uillinois.edu<mailto:sashwill at uillinois.edu>>
Předmět: [midPoint] Speed up Ninja

We are working on migrating to Postgres using the ninja tool(4.4.1 now, then 4.8.1 this summer). We have more than 3 million user objects and the exports are taking a long time.   What options are there to speed up the export and the import?

STEVEN L ASHWILL
University of Illinois Urbana-Champaign

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://urldefense.com/v3/__https://lists.evolveum.com/pipermail/midpoint/attachments/20240205/757ee92f/attachment-0001.htm__;!!DZ3fjg!4FmunPgtH3lNRHTdz604Ya4KL4bNJB5bWok58lwWmLH1-TEXEkq4wsqLnjFeBCmLqZFAQ57dFVQGLmiTxON4DVHRo_98wBf0uT3NDA$ >

------------------------------

Message: 2
Date: Mon, 5 Feb 2024 20:37:47 +0100
From: Viliam Repan <viliam.repan at evolveum.com>
To: midPoint General Discussion <midpoint at lists.evolveum.com>
Cc: Martin Lízner <martin.lizner at ami.cz>, "Ashwill, Steven L"
	<sashwill at uillinois.edu>
Subject: Re: [midPoint] Speed up Ninja
Message-ID: <217D9554-A1D5-45D5-8894-7D3922B828EE at evolveum.com>
Content-Type: text/plain; charset="utf-8"

Hi Ashwill,

where do you start ninja from? Directly from application server?

Ninja connects separately to midpoint repository (DB) using it’s own JDBC connection(s), so unless it’s running on application server, it doesn’t matter how many CPU app server has.
Mostly only speed of DB matters (CPU, disk I/O).

Can you please check the following?

1/ Is ninja running in the same VPN/network as database? Or at least the same network as application server is located?
2/ When you start ninja with -l option, can you check database stats (cpu, disk i/o). Also what is cpu/disk usage on machine running ninja?
3/ Is ninja writing export file to local disk or NFS share?
4/ Do you use -r (raw) export?

What I’m trying to find is where are the current bottlenecks during export.

Can you please post speeds with different number of threads used during export?

Thank you very much.

Best regards,

viliam

> On 5 Feb 2024, at 16:55, Ashwill, Steven L via midPoint <midpoint at lists.evolveum.com> wrote:
> 
> I saw the -l switch but no documentation on the website.  I tried 8 threads to 16 threads but there was no change in the time it took. We have 16 cores on the application server. We are using version 4.4.3.  Is thread perhaps a feature introduced later? 
>  
> From: Martin Lízner <martin.lizner at ami.cz>
> Sent: Monday, February 5, 2024 8:04 AM
> To: midpoint at lists.evolveum.com
> Cc: Ashwill, Steven L <sashwill at uillinois.edu>
> Subject: Re: Speed up Ninja
>  
> Hi, use threads... e.g. 32
>  
> java -jar ninja.jar export -O export.zip -z -l 32
> Od: midPoint <midpoint-bounces at lists.evolveum.com 
> <mailto:midpoint-bounces at lists.evolveum.com>> za uživatele Ashwill, 
> Steven L via midPoint <midpoint at lists.evolveum.com 
> <mailto:midpoint at lists.evolveum.com>>
> Odesláno: pondělí 5. února 2024 14:22
> Komu: midpoint at lists.evolveum.com <mailto:midpoint at lists.evolveum.com> 
> <midpoint at lists.evolveum.com <mailto:midpoint at lists.evolveum.com>>
> Kopie: Ashwill, Steven L <sashwill at uillinois.edu 
> <mailto:sashwill at uillinois.edu>>
> Předmět: [midPoint] Speed up Ninja
>  
> We are working on migrating to Postgres using the ninja tool(4.4.1 now, then 4.8.1 this summer). We have more than 3 million user objects and the exports are taking a long time.   What options are there to speed up the export and the import?
>  
> STEVEN L ASHWILL
> University of Illinois Urbana-Champaign
> 
>  
> _______________________________________________
> midPoint mailing list
> midPoint at lists.evolveum.com
> https://urldefense.com/v3/__https://lists.evolveum.com/mailman/listinf
> o/midpoint__;!!DZ3fjg!4FmunPgtH3lNRHTdz604Ya4KL4bNJB5bWok58lwWmLH1-TEX
> Ekq4wsqLnjFeBCmLqZFAQ57dFVQGLmiTxON4DVHRo_98wBfEIYNK5Q$

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://urldefense.com/v3/__https://lists.evolveum.com/pipermail/midpoint/attachments/20240205/041b83ee/attachment.htm__;!!DZ3fjg!4FmunPgtH3lNRHTdz604Ya4KL4bNJB5bWok58lwWmLH1-TEXEkq4wsqLnjFeBCmLqZFAQ57dFVQGLmiTxON4DVHRo_98wBfWYrapUA$ >

------------------------------

Subject: Digest Footer

_______________________________________________
midPoint mailing list
midPoint at lists.evolveum.com
https://urldefense.com/v3/__https://lists.evolveum.com/mailman/listinfo/midpoint__;!!DZ3fjg!4FmunPgtH3lNRHTdz604Ya4KL4bNJB5bWok58lwWmLH1-TEXEkq4wsqLnjFeBCmLqZFAQ57dFVQGLmiTxON4DVHRo_98wBfEIYNK5Q$ 

------------------------------

End of midPoint Digest, Vol 142, Issue 3
****************************************