[midPoint] Speed up Ninja

[Viliam Repan]

Hi Ashwill,

where do you start ninja from? Directly from application server?

Ninja connects separately to midpoint repository (DB) using it’s own JDBC connection(s), so unless it’s running on application server, it doesn’t matter how many CPU app server has.
Mostly only speed of DB matters (CPU, disk I/O).

Can you please check the following?

1/ Is ninja running in the same VPN/network as database? Or at least the same network as application server is located?
2/ When you start ninja with -l option, can you check database stats (cpu, disk i/o). Also what is cpu/disk usage on machine running ninja?
3/ Is ninja writing export file to local disk or NFS share?
4/ Do you use -r (raw) export?

What I’m trying to find is where are the current bottlenecks during export.

Can you please post speeds with different number of threads used during export?

Thank you very much.


Best regards,


viliam

> On 5 Feb 2024, at 16:55, Ashwill, Steven L via midPoint <midpoint at lists.evolveum.com> wrote:
> 
> I saw the -l switch but no documentation on the website.  I tried 8 threads to 16 threads but there was no change in the time it took. We have 16 cores on the application server. We are using version 4.4.3.  Is thread perhaps a feature introduced later? 
>  
> From: Martin Lízner <martin.lizner at ami.cz> 
> Sent: Monday, February 5, 2024 8:04 AM
> To: midpoint at lists.evolveum.com
> Cc: Ashwill, Steven L <sashwill at uillinois.edu>
> Subject: Re: Speed up Ninja
>  
> Hi, use threads... e.g. 32
>  
> java -jar ninja.jar export -O export.zip -z -l 32
> Od: midPoint <midpoint-bounces at lists.evolveum.com <mailto:midpoint-bounces at lists.evolveum.com>> za uživatele Ashwill, Steven L via midPoint <midpoint at lists.evolveum.com <mailto:midpoint at lists.evolveum.com>>
> Odesláno: pondělí 5. února 2024 14:22
> Komu: midpoint at lists.evolveum.com <mailto:midpoint at lists.evolveum.com> <midpoint at lists.evolveum.com <mailto:midpoint at lists.evolveum.com>>
> Kopie: Ashwill, Steven L <sashwill at uillinois.edu <mailto:sashwill at uillinois.edu>>
> Předmět: [midPoint] Speed up Ninja
>  
> We are working on migrating to Postgres using the ninja tool(4.4.1 now, then 4.8.1 this summer). We have more than 3 million user objects and the exports are taking a long time.   What options are there to speed up the export and the import?
>  
> STEVEN L ASHWILL
> University of Illinois Urbana-Champaign
> 
>  
> _______________________________________________
> midPoint mailing list
> midPoint at lists.evolveum.com
> https://lists.evolveum.com/mailman/listinfo/midpoint

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20240205/041b83ee/attachment-0001.htm>