[midPoint] Speeding up Ninja

Viliam Repan viliam.repan at evolveum.com
Mon Feb 19 10:02:19 CET 2024 

Hi Steven,

as pointed out today by my colleague I forgot to mention one other needed option in my examples —raw (-r) to correctly export data from repository.
It’s also mentioned in docs: https://docs.evolveum.com/midpoint/reference/support-4.8/repository/native-postgresql/migration <https://docs.evolveum.com/midpoint/reference/support-4.8/repository/native-postgresql/migration/>

Best regards,


viliam

> On 17 Feb 2024, at 17:17, Viliam Repan via midPoint <midpoint at lists.evolveum.com> wrote:
> 
> Hi Steven,
> 
> you can’t run ninja 4.8.x against midPoint 4.4.* because midPoint repository database schema doesn’t match.
> However, it should be ok to run ninja from midPoint 4.4.7 against midPoint 4.4.1 (sql server repository) - because DB schema didn’t change between these versions of midPoint for sql server.
> I’ve quickly tested this agains H2 database, seems ok. Please test this one correctly before running it in production environment.
> 
> As for the second idea - divide would be a bit complicated since there are cases where objects don’t have create/modify timestamp in metadata available, therefore such query wouldn’t work (was fixed in midPoint 4.8)
> You can check whether this is your case by querying database directly with query such as:
> 
> select count(*) from m_object where createTimestamp is null or modifyTimestamp is null;
> 
> This could potentially work with exporting everything up to 2/14/24 (for objects which have create/modify timestamp is available) and then export/import everything since 2/14/24 + everything with null create/modify timestamp…
> 
> Examples of ninja exports with custom filter searching by create/modify timestamp (or it’s existence) using midPoint query language:
> 
> ./bin/ninja.sh export -f '%metadata/modifyTimestamp > "2020-01-01”'
> 
> ./bin/ninja.sh export -f '%metadata/modifyTimestamp not exists’
> 
> ./bin/ninja.sh export -f '%metadata/createTimestamp > "2020-01-01”'
> 
> These can be combined with other options like —output (-o), —type (-t), —zip (-z), —multi-thread (-l). XML query filter can be used with -f option or can be read from file as well.
> 
> Best regards,
> 
> 
> viliam
> 
> 
>> On 15 Feb 2024, at 15:22, Ashwill, Steven L via midPoint <midpoint at lists.evolveum.com> wrote:
>> 
>> If we must use a later version of Ninja to activate multiple threads, as recommended below, can I use the ninja from 4.8.1, but only upgrade to 4.4.3.  We are currently running on 4.4.1 with a SQL Server and we are trying to go to 4.4.3 Postgres.(Smallest software change possible). The main concern is that with over 3 million user objects our downtime window for production is too long.
>> If this is possible, can you provide both the export command and the import command that would be the most efficient.  
>> 
>> Another idea would be to divide up the work.  Is it possible to export everything up to 2/14/24, then import everything.  Then export everything since 2/14/24 and then import just a few days of changes during the cut over. If this is possible how would I do that?
>> 
>> Thanks
>> 
>> -----Original Message-----
>> From: midPoint <midpoint-bounces at lists.evolveum.com> On Behalf Of midpoint-request at lists.evolveum.com
>> Sent: Monday, February 5, 2024 1:38 PM
>> To: midpoint at lists.evolveum.com
>> Subject: midPoint Digest, Vol 142, Issue 3
>> 
>> Send midPoint mailing list submissions to
>> 	midpoint at lists.evolveum.com
>> 
>> To subscribe or unsubscribe via the World Wide Web, visit
>> 	https://urldefense.com/v3/__https://lists.evolveum.com/mailman/listinfo/midpoint__;!!DZ3fjg!4FmunPgtH3lNRHTdz604Ya4KL4bNJB5bWok58lwWmLH1-TEXEkq4wsqLnjFeBCmLqZFAQ57dFVQGLmiTxON4DVHRo_98wBfEIYNK5Q$
>> or, via email, send a message with subject or body 'help' to
>> 	midpoint-request at lists.evolveum.com
>> 
>> You can reach the person managing the list at
>> 	midpoint-owner at lists.evolveum.com
>> 
>> When replying, please edit your Subject line so it is more specific than "Re: Contents of midPoint digest..."
>> 
>> 
>> Today's Topics:
>> 
>>  1. Re: Speed up Ninja (Fabian Noll-Dukiewicz)
>>  2. Re: Speed up Ninja (Viliam Repan)
>> 
>> 
>> ----------------------------------------------------------------------
>> 
>> Message: 1
>> Date: Mon, 5 Feb 2024 19:23:27 +0000
>> From: Fabian Noll-Dukiewicz <fabian.noll-dukiewicz at veryfy.gmbh>
>> To: midPoint General Discussion <midpoint at lists.evolveum.com>, Martin
>> 	Lízner <martin.lizner at ami.cz>
>> Cc: "Ashwill, Steven L" <sashwill at uillinois.edu>
>> Subject: Re: [midPoint] Speed up Ninja
>> Message-ID:
>> 	<BE1P281MB1939794B5D06E2CD169BE88EA3472 at BE1P281MB1939.DEUP281.PROD.OUTLOOK.COM>
>> 	
>> Content-Type: text/plain; charset="iso-8859-2"
>> 
>> I think you have to use the ninja tool provided by the 4.8.1 release.
>> 
>> Kind regards,
>> Fabian
>> 
>> Fabian Noll-Dukiewicz
>> Spezialist Identity & Access Management | Geschäftsführer
>> Tel.: +49 152 244 63 211
>> Email: fabian.noll-dukiewicz at veryfy.gmbh
>> Web: https://urldefense.com/v3/__https://veryfy.gmbh__;!!DZ3fjg!4FmunPgtH3lNRHTdz604Ya4KL4bNJB5bWok58lwWmLH1-TEXEkq4wsqLnjFeBCmLqZFAQ57dFVQGLmiTxON4DVHRo_98wBdkSBh3Ag$ 
>> 
>> 
>> Von: midPoint <midpoint-bounces at lists.evolveum.com> im Auftrag von Ashwill, Steven L via midPoint <midpoint at lists.evolveum.com>
>> Datum: Montag, 5. Februar 2024 um 16:55
>> An: Martin Lízner <martin.lizner at ami.cz>, midpoint at lists.evolveum.com <midpoint at lists.evolveum.com>
>> Cc: Ashwill, Steven L <sashwill at uillinois.edu>
>> Betreff: Re: [midPoint] Speed up Ninja
>> I saw the -l switch but no documentation on the website.  I tried 8 threads to 16 threads but there was no change in the time it took. We have 16 cores on the application server. We are using version 4.4.3.  Is thread perhaps a feature introduced later?
>> 
>> From: Martin Lízner <martin.lizner at ami.cz>
>> Sent: Monday, February 5, 2024 8:04 AM
>> To: midpoint at lists.evolveum.com
>> Cc: Ashwill, Steven L <sashwill at uillinois.edu>
>> Subject: Re: Speed up Ninja
>> 
>> Hi, use threads... e.g. 32
>> 
>> java -jar ninja.jar export -O export.zip -z -l 32 ________________________________
>> Od: midPoint <midpoint-bounces at lists.evolveum.com<mailto:midpoint-bounces at lists.evolveum.com>> za uživatele Ashwill, Steven L via midPoint <midpoint at lists.evolveum.com<mailto:midpoint at lists.evolveum.com>>
>> Odesláno: pondělí 5. února 2024 14:22
>> Komu: midpoint at lists.evolveum.com<mailto:midpoint at lists.evolveum.com> <midpoint at lists.evolveum.com<mailto:midpoint at lists.evolveum.com>>
>> Kopie: Ashwill, Steven L <sashwill at uillinois.edu<mailto:sashwill at uillinois.edu>>
>> Předmět: [midPoint] Speed up Ninja
>> 
>> 
>> We are working on migrating to Postgres using the ninja tool(4.4.1 now, then 4.8.1 this summer). We have more than 3 million user objects and the exports are taking a long time.   What options are there to speed up the export and the import?
>> 
>> 
>> 
>> STEVEN L ASHWILL
>> University of Illinois Urbana-Champaign
>> 
>> 
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <https://urldefense.com/v3/__https://lists.evolveum.com/pipermail/midpoint/attachments/20240205/757ee92f/attachment-0001.htm__;!!DZ3fjg!4FmunPgtH3lNRHTdz604Ya4KL4bNJB5bWok58lwWmLH1-TEXEkq4wsqLnjFeBCmLqZFAQ57dFVQGLmiTxON4DVHRo_98wBf0uT3NDA$ >
>> 
>> ------------------------------
>> 
>> Message: 2
>> Date: Mon, 5 Feb 2024 20:37:47 +0100
>> From: Viliam Repan <viliam.repan at evolveum.com>
>> To: midPoint General Discussion <midpoint at lists.evolveum.com>
>> Cc: Martin Lízner <martin.lizner at ami.cz>, "Ashwill, Steven L"
>> 	<sashwill at uillinois.edu>
>> Subject: Re: [midPoint] Speed up Ninja
>> Message-ID: <217D9554-A1D5-45D5-8894-7D3922B828EE at evolveum.com>
>> Content-Type: text/plain; charset="utf-8"
>> 
>> Hi Ashwill,
>> 
>> where do you start ninja from? Directly from application server?
>> 
>> Ninja connects separately to midpoint repository (DB) using it’s own JDBC connection(s), so unless it’s running on application server, it doesn’t matter how many CPU app server has.
>> Mostly only speed of DB matters (CPU, disk I/O).
>> 
>> Can you please check the following?
>> 
>> 1/ Is ninja running in the same VPN/network as database? Or at least the same network as application server is located?
>> 2/ When you start ninja with -l option, can you check database stats (cpu, disk i/o). Also what is cpu/disk usage on machine running ninja?
>> 3/ Is ninja writing export file to local disk or NFS share?
>> 4/ Do you use -r (raw) export?
>> 
>> What I’m trying to find is where are the current bottlenecks during export.
>> 
>> Can you please post speeds with different number of threads used during export?
>> 
>> Thank you very much.
>> 
>> 
>> Best regards,
>> 
>> 
>> viliam
>> 
>>> On 5 Feb 2024, at 16:55, Ashwill, Steven L via midPoint <midpoint at lists.evolveum.com> wrote:
>>> 
>>> I saw the -l switch but no documentation on the website.  I tried 8 threads to 16 threads but there was no change in the time it took. We have 16 cores on the application server. We are using version 4.4.3.  Is thread perhaps a feature introduced later? 
>>> 
>>> From: Martin Lízner <martin.lizner at ami.cz>
>>> Sent: Monday, February 5, 2024 8:04 AM
>>> To: midpoint at lists.evolveum.com
>>> Cc: Ashwill, Steven L <sashwill at uillinois.edu>
>>> Subject: Re: Speed up Ninja
>>> 
>>> Hi, use threads... e.g. 32
>>> 
>>> java -jar ninja.jar export -O export.zip -z -l 32
>>> Od: midPoint <midpoint-bounces at lists.evolveum.com 
>>> <mailto:midpoint-bounces at lists.evolveum.com>> za uživatele Ashwill, 
>>> Steven L via midPoint <midpoint at lists.evolveum.com 
>>> <mailto:midpoint at lists.evolveum.com>>
>>> Odesláno: pondělí 5. února 2024 14:22
>>> Komu: midpoint at lists.evolveum.com <mailto:midpoint at lists.evolveum.com> 
>>> <midpoint at lists.evolveum.com <mailto:midpoint at lists.evolveum.com>>
>>> Kopie: Ashwill, Steven L <sashwill at uillinois.edu 
>>> <mailto:sashwill at uillinois.edu>>
>>> Předmět: [midPoint] Speed up Ninja
>>> 
>>> We are working on migrating to Postgres using the ninja tool(4.4.1 now, then 4.8.1 this summer). We have more than 3 million user objects and the exports are taking a long time.   What options are there to speed up the export and the import?
>>> 
>>> STEVEN L ASHWILL
>>> University of Illinois Urbana-Champaign
>>> 
>>> 
>>> _______________________________________________
>>> midPoint mailing list
>>> midPoint at lists.evolveum.com
>>> https://urldefense.com/v3/__https://lists.evolveum.com/mailman/listinf
>>> o/midpoint__;!!DZ3fjg!4FmunPgtH3lNRHTdz604Ya4KL4bNJB5bWok58lwWmLH1-TEX
>>> Ekq4wsqLnjFeBCmLqZFAQ57dFVQGLmiTxON4DVHRo_98wBfEIYNK5Q$
>> 
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <https://urldefense.com/v3/__https://lists.evolveum.com/pipermail/midpoint/attachments/20240205/041b83ee/attachment.htm__;!!DZ3fjg!4FmunPgtH3lNRHTdz604Ya4KL4bNJB5bWok58lwWmLH1-TEXEkq4wsqLnjFeBCmLqZFAQ57dFVQGLmiTxON4DVHRo_98wBfWYrapUA$ >
>> 
>> ------------------------------
>> 
>> Subject: Digest Footer
>> 
>> _______________________________________________
>> midPoint mailing list
>> midPoint at lists.evolveum.com
>> https://urldefense.com/v3/__https://lists.evolveum.com/mailman/listinfo/midpoint__;!!DZ3fjg!4FmunPgtH3lNRHTdz604Ya4KL4bNJB5bWok58lwWmLH1-TEXEkq4wsqLnjFeBCmLqZFAQ57dFVQGLmiTxON4DVHRo_98wBfEIYNK5Q$ 
>> 
>> 
>> ------------------------------
>> 
>> End of midPoint Digest, Vol 142, Issue 3
>> ****************************************
>> _______________________________________________
>> midPoint mailing list
>> midPoint at lists.evolveum.com
>> https://lists.evolveum.com/mailman/listinfo/midpoint
> 
> _______________________________________________
> midPoint mailing list
> midPoint at lists.evolveum.com
> https://lists.evolveum.com/mailman/listinfo/midpoint

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20240219/3c097ab8/attachment.htm>

More information about the midPoint mailing list