<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">Hi Steven,<div><br></div><div>as pointed out today by my colleague I forgot to mention one other needed option in my examples —raw (-r) to correctly export data from repository.</div><div>It’s also mentioned in docs: <a href="https://docs.evolveum.com/midpoint/reference/support-4.8/repository/native-postgresql/migration/">https://docs.evolveum.com/midpoint/reference/support-4.8/repository/native-postgresql/migration</a></div><div><br></div><div>Best regards,</div><div><br></div><div><br></div><div>viliam</div><div><div><br><blockquote type="cite"><div>On 17 Feb 2024, at 17:17, Viliam Repan via midPoint <midpoint@lists.evolveum.com> wrote:</div><br class="Apple-interchange-newline"><div><div>Hi Steven,<br><br>you can’t run ninja 4.8.x against midPoint 4.4.* because midPoint repository database schema doesn’t match.<br>However, it should be ok to run ninja from midPoint 4.4.7 against midPoint 4.4.1 (sql server repository) - because DB schema didn’t change between these versions of midPoint for sql server.<br>I’ve quickly tested this agains H2 database, seems ok. Please test this one correctly before running it in production environment.<br><br>As for the second idea - divide would be a bit complicated since there are cases where objects don’t have create/modify timestamp in metadata available, therefore such query wouldn’t work (was fixed in midPoint 4.8)<br>You can check whether this is your case by querying database directly with query such as:<br><br>select count(*) from m_object where createTimestamp is null or modifyTimestamp is null;<br><br>This could potentially work with exporting everything up to 2/14/24 (for objects which have create/modify timestamp is available) and then export/import everything since 2/14/24 + everything with null create/modify timestamp…<br><br>Examples of ninja exports with custom filter searching by create/modify timestamp (or it’s existence) using midPoint query language:<br><br>./bin/ninja.sh export -f '%metadata/modifyTimestamp > "2020-01-01”'<br><br>./bin/ninja.sh export -f '%metadata/modifyTimestamp not exists’<br><br>./bin/ninja.sh export -f '%metadata/createTimestamp > "2020-01-01”'<br><br>These can be combined with other options like —output (-o), —type (-t), —zip (-z), —multi-thread (-l). XML query filter can be used with -f option or can be read from file as well.<br><br>Best regards,<br><br><br>viliam<br><br><br><blockquote type="cite">On 15 Feb 2024, at 15:22, Ashwill, Steven L via midPoint <midpoint@lists.evolveum.com> wrote:<br><br>If we must use a later version of Ninja to activate multiple threads, as recommended below, can I use the ninja from 4.8.1, but only upgrade to 4.4.3. We are currently running on 4.4.1 with a SQL Server and we are trying to go to 4.4.3 Postgres.(Smallest software change possible). The main concern is that with over 3 million user objects our downtime window for production is too long.<br>If this is possible, can you provide both the export command and the import command that would be the most efficient. <br><br>Another idea would be to divide up the work. Is it possible to export everything up to 2/14/24, then import everything. Then export everything since 2/14/24 and then import just a few days of changes during the cut over. If this is possible how would I do that?<br><br>Thanks<br><br>-----Original Message-----<br>From: midPoint <midpoint-bounces@lists.evolveum.com> On Behalf Of midpoint-request@lists.evolveum.com<br>Sent: Monday, February 5, 2024 1:38 PM<br>To: midpoint@lists.evolveum.com<br>Subject: midPoint Digest, Vol 142, Issue 3<br><br>Send midPoint mailing list submissions to<br><span class="Apple-tab-span" style="white-space:pre"> </span>midpoint@lists.evolveum.com<br><br>To subscribe or unsubscribe via the World Wide Web, visit<br><span class="Apple-tab-span" style="white-space:pre"> </span>https://urldefense.com/v3/__https://lists.evolveum.com/mailman/listinfo/midpoint__;!!DZ3fjg!4FmunPgtH3lNRHTdz604Ya4KL4bNJB5bWok58lwWmLH1-TEXEkq4wsqLnjFeBCmLqZFAQ57dFVQGLmiTxON4DVHRo_98wBfEIYNK5Q$<br>or, via email, send a message with subject or body 'help' to<br><span class="Apple-tab-span" style="white-space:pre"> </span>midpoint-request@lists.evolveum.com<br><br>You can reach the person managing the list at<br><span class="Apple-tab-span" style="white-space:pre"> </span>midpoint-owner@lists.evolveum.com<br><br>When replying, please edit your Subject line so it is more specific than "Re: Contents of midPoint digest..."<br><br><br>Today's Topics:<br><br> 1. Re: Speed up Ninja (Fabian Noll-Dukiewicz)<br> 2. Re: Speed up Ninja (Viliam Repan)<br><br><br>----------------------------------------------------------------------<br><br>Message: 1<br>Date: Mon, 5 Feb 2024 19:23:27 +0000<br>From: Fabian Noll-Dukiewicz <fabian.noll-dukiewicz@veryfy.gmbh><br>To: midPoint General Discussion <midpoint@lists.evolveum.com>, Martin<br><span class="Apple-tab-span" style="white-space:pre"> </span>Lízner <martin.lizner@ami.cz><br>Cc: "Ashwill, Steven L" <sashwill@uillinois.edu><br>Subject: Re: [midPoint] Speed up Ninja<br>Message-ID:<br><span class="Apple-tab-span" style="white-space:pre"> </span><BE1P281MB1939794B5D06E2CD169BE88EA3472@BE1P281MB1939.DEUP281.PROD.OUTLOOK.COM><br><span class="Apple-tab-span" style="white-space:pre"> </span><br>Content-Type: text/plain; charset="iso-8859-2"<br><br>I think you have to use the ninja tool provided by the 4.8.1 release.<br><br>Kind regards,<br>Fabian<br><br>Fabian Noll-Dukiewicz<br>Spezialist Identity & Access Management | Geschäftsführer<br>Tel.: +49 152 244 63 211<br>Email: fabian.noll-dukiewicz@veryfy.gmbh<br>Web: https://urldefense.com/v3/__https://veryfy.gmbh__;!!DZ3fjg!4FmunPgtH3lNRHTdz604Ya4KL4bNJB5bWok58lwWmLH1-TEXEkq4wsqLnjFeBCmLqZFAQ57dFVQGLmiTxON4DVHRo_98wBdkSBh3Ag$ <br><br><br>Von: midPoint <midpoint-bounces@lists.evolveum.com> im Auftrag von Ashwill, Steven L via midPoint <midpoint@lists.evolveum.com><br>Datum: Montag, 5. Februar 2024 um 16:55<br>An: Martin Lízner <martin.lizner@ami.cz>, midpoint@lists.evolveum.com <midpoint@lists.evolveum.com><br>Cc: Ashwill, Steven L <sashwill@uillinois.edu><br>Betreff: Re: [midPoint] Speed up Ninja<br>I saw the -l switch but no documentation on the website. I tried 8 threads to 16 threads but there was no change in the time it took. We have 16 cores on the application server. We are using version 4.4.3. Is thread perhaps a feature introduced later?<br><br>From: Martin Lízner <martin.lizner@ami.cz><br>Sent: Monday, February 5, 2024 8:04 AM<br>To: midpoint@lists.evolveum.com<br>Cc: Ashwill, Steven L <sashwill@uillinois.edu><br>Subject: Re: Speed up Ninja<br><br>Hi, use threads... e.g. 32<br><br>java -jar ninja.jar export -O export.zip -z -l 32 ________________________________<br>Od: midPoint <midpoint-bounces@lists.evolveum.com<mailto:midpoint-bounces@lists.evolveum.com>> za uživatele Ashwill, Steven L via midPoint <midpoint@lists.evolveum.com<mailto:midpoint@lists.evolveum.com>><br>Odesláno: pondělí 5. února 2024 14:22<br>Komu: midpoint@lists.evolveum.com<mailto:midpoint@lists.evolveum.com> <midpoint@lists.evolveum.com<mailto:midpoint@lists.evolveum.com>><br>Kopie: Ashwill, Steven L <sashwill@uillinois.edu<mailto:sashwill@uillinois.edu>><br>Předmět: [midPoint] Speed up Ninja<br><br><br>We are working on migrating to Postgres using the ninja tool(4.4.1 now, then 4.8.1 this summer). We have more than 3 million user objects and the exports are taking a long time. What options are there to speed up the export and the import?<br><br><br><br>STEVEN L ASHWILL<br>University of Illinois Urbana-Champaign<br><br><br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <https://urldefense.com/v3/__https://lists.evolveum.com/pipermail/midpoint/attachments/20240205/757ee92f/attachment-0001.htm__;!!DZ3fjg!4FmunPgtH3lNRHTdz604Ya4KL4bNJB5bWok58lwWmLH1-TEXEkq4wsqLnjFeBCmLqZFAQ57dFVQGLmiTxON4DVHRo_98wBf0uT3NDA$ ><br><br>------------------------------<br><br>Message: 2<br>Date: Mon, 5 Feb 2024 20:37:47 +0100<br>From: Viliam Repan <viliam.repan@evolveum.com><br>To: midPoint General Discussion <midpoint@lists.evolveum.com><br>Cc: Martin Lízner <martin.lizner@ami.cz>, "Ashwill, Steven L"<br><span class="Apple-tab-span" style="white-space:pre"> </span><sashwill@uillinois.edu><br>Subject: Re: [midPoint] Speed up Ninja<br>Message-ID: <217D9554-A1D5-45D5-8894-7D3922B828EE@evolveum.com><br>Content-Type: text/plain; charset="utf-8"<br><br>Hi Ashwill,<br><br>where do you start ninja from? Directly from application server?<br><br>Ninja connects separately to midpoint repository (DB) using it’s own JDBC connection(s), so unless it’s running on application server, it doesn’t matter how many CPU app server has.<br>Mostly only speed of DB matters (CPU, disk I/O).<br><br>Can you please check the following?<br><br>1/ Is ninja running in the same VPN/network as database? Or at least the same network as application server is located?<br>2/ When you start ninja with -l option, can you check database stats (cpu, disk i/o). Also what is cpu/disk usage on machine running ninja?<br>3/ Is ninja writing export file to local disk or NFS share?<br>4/ Do you use -r (raw) export?<br><br>What I’m trying to find is where are the current bottlenecks during export.<br><br>Can you please post speeds with different number of threads used during export?<br><br>Thank you very much.<br><br><br>Best regards,<br><br><br>viliam<br><br><blockquote type="cite">On 5 Feb 2024, at 16:55, Ashwill, Steven L via midPoint <midpoint@lists.evolveum.com> wrote:<br><br>I saw the -l switch but no documentation on the website. I tried 8 threads to 16 threads but there was no change in the time it took. We have 16 cores on the application server. We are using version 4.4.3. Is thread perhaps a feature introduced later? <br><br>From: Martin Lízner <martin.lizner@ami.cz><br>Sent: Monday, February 5, 2024 8:04 AM<br>To: midpoint@lists.evolveum.com<br>Cc: Ashwill, Steven L <sashwill@uillinois.edu><br>Subject: Re: Speed up Ninja<br><br>Hi, use threads... e.g. 32<br><br>java -jar ninja.jar export -O export.zip -z -l 32<br>Od: midPoint <midpoint-bounces@lists.evolveum.com <br><mailto:midpoint-bounces@lists.evolveum.com>> za uživatele Ashwill, <br>Steven L via midPoint <midpoint@lists.evolveum.com <br><mailto:midpoint@lists.evolveum.com>><br>Odesláno: pondělí 5. února 2024 14:22<br>Komu: midpoint@lists.evolveum.com <mailto:midpoint@lists.evolveum.com> <br><midpoint@lists.evolveum.com <mailto:midpoint@lists.evolveum.com>><br>Kopie: Ashwill, Steven L <sashwill@uillinois.edu <br><mailto:sashwill@uillinois.edu>><br>Předmět: [midPoint] Speed up Ninja<br><br>We are working on migrating to Postgres using the ninja tool(4.4.1 now, then 4.8.1 this summer). We have more than 3 million user objects and the exports are taking a long time. What options are there to speed up the export and the import?<br><br>STEVEN L ASHWILL<br>University of Illinois Urbana-Champaign<br><br><br>_______________________________________________<br>midPoint mailing list<br>midPoint@lists.evolveum.com<br>https://urldefense.com/v3/__https://lists.evolveum.com/mailman/listinf<br>o/midpoint__;!!DZ3fjg!4FmunPgtH3lNRHTdz604Ya4KL4bNJB5bWok58lwWmLH1-TEX<br>Ekq4wsqLnjFeBCmLqZFAQ57dFVQGLmiTxON4DVHRo_98wBfEIYNK5Q$<br></blockquote><br>-------------- next part --------------<br>An HTML attachment was scrubbed...<br>URL: <https://urldefense.com/v3/__https://lists.evolveum.com/pipermail/midpoint/attachments/20240205/041b83ee/attachment.htm__;!!DZ3fjg!4FmunPgtH3lNRHTdz604Ya4KL4bNJB5bWok58lwWmLH1-TEXEkq4wsqLnjFeBCmLqZFAQ57dFVQGLmiTxON4DVHRo_98wBfWYrapUA$ ><br><br>------------------------------<br><br>Subject: Digest Footer<br><br>_______________________________________________<br>midPoint mailing list<br>midPoint@lists.evolveum.com<br>https://urldefense.com/v3/__https://lists.evolveum.com/mailman/listinfo/midpoint__;!!DZ3fjg!4FmunPgtH3lNRHTdz604Ya4KL4bNJB5bWok58lwWmLH1-TEXEkq4wsqLnjFeBCmLqZFAQ57dFVQGLmiTxON4DVHRo_98wBfEIYNK5Q$ <br><br><br>------------------------------<br><br>End of midPoint Digest, Vol 142, Issue 3<br>****************************************<br>_______________________________________________<br>midPoint mailing list<br>midPoint@lists.evolveum.com<br>https://lists.evolveum.com/mailman/listinfo/midpoint<br></blockquote><br>_______________________________________________<br>midPoint mailing list<br>midPoint@lists.evolveum.com<br>https://lists.evolveum.com/mailman/listinfo/midpoint<br></div></div></blockquote></div><br></div></body></html>