<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">Hi Ashwill,<div><br></div><div>where do you start ninja from? Directly from application server?</div><div><br></div><div>Ninja connects separately to midpoint repository (DB) using it’s own JDBC connection(s), so unless it’s running on application server, it doesn’t matter how many CPU app server has.</div><div>Mostly only speed of DB matters (CPU, disk I/O).</div><div><br></div><div>Can you please check the following?</div><div><br></div><div>1/ Is ninja running in the same VPN/network as database? Or at least the same network as application server is located?</div><div>2/ When you start ninja with -l option, can you check database stats (cpu, disk i/o). Also what is cpu/disk usage on machine running ninja?</div><div>3/ Is ninja writing export file to local disk or NFS share?</div><div>4/ Do you use -r (raw) export?</div><div><br></div><div>What I’m trying to find is where are the current bottlenecks during export.</div><div><br></div><div>Can you please post speeds with different number of threads used during export?</div><div><br></div><div>Thank you very much.</div><div><br></div><div><br></div><div>Best regards,</div><div><br></div><div><br></div><div>viliam</div><div><div><br><blockquote type="cite"><div>On 5 Feb 2024, at 16:55, Ashwill, Steven L via midPoint <midpoint@lists.evolveum.com> wrote:</div><br class="Apple-interchange-newline"><div><meta charset="UTF-8"><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">I saw the -l switch but no documentation on the website. I tried 8 threads to 16 threads but there was no change in the time it took. We have 16 cores on the application server. We are using version 4.4.3. Is thread perhaps a feature introduced later?<span class="Apple-converted-space"> </span><o:p></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div><div><div style="border-width: 1pt medium medium; border-style: solid none none; border-color: rgb(225, 225, 225) currentcolor currentcolor; border-image: none; padding: 3pt 0in 0in;"><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><b>From:</b><span class="Apple-converted-space"> </span>Martin Lízner <martin.lizner@ami.cz><span class="Apple-converted-space"> </span><br><b>Sent:</b><span class="Apple-converted-space"> </span>Monday, February 5, 2024 8:04 AM<br><b>To:</b><span class="Apple-converted-space"> </span>midpoint@lists.evolveum.com<br><b>Cc:</b><span class="Apple-converted-space"> </span>Ashwill, Steven L <sashwill@uillinois.edu><br><b>Subject:</b><span class="Apple-converted-space"> </span>Re: Speed up Ninja<o:p></o:p></div></div></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div><div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 12pt; font-family: Aptos, sans-serif;">Hi, use threads... e.g. 32<o:p></o:p></span></div></div><div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 12pt; font-family: Aptos, sans-serif;"><o:p> </o:p></span></div></div><div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 12pt; font-family: "Roboto Mono"; background: rgb(248, 248, 248);">java<span class="Apple-converted-space"> </span></span><span style="font-size: 12pt; font-family: "Roboto Mono"; color: navy; background: rgb(248, 248, 248);">-jar</span><span style="font-size: 12pt; font-family: "Roboto Mono"; background: rgb(248, 248, 248);"> ninja.jar<span class="Apple-converted-space"> </span></span><span style="font-size: 12pt; font-family: "Roboto Mono"; color: rgb(0, 134, 179); background: rgb(248, 248, 248);">export</span><span style="font-size: 12pt; font-family: "Roboto Mono"; background: rgb(248, 248, 248);"><span class="Apple-converted-space"> </span></span><span style="font-size: 12pt; font-family: "Roboto Mono"; color: navy; background: rgb(248, 248, 248);">-O</span><span style="font-size: 12pt; font-family: "Roboto Mono"; background: rgb(248, 248, 248);"> export.zip<span class="Apple-converted-space"> </span></span><span style="font-size: 12pt; font-family: "Roboto Mono"; color: navy; background: rgb(248, 248, 248);">-z</span><span style="font-size: 12pt; font-family: "Roboto Mono"; background: rgb(248, 248, 248);"><span class="Apple-converted-space"> </span></span><span style="font-size: 12pt; font-family: "Roboto Mono"; color: navy; background: rgb(248, 248, 248);">-l</span><span style="font-size: 12pt; font-family: "Roboto Mono"; background: rgb(248, 248, 248);"> 32</span><o:p></o:p></div></div><div class="MsoNormal" align="center" style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; text-align: center;"><hr size="2" width="98%" align="center"></div><div id="divRplyFwdMsg"><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"><b><span style="">Od:</span></b><span style=""><span class="Apple-converted-space"> </span>midPoint <<a href="mailto:midpoint-bounces@lists.evolveum.com" style="color: rgb(5, 99, 193); text-decoration: underline;">midpoint-bounces@lists.evolveum.com</a>> za uživatele Ashwill, Steven L via midPoint <<a href="mailto:midpoint@lists.evolveum.com" style="color: rgb(5, 99, 193); text-decoration: underline;">midpoint@lists.evolveum.com</a>><br><b>Odesláno:</b><span class="Apple-converted-space"> </span>pondělí 5. února 2024 14:22<br><b>Komu:</b><span class="Apple-converted-space"> </span><a href="mailto:midpoint@lists.evolveum.com" style="color: rgb(5, 99, 193); text-decoration: underline;">midpoint@lists.evolveum.com</a><span class="Apple-converted-space"> </span><<a href="mailto:midpoint@lists.evolveum.com" style="color: rgb(5, 99, 193); text-decoration: underline;">midpoint@lists.evolveum.com</a>><br><b>Kopie:</b><span class="Apple-converted-space"> </span>Ashwill, Steven L <<a href="mailto:sashwill@uillinois.edu" style="color: rgb(5, 99, 193); text-decoration: underline;">sashwill@uillinois.edu</a>><br><b>Předmět:</b><span class="Apple-converted-space"> </span>[midPoint] Speed up Ninja</span><o:p></o:p></div><div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div><div><div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;">We are working on migrating to Postgres using the ninja tool(4.4.1 now, then 4.8.1 this summer). We have more than 3 million user objects and the exports are taking a long time. What options are there to speed up the export and the import?<o:p></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div><p class="xmsonormal" style="margin: 0in 0in 12pt; font-size: 11pt; font-family: Calibri, sans-serif;"><b><span style="color: rgb(19, 41, 75); text-transform: uppercase;">STEVEN L ASHWILL</span></b><span style=""><br></span><span style="color: rgb(19, 41, 75);">University of Illinois Urbana-Champaign</span><o:p></o:p></p><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;"> <o:p></o:p></div></div></div></div><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">_______________________________________________</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">midPoint mailing list</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">midPoint@lists.evolveum.com</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;">https://lists.evolveum.com/mailman/listinfo/midpoint</span></div></blockquote></div><br></div></body></html>