<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Segoe UI";
panose-1:2 11 5 2 4 2 4 2 2 3;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
font-size:10.0pt;
font-family:"Courier New";}
tt
{mso-style-priority:99;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Consolas",serif;}
span.EmailStyle26
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style></head><body lang=EN-US link=blue vlink=purple style='word-wrap:break-word'><div class=WordSection1><p class=MsoNormal>Thanks for the information and looking into that log file. Would this be addressed by having a tight binding to a specific node for the livesynch task? Or would increasing the <tt><span style='font-size:10.5pt;color:#172B4D;background:white'>quartzClusterCheckinGracePeriod</span></tt><span style='font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#172B4D;background:white'> still be required (based off details in </span><a href="https://jira.evolveum.com/browse/MID-5500">[MID-5500] Reconc error: Couldn't complete work bucket for task - Evolveum Jira</a><span style='font-size:10.5pt;font-family:"Segoe UI",sans-serif;color:#172B4D;background:white'>)?</span><o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Thank you,<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Jeremiah Haywood<o:p></o:p></p><p class=MsoNormal>Lead IAM Administrator<o:p></o:p></p><p class=MsoNormal>Office of Technology Solutions | Illinois State University<o:p></o:p></p><p class=MsoNormal>Phone Number (309) 438-3829<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b>From:</b> midPoint <midpoint-bounces@lists.evolveum.com> <b>On Behalf Of </b>Pavol Mederly via midPoint<br><b>Sent:</b> Tuesday, October 4, 2022 2:07 PM<br><b>To:</b> midpoint@lists.evolveum.com<br><b>Cc:</b> Pavol Mederly <mederly@evolveum.com><br><b>Subject:</b> Re: [midPoint] Livesynch Task Suspended on Error<o:p></o:p></p></div></div><p class=MsoNormal><o:p> </o:p></p><table class=MsoNormalTable border=0 cellspacing=0 cellpadding=0 align=left width="100%" style='width:100.0%'><tr><td style='background:#A6A6A6;padding:5.25pt 1.5pt 5.25pt 1.5pt'></td><td width="100%" style='width:100.0%;background:#EAEAEA;padding:5.25pt 3.75pt 5.25pt 11.25pt;overflow-wrap: break-word'><div><p class=MsoNormal style='mso-element:frame;mso-element-frame-hspace:2.25pt;mso-element-wrap:around;mso-element-anchor-vertical:paragraph;mso-element-anchor-horizontal:column;mso-height-rule:exactly'><span style='font-size:9.0pt;font-family:"Segoe UI",sans-serif;color:#212121'>This message originated from outside of the Illinois State University email system. <a href="https://help.illinoisstate.edu/technology/page-88179827.html" target="_blank">Learn why this is important</a><o:p></o:p></span></p></div></td><td width=75 style='width:56.25pt;background:#EAEAEA;padding:5.25pt 3.75pt 5.25pt 3.75pt;overflow-wrap: break-word'></td></tr></table><div><p>I've just looked at the details...<o:p></o:p></p><p><span style='font-family:"Courier New"'>2022-10-03 19:37:01,932 [] [QuartzScheduler_midPointScheduler-ilstu-midpoint-dev-02_ClusterManager] WARN (org.quartz.impl.jdbcjobstore.JobStoreTX): This scheduler instance (ilstu-midpoint-dev-02) is still active but was recovered by another instance in the cluster. This may cause inconsistent behavior.<br>2022-10-03 19:48:25,090 [] [QuartzScheduler_midPointScheduler-ilstu-midpoint-dev-02_ClusterManager] INFO (org.quartz.impl.jdbcjobstore.JobStoreTX): ClusterManager: detected 1 failed or restarted instances.<br>2022-10-03 19:48:25,090 [] [QuartzScheduler_midPointScheduler-ilstu-midpoint-dev-02_ClusterManager] INFO (org.quartz.impl.jdbcjobstore.JobStoreTX): ClusterManager: Scanning for instance "ilstu-midpoint-dev-01"'s failed in-progress jobs.<br>2022-10-03 19:50:11,947 [] [QuartzScheduler_midPointScheduler-ilstu-midpoint-dev-02_ClusterManager] WARN (org.quartz.impl.jdbcjobstore.JobStoreTX): This scheduler instance (ilstu-midpoint-dev-02) is still active but was recovered by another instance in the cluster. This may cause inconsistent behavior.<br>2022-10-03 20:05:53,283 [] [QuartzScheduler_midPointScheduler-ilstu-midpoint-dev-02_ClusterManager] WARN (org.quartz.impl.jdbcjobstore.JobStoreTX): This scheduler instance (ilstu-midpoint-dev-02) is still active but was recovered by another instance in the cluster. This may cause inconsistent behavior.<br>2022-10-03 20:19:10,271 [] [QuartzScheduler_midPointScheduler-ilstu-midpoint-dev-02_ClusterManager] WARN (org.quartz.impl.jdbcjobstore.JobStoreTX): This scheduler instance (ilstu-midpoint-dev-02) is still active but was recovered by another instance in the cluster. This may cause inconsistent behavior.<br>2022-10-03 20:32:45,142 [] [QuartzScheduler_midPointScheduler-ilstu-midpoint-dev-02_ClusterManager] INFO (org.quartz.impl.jdbcjobstore.JobStoreTX): ClusterManager: detected 1 failed or restarted instances.<br>2022-10-03 20:32:45,142 [] [QuartzScheduler_midPointScheduler-ilstu-midpoint-dev-02_ClusterManager] INFO (org.quartz.impl.jdbcjobstore.JobStoreTX): ClusterManager: Scanning for instance "ilstu-midpoint-dev-01"'s failed in-progress jobs.<br>2022-10-03 20:32:45,164 [] [QuartzScheduler_midPointScheduler-ilstu-midpoint-dev-02_ClusterManager] INFO (org.quartz.impl.jdbcjobstore.JobStoreTX): ClusterManager: ......Scheduled 1 recoverable job(s) for recovery.<br>2022-10-03 20:32:45,193 [] [midPointScheduler_Worker-9] INFO (com.evolveum.midpoint.task.quartzimpl.run.JobExecutor): Task Task(id:1664827507317-0-1, name:identityTable (livesynch), oid:9d446a2e-0819-4041-9f66-bf716d76b084) is recovering</span><o:p></o:p></p><p>This points to the cluster issue with 100% certainty. :)<o:p></o:p></p><pre>-- <o:p></o:p></pre><pre>Pavol Mederly<o:p></o:p></pre><pre>Software developer<o:p></o:p></pre><pre>evolveum.com<o:p></o:p></pre><div><p class=MsoNormal>On 04/10/2022 20:32, Haywood, Jeremiah via midPoint wrote:<o:p></o:p></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal>Afternoon all,<o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal>We are experimenting with live synch in a clustered deployment and have come across some situations where it experiences a fatal error and moves to a suspended state. The most recent error we received was the following after container reboot: <o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal>GUI Results:<o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal><img border=0 width=746 height=583 style='width:7.7708in;height:6.0729in' id="Picture_x0020_1" src="cid:image001.png@01D8D7FB.93143D90"><o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal>Operation: com.evolveum.midpoint.repo.common.activity.run.LocalActivityRun.runLocally<o:p></o:p></p><p class=MsoNormal>Status: FATAL_ERROR<o:p></o:p></p><p class=MsoNormal>Message: <o:p></o:p></p><p class=MsoNormal>Bucket PCV(4531):[PP({.../common/common-3}sequentialNumber):[PPV(Integer:1)], PP({.../common/common-3}state):[PPV(WorkBucketStateType:COMPLETE)], PP({.../common/common-3}content):[PPV(NullWorkBucketContentType:com.evolveum.midpoint.xml.ns._public.common.common_3.NullWorkBucketContentType@6e57bf8b[])]] is not ready<o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal>I have also attached the stack trace from the logs. Is it possible to configure Midpoint to continue subsequent runs after permanent errors? Additionally, have others experienced similar behavior?<o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal>Thank you,<o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal>Jeremiah Haywood<o:p></o:p></p><p class=MsoNormal>Lead IAM Administrator<o:p></o:p></p><p class=MsoNormal>Office of Technology Solutions | Illinois State University<o:p></o:p></p><p class=MsoNormal>Phone Number (309) 438-3829<o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal><br><br><o:p></o:p></p><pre>_______________________________________________<o:p></o:p></pre><pre>midPoint mailing list<o:p></o:p></pre><pre><a href="mailto:midPoint@lists.evolveum.com">midPoint@lists.evolveum.com</a><o:p></o:p></pre><pre><a href="https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.evolveum.com%2Fmailman%2Flistinfo%2Fmidpoint&data=05%7C01%7Cjhaywo1%40ilstu.edu%7Cdc5111b189304a7a3dda08daa63b9833%7C085f983a0b694270b71d10695076bafe%7C1%7C0%7C638005072145908451%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=gwxJeyzJAirBwI2HMSbMbuDc0ghHmZ6xa0%2BvSTl2Qp8%3D&reserved=0">https://lists.evolveum.com/mailman/listinfo/midpoint</a><o:p></o:p></pre></blockquote></div></div></body></html>