<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Source Sans Pro";}
@font-face
{font-family:-apple-system;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">I have an update related to this issue. This is now fixed and it ended up being caused by the sql jdbc driver.<o:p></o:p></p>
<p class="MsoNormal">Now, although to reproduce the issue, there are what seem to be very specific conditions, I would like to document them here in case someone else finds them.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Environment:<o:p></o:p></p>
<p class="MsoNormal">MidPoint v.4.4.3<o:p></o:p></p>
<p class="MsoNormal">Apache Tomcat 9<o:p></o:p></p>
<p class="MsoNormal">OpenJDK_11<o:p></o:p></p>
<p class="MsoNormal">MSSQL as backend database<br>
SQL JDBC driver 7.2 (v7.2.0 specifically)<o:p></o:p></p>
<p class="MsoNormal">Two server nodes in clustered mode<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">A possible bug in the driver (specifically v7.2.0) caused the error mentioned below to appear when running two nodes in clustered mode (No errors when running a single node). The fix was to update the driver to v7.2.2.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><span style="color:#1F497D">Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">William Velasco<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Velasco, William <br>
<b>Sent:</b> Monday, November 14, 2022 9:22 AM<br>
<b>To:</b> midPoint General Discussion <midpoint@lists.evolveum.com><br>
<b>Subject:</b> RE: Tasks not running after upgrade to v4.4.3<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">There is one more piece of information to add:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">This issue seems to be specific to a cluster configuration. We completed the same upgrade path (v4.0.4->v4.4.3) on a single node installation and there are no issues at all with the tasks triggering and running. However, after attempting
the upgrade on another instance with a two nodes cluster, we saw the same issues as reported in my previous email.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The only error showing in the logs is exactly the same as reported in my previous email too.<o:p></o:p></p>
<div>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">William Velasco<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> midPoint <<a href="mailto:midpoint-bounces@lists.evolveum.com">midpoint-bounces@lists.evolveum.com</a>>
<b>On Behalf Of </b>Velasco, William via midPoint<br>
<b>Sent:</b> Thursday, November 3, 2022 9:23 AM<br>
<b>To:</b> midPoint General Discussion <<a href="mailto:midpoint@lists.evolveum.com">midpoint@lists.evolveum.com</a>><br>
<b>Cc:</b> Velasco, William <<a href="mailto:William.Velasco@devry.edu">William.Velasco@devry.edu</a>><br>
<b>Subject:</b> Re: [midPoint] Tasks not running after upgrade to v4.4.3<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I want to point out that the custom tasks were migrated to the new task definition style using
<span style="font-size:12.0pt;font-family:-apple-system;color:black;background:white">
midPoint Studio and then re-uploaded</span>. Also, the system tasks (cleanup, trigger, validation) have been recreated (deleted using the GUI and restarting the nodes).<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The issue is that none of the tasks run at all. Whether by the scheduler, manually running the task, or being called via the REST API.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Whenever the task is submitted to run, there are no errors or activity in the logs that indicates the task is indeed running.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">William Velasco<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> midPoint <<a href="mailto:midpoint-bounces@lists.evolveum.com">midpoint-bounces@lists.evolveum.com</a>>
<b>On Behalf Of </b>Velasco, William via midPoint<br>
<b>Sent:</b> Tuesday, November 1, 2022 5:39 PM<br>
<b>To:</b> <a href="mailto:midpoint@lists.evolveum.com">midpoint@lists.evolveum.com</a><br>
<b>Cc:</b> Velasco, William <<a href="mailto:William.Velasco@devry.edu">William.Velasco@devry.edu</a>><br>
<b>Subject:</b> [midPoint] Tasks not running after upgrade to v4.4.3<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hi,<o:p></o:p></p>
<p class="MsoNormal">We have recently upgraded to MidPoint v4.4.3 from v4.0.4, and while the upgrade of the different components was completed without any issues or warnings, we have noticed that none of the tasks (system or custom) would run at all after the
upgrade. <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The task configuration looks fine, and the scheduler seems to think they are running (The Status tab shows “In Progress”). However, no actions seem to be performed (no activities in the logs). Eventually, the
<b><span style="font-size:10.5pt;font-family:"Source Sans Pro",sans-serif;color:#333333;background:white">Scheduled to start again</span></b> tab shows as “<span style="font-size:10.5pt;font-family:"Source Sans Pro",sans-serif;color:#333333;background:#F9F9F9">already
passed</span>”.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Although the timestamp does not seem to relate directly to when the task is supposed to trigger, there is an error that seems to be recurrent on the logs (about every four minutes in each of the two nodes we run).<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">2022-11-01 17:31:32,308 [] [QuartzScheduler_midPointScheduler-Node1_MisfireHandler] ERROR (org.quartz.impl.jdbcjobstore.JobStoreTX): MisfireHandler: Error handling misfires: Database error recovering from misfires.<o:p></o:p></p>
<p class="MsoNormal">org.quartz.JobPersistenceException: Database error recovering from misfires.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Can you please advise how to handle this issue?<o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">William Velasco<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="margin-top:7.5pt">
<p class="MsoNormal"><img border="0" width="25" height="21" style="width:.2604in;height:.2187in" id="_x0000_i1025" src="https://get.paubox.com/hubfs/Email%20folder/green_lock_v2.png">Secured by
<a href="https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.paubox.com%2F&data=05%7C01%7CWilliam.Velasco%40devry.edu%7Ce54ff9ec79fb46b9824308dabda6e851%7Cf979c99c51ab4ff686e035fd2874788f%7C0%7C0%7C638030821804916832%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000%7C%7C%7C&sdata=uc0a8SJWFW7E43XzOEtS8mNWE1FPHF%2BHEq2jzBECnLM%3D&reserved=0">
Paubox</a> - HITRUST CSF certified<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="margin-top:7.5pt">
<p class="MsoNormal"><img border="0" width="25" height="21" style="width:.2604in;height:.2187in" id="_x0000_i1026" src="https://get.paubox.com/hubfs/Email%20folder/green_lock_v2.png">Secured by
<a href="https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.paubox.com%2F&data=05%7C01%7CWilliam.Velasco%40devry.edu%7Ce54ff9ec79fb46b9824308dabda6e851%7Cf979c99c51ab4ff686e035fd2874788f%7C0%7C0%7C638030821804916832%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000%7C%7C%7C&sdata=uc0a8SJWFW7E43XzOEtS8mNWE1FPHF%2BHEq2jzBECnLM%3D&reserved=0">
Paubox</a> - HITRUST CSF certified<o:p></o:p></p>
</div>
</div>
</body>
</html>
<br><div style="display: block; margin-top:10px"><img width="25" height="21" src="https://get.paubox.com/hubfs/Email%20folder/green_lock_v2.png" /> Secured by <a href="https://www.paubox.com">Paubox</a> - HITRUST CSF certified</div>