[midPoint] Tasks not running after upgrade to v4.4.3

[Velasco, William]

I have an update related to this issue. This is now fixed and it ended up being caused by the sql jdbc driver.
Now, although to reproduce the issue, there are what seem to be very specific conditions, I would like to document them here in case someone else finds them.

Environment:
MidPoint v.4.4.3
Apache Tomcat 9
OpenJDK_11
MSSQL as backend database
SQL JDBC driver 7.2 (v7.2.0 specifically)
Two server nodes in clustered mode

A possible bug in the driver (specifically v7.2.0) caused the error mentioned below to appear when running two nodes in clustered mode (No errors when running a single node). The fix was to update the driver to v7.2.2.


Regards,
William Velasco

From: Velasco, William
Sent: Monday, November 14, 2022 9:22 AM
To: midPoint General Discussion <midpoint at lists.evolveum.com>
Subject: RE: Tasks not running after upgrade to v4.4.3

There is one more piece of information to add:

This issue seems to be specific to a cluster configuration. We completed the same upgrade path (v4.0.4->v4.4.3) on a single node installation and there are no issues at all with the tasks triggering and running. However, after attempting the upgrade on another instance with a two nodes cluster, we saw the same issues as reported in my previous email.

The only error showing in the logs is exactly the same as reported in my previous email too.


Regards,
William Velasco

From: midPoint <midpoint-bounces at lists.evolveum.com<mailto:midpoint-bounces at lists.evolveum.com>> On Behalf Of Velasco, William via midPoint
Sent: Thursday, November 3, 2022 9:23 AM
To: midPoint General Discussion <midpoint at lists.evolveum.com<mailto:midpoint at lists.evolveum.com>>
Cc: Velasco, William <William.Velasco at devry.edu<mailto:William.Velasco at devry.edu>>
Subject: Re: [midPoint] Tasks not running after upgrade to v4.4.3

I want to point out that the custom tasks were migrated to the new task definition style using midPoint Studio and then re-uploaded. Also, the system tasks (cleanup, trigger, validation) have been recreated (deleted using the GUI and restarting the nodes).

The issue is that none of the tasks run at all. Whether by the scheduler, manually running the task, or being called via the REST API.

Whenever the task is submitted to run, there are no errors or activity in the logs that indicates the task is indeed running.


Regards,
William Velasco

From: midPoint <midpoint-bounces at lists.evolveum.com<mailto:midpoint-bounces at lists.evolveum.com>> On Behalf Of Velasco, William via midPoint
Sent: Tuesday, November 1, 2022 5:39 PM
To: midpoint at lists.evolveum.com<mailto:midpoint at lists.evolveum.com>
Cc: Velasco, William <William.Velasco at devry.edu<mailto:William.Velasco at devry.edu>>
Subject: [midPoint] Tasks not running after upgrade to v4.4.3

Hi,
We have recently upgraded to MidPoint v4.4.3 from v4.0.4, and while the upgrade of the different components was completed without any issues or warnings, we have noticed that none of the tasks (system or custom) would run at all after the upgrade.

The task configuration looks fine, and the scheduler seems to think they are running (The Status tab shows "In Progress"). However, no actions seem to be performed (no activities in the logs). Eventually, the Scheduled to start again tab shows as "already passed".

Although the timestamp does not seem to relate directly to when the task is supposed to trigger, there is an error that seems to be recurrent on the logs (about every four minutes in each of the two nodes we run).

2022-11-01 17:31:32,308 [] [QuartzScheduler_midPointScheduler-Node1_MisfireHandler] ERROR (org.quartz.impl.jdbcjobstore.JobStoreTX): MisfireHandler: Error handling misfires: Database error recovering from misfires.
org.quartz.JobPersistenceException: Database error recovering from misfires.

Can you please advise how to handle this issue?


Regards,
William Velasco


[https://get.paubox.com/hubfs/Email%20folder/green_lock_v2.png]Secured by Paubox<https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.paubox.com%2F&data=05%7C01%7CWilliam.Velasco%40devry.edu%7Ce54ff9ec79fb46b9824308dabda6e851%7Cf979c99c51ab4ff686e035fd2874788f%7C0%7C0%7C638030821804916832%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000%7C%7C%7C&sdata=uc0a8SJWFW7E43XzOEtS8mNWE1FPHF%2BHEq2jzBECnLM%3D&reserved=0> - HITRUST CSF certified

[https://get.paubox.com/hubfs/Email%20folder/green_lock_v2.png]Secured by Paubox<https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.paubox.com%2F&data=05%7C01%7CWilliam.Velasco%40devry.edu%7Ce54ff9ec79fb46b9824308dabda6e851%7Cf979c99c51ab4ff686e035fd2874788f%7C0%7C0%7C638030821804916832%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000%7C%7C%7C&sdata=uc0a8SJWFW7E43XzOEtS8mNWE1FPHF%2BHEq2jzBECnLM%3D&reserved=0> - HITRUST CSF certified
 
 ----
Secured by Paubox - HITRUST CSF certified
https://www.paubox.com
----
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20221206/4bac1a35/attachment.htm>