[midPoint] Livesynch Task Suspended on Error

Tue Oct 4 21:14:58 CEST 2022

Thanks for the information and looking into that log file. Would this be
addressed by having a tight binding to a specific node for the livesynch
task? Or would increasing the quartzClusterCheckinGracePeriod still be
required (based off details in [MID-5500] Reconc error: Couldn't complete
work bucket for task - Evolveum Jira
<https://jira.evolveum.com/browse/MID-5500> )?

Thank you,

Jeremiah Haywood

Lead IAM Administrator

Office of Technology Solutions | Illinois State University

Phone Number (309) 438-3829

From: midPoint <midpoint-bounces at lists.evolveum.com> On Behalf Of Pavol
Mederly via midPoint
Sent: Tuesday, October 4, 2022 2:07 PM
To: midpoint at lists.evolveum.com
Cc: Pavol Mederly <mederly at evolveum.com>
Subject: Re: [midPoint] Livesynch Task Suspended on Error

This message originated from outside of the Illinois State University email
system. Learn why this is important
<https://help.illinoisstate.edu/technology/page-88179827.html> 

I've just looked at the details...

2022-10-03 19:37:01,932 []
[QuartzScheduler_midPointScheduler-ilstu-midpoint-dev-02_ClusterManager]
WARN (org.quartz.impl.jdbcjobstore.JobStoreTX): This scheduler instance
(ilstu-midpoint-dev-02) is still active but was recovered by another
instance in the cluster.  This may cause inconsistent behavior.
2022-10-03 19:48:25,090 []
[QuartzScheduler_midPointScheduler-ilstu-midpoint-dev-02_ClusterManager]
INFO (org.quartz.impl.jdbcjobstore.JobStoreTX): ClusterManager: detected 1
failed or restarted instances.
2022-10-03 19:48:25,090 []
[QuartzScheduler_midPointScheduler-ilstu-midpoint-dev-02_ClusterManager]
INFO (org.quartz.impl.jdbcjobstore.JobStoreTX): ClusterManager: Scanning for
instance "ilstu-midpoint-dev-01"'s failed in-progress jobs.
2022-10-03 19:50:11,947 []
[QuartzScheduler_midPointScheduler-ilstu-midpoint-dev-02_ClusterManager]
WARN (org.quartz.impl.jdbcjobstore.JobStoreTX): This scheduler instance
(ilstu-midpoint-dev-02) is still active but was recovered by another
instance in the cluster.  This may cause inconsistent behavior.
2022-10-03 20:05:53,283 []
[QuartzScheduler_midPointScheduler-ilstu-midpoint-dev-02_ClusterManager]
WARN (org.quartz.impl.jdbcjobstore.JobStoreTX): This scheduler instance
(ilstu-midpoint-dev-02) is still active but was recovered by another
instance in the cluster.  This may cause inconsistent behavior.
2022-10-03 20:19:10,271 []
[QuartzScheduler_midPointScheduler-ilstu-midpoint-dev-02_ClusterManager]
WARN (org.quartz.impl.jdbcjobstore.JobStoreTX): This scheduler instance
(ilstu-midpoint-dev-02) is still active but was recovered by another
instance in the cluster.  This may cause inconsistent behavior.
2022-10-03 20:32:45,142 []
[QuartzScheduler_midPointScheduler-ilstu-midpoint-dev-02_ClusterManager]
INFO (org.quartz.impl.jdbcjobstore.JobStoreTX): ClusterManager: detected 1
failed or restarted instances.
2022-10-03 20:32:45,142 []
[QuartzScheduler_midPointScheduler-ilstu-midpoint-dev-02_ClusterManager]
INFO (org.quartz.impl.jdbcjobstore.JobStoreTX): ClusterManager: Scanning for
instance "ilstu-midpoint-dev-01"'s failed in-progress jobs.
2022-10-03 20:32:45,164 []
[QuartzScheduler_midPointScheduler-ilstu-midpoint-dev-02_ClusterManager]
INFO (org.quartz.impl.jdbcjobstore.JobStoreTX): ClusterManager:
......Scheduled 1 recoverable job(s) for recovery.
2022-10-03 20:32:45,193 [] [midPointScheduler_Worker-9] INFO
(com.evolveum.midpoint.task.quartzimpl.run.JobExecutor): Task
Task(id:1664827507317-0-1, name:identityTable (livesynch),
oid:9d446a2e-0819-4041-9f66-bf716d76b084) is recovering

This points to the cluster issue with 100% certainty. :)

-- 
Pavol Mederly
Software developer
evolveum.com

On 04/10/2022 20:32, Haywood, Jeremiah via midPoint wrote:

Afternoon all,

We are experimenting with live synch in a clustered deployment and have come
across some situations where it experiences a fatal error and moves to a
suspended state. The most recent error we received was the following after
container reboot: 

GUI Results:

Operation:
com.evolveum.midpoint.repo.common.activity.run.LocalActivityRun.runLocally

Status: FATAL_ERROR

Message: 

Bucket
PCV(4531):[PP({.../common/common-3}sequentialNumber):[PPV(Integer:1)],
PP({.../common/common-3}state):[PPV(WorkBucketStateType:COMPLETE)],
PP({.../common/common-3}content):[PPV(NullWorkBucketContentType:com.evolveum
.midpoint.xml.ns._public.common.common_3.NullWorkBucketContentType at 6e57bf8b[
])]] is not ready

I have also attached the stack trace from the logs. Is it possible to
configure Midpoint to continue subsequent runs after permanent errors?
Additionally, have others experienced similar behavior?

Thank you,

Jeremiah Haywood

Lead IAM Administrator

Office of Technology Solutions | Illinois State University

Phone Number (309) 438-3829

_______________________________________________
midPoint mailing list
midPoint at lists.evolveum.com <mailto:midPoint at lists.evolveum.com> 
https://lists.evolveum.com/mailman/listinfo/midpoint
<https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.evo
lveum.com%2Fmailman%2Flistinfo%2Fmidpoint&data=05%7C01%7Cjhaywo1%40ilstu.edu
%7Cdc5111b189304a7a3dda08daa63b9833%7C085f983a0b694270b71d10695076bafe%7C1%7
C0%7C638005072145908451%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoi
V2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=gwxJeyzJAirBwI2
HMSbMbuDc0ghHmZ6xa0%2BvSTl2Qp8%3D&reserved=0> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20221004/15534242/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 58001 bytes
Desc: not available
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20221004/15534242/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 7264 bytes
Desc: not available
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20221004/15534242/attachment-0001.bin>