2022-10-03 19:37:01,932 [] [QuartzScheduler_midPointScheduler-ilstu-midpoint-dev-02_ClusterManager] WARN (org.quartz.impl.jdbcjobstore.JobStoreTX): This scheduler instance (ilstu-midpoint-dev-02) is still active but was recovered by another instance in the cluster. This may cause inconsistent behavior. 2022-10-03 19:48:25,090 [] [QuartzScheduler_midPointScheduler-ilstu-midpoint-dev-02_ClusterManager] INFO (org.quartz.impl.jdbcjobstore.JobStoreTX): ClusterManager: detected 1 failed or restarted instances. 2022-10-03 19:48:25,090 [] [QuartzScheduler_midPointScheduler-ilstu-midpoint-dev-02_ClusterManager] INFO (org.quartz.impl.jdbcjobstore.JobStoreTX): ClusterManager: Scanning for instance "ilstu-midpoint-dev-01"'s failed in-progress jobs. 2022-10-03 19:50:11,947 [] [QuartzScheduler_midPointScheduler-ilstu-midpoint-dev-02_ClusterManager] WARN (org.quartz.impl.jdbcjobstore.JobStoreTX): This scheduler instance (ilstu-midpoint-dev-02) is still active but was recovered by another instance in the cluster. This may cause inconsistent behavior. 2022-10-03 20:05:53,283 [] [QuartzScheduler_midPointScheduler-ilstu-midpoint-dev-02_ClusterManager] WARN (org.quartz.impl.jdbcjobstore.JobStoreTX): This scheduler instance (ilstu-midpoint-dev-02) is still active but was recovered by another instance in the cluster. This may cause inconsistent behavior. 2022-10-03 20:19:10,271 [] [QuartzScheduler_midPointScheduler-ilstu-midpoint-dev-02_ClusterManager] WARN (org.quartz.impl.jdbcjobstore.JobStoreTX): This scheduler instance (ilstu-midpoint-dev-02) is still active but was recovered by another instance in the cluster. This may cause inconsistent behavior. 2022-10-03 20:32:45,142 [] [QuartzScheduler_midPointScheduler-ilstu-midpoint-dev-02_ClusterManager] INFO (org.quartz.impl.jdbcjobstore.JobStoreTX): ClusterManager: detected 1 failed or restarted instances. 2022-10-03 20:32:45,142 [] [QuartzScheduler_midPointScheduler-ilstu-midpoint-dev-02_ClusterManager] INFO (org.quartz.impl.jdbcjobstore.JobStoreTX): ClusterManager: Scanning for instance "ilstu-midpoint-dev-01"'s failed in-progress jobs. 2022-10-03 20:32:45,164 [] [QuartzScheduler_midPointScheduler-ilstu-midpoint-dev-02_ClusterManager] INFO (org.quartz.impl.jdbcjobstore.JobStoreTX): ClusterManager: ......Scheduled 1 recoverable job(s) for recovery. 2022-10-03 20:32:45,193 [] [midPointScheduler_Worker-9] INFO (com.evolveum.midpoint.task.quartzimpl.run.JobExecutor): Task Task(id:1664827507317-0-1, name:identityTable (livesynch), oid:9d446a2e-0819-4041-9f66-bf716d76b084) is recovering 2022-10-03 20:32:45,193 [] [midPointScheduler_Worker-9] INFO (com.evolveum.midpoint.task.quartzimpl.run.JobExecutor): Clearing executing node information (was: ilstu-midpoint-dev-01) for Task(id:1664827507317-0-1, name:identityTable (livesynch), oid:9d446a2e-0819-4041-9f66-bf716d76b084) 2022-10-03 20:32:45,194 [] [midPointScheduler_Worker-9] INFO (com.evolveum.midpoint.task.quartzimpl.run.JobExecutor): Fixing execution state from RUNNING to RUNNABLE for Task(id:1664827507317-0-1, name:identityTable (livesynch), oid:9d446a2e-0819-4041-9f66-bf716d76b084) 2022-10-03 20:32:45,226 [] [midPointScheduler_Worker-9] INFO (com.evolveum.midpoint.task.quartzimpl.run.JobExecutor): Recovering resilient task Task(id:1664827507317-0-1, name:identityTable (livesynch), oid:9d446a2e-0819-4041-9f66-bf716d76b084) 2022-10-03 20:32:45,740 [REPOSITORY] [midPointScheduler_Worker-9] ERROR (com.evolveum.midpoint.repo.common.activity.run.ActivityRunResult): Unhandled exception in LiveSyncActivityRun{act=StandaloneActivity{path=''}}. com.evolveum.midpoint.util.exception.SystemException: Bucket PCV(4531):[PP({.../common/common-3}sequentialNumber):[PPV(Integer:1)], PP({.../common/common-3}state):[PPV(WorkBucketStateType:COMPLETE)], PP({.../common/common-3}content):[PPV(NullWorkBucketContentType:com.evolveum.midpoint.xml.ns._public.common.common_3.NullWorkBucketContentType@6e57bf8b[])]] is not ready at com.evolveum.midpoint.repo.sqale.SqaleServiceBase.handledGeneralException(SqaleServiceBase.java:88) at com.evolveum.midpoint.repo.sqale.SqaleRepositoryService.modifyObjectDynamically(SqaleRepositoryService.java:521) at com.evolveum.midpoint.repo.sqale.SqaleRepositoryService$$FastClassBySpringCGLIB$$11fc463.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at com.evolveum.midpoint.util.aspect.MidpointInterceptor.invoke(MidpointInterceptor.java:32) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708) at com.evolveum.midpoint.repo.sqale.SqaleRepositoryService$$EnhancerBySpringCGLIB$$e81114e0.modifyObjectDynamically() at com.evolveum.midpoint.repo.common.activity.run.buckets.CompleteBucketOperation.execute(CompleteBucketOperation.java:60) at com.evolveum.midpoint.repo.common.activity.run.buckets.BucketingManager.completeWorkBucket(BucketingManager.java:79) at com.evolveum.midpoint.repo.common.activity.run.IterativeActivityRun.completeWorkBucketAndUpdateStatistics(IterativeActivityRun.java:331) at com.evolveum.midpoint.repo.common.activity.run.IterativeActivityRun.processSingleBucket(IterativeActivityRun.java:429) at com.evolveum.midpoint.repo.common.activity.run.IterativeActivityRun.processOrAnalyzeOrSkipSingleBucket(IterativeActivityRun.java:370) at com.evolveum.midpoint.repo.common.activity.run.IterativeActivityRun.doRun(IterativeActivityRun.java:230) at com.evolveum.midpoint.repo.common.activity.run.IterativeActivityRun.runLocally(IterativeActivityRun.java:186) at com.evolveum.midpoint.repo.common.activity.run.LocalActivityRun.runInternal(LocalActivityRun.java:81) at com.evolveum.midpoint.repo.common.activity.run.AbstractActivityRun.runTreatingExceptions(AbstractActivityRun.java:253) at com.evolveum.midpoint.repo.common.activity.run.AbstractActivityRun.run(AbstractActivityRun.java:215) at com.evolveum.midpoint.repo.common.activity.run.task.ActivityBasedTaskRun.run(ActivityBasedTaskRun.java:82) at com.evolveum.midpoint.repo.common.activity.run.task.ActivityBasedTaskHandler.run(ActivityBasedTaskHandler.java:91) at com.evolveum.midpoint.task.quartzimpl.run.HandlerExecutor.executeHandler(HandlerExecutor.java:37) at com.evolveum.midpoint.task.quartzimpl.run.TaskCycleExecutor.executeHandler(TaskCycleExecutor.java:134) at com.evolveum.midpoint.task.quartzimpl.run.TaskCycleExecutor.executeTaskCycleRun(TaskCycleExecutor.java:127) at com.evolveum.midpoint.task.quartzimpl.run.TaskCycleExecutor.executeRecurringTask(TaskCycleExecutor.java:97) at com.evolveum.midpoint.task.quartzimpl.run.TaskCycleExecutor.execute(TaskCycleExecutor.java:70) at com.evolveum.midpoint.task.quartzimpl.run.JobExecutor.executeHandler(JobExecutor.java:156) at com.evolveum.midpoint.task.quartzimpl.run.JobExecutor.executeInternal(JobExecutor.java:125) at com.evolveum.midpoint.task.quartzimpl.run.JobExecutor.execute(JobExecutor.java:68) at org.quartz.core.JobRunShell.run(JobRunShell.java:202) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:588) Caused by: java.lang.IllegalStateException: Bucket PCV(4531):[PP({.../common/common-3}sequentialNumber):[PPV(Integer:1)], PP({.../common/common-3}state):[PPV(WorkBucketStateType:COMPLETE)], PP({.../common/common-3}content):[PPV(NullWorkBucketContentType:com.evolveum.midpoint.xml.ns._public.common.common_3.NullWorkBucketContentType@6e57bf8b[])]] is not ready at com.evolveum.midpoint.util.MiscUtil.stateCheck(MiscUtil.java:1084) at com.evolveum.midpoint.repo.common.activity.run.buckets.BucketOperation.checkBucketReadyOrDelegated(BucketOperation.java:176) at com.evolveum.midpoint.repo.common.activity.run.buckets.CompleteBucketOperation.computeCompletionModifications(CompleteBucketOperation.java:74) at com.evolveum.midpoint.repo.sqale.SqaleRepositoryService.executeModifyObjectDynamically(SqaleRepositoryService.java:559) at com.evolveum.midpoint.repo.sqale.SqaleRepositoryService.modifyObjectDynamically(SqaleRepositoryService.java:515) ... 32 common frames omitted 2022-10-03 20:32:45,946 [] [midPointScheduler_Worker-9] INFO (com.evolveum.midpoint.task.quartzimpl.run.JobExecutor): Task encountered permanent error. Suspending it. Task = Task(id:1664827507317-0-1, name:identityTable (livesynch), oid:9d446a2e-0819-4041-9f66-bf716d76b084) 2022-10-03 20:32:45,947 [] [midPointScheduler_Worker-9] INFO (com.evolveum.midpoint.task.quartzimpl.tasks.SuspendAndDeleteHelper): Suspending task Task(id:1664827507317-0-1, name:identityTable (livesynch), oid:9d446a2e-0819-4041-9f66-bf716d76b084); do not stop tasks. 2022-10-03 20:46:39,706 [] [QuartzScheduler_midPointScheduler-ilstu-midpoint-dev-02_ClusterManager] WARN (org.quartz.impl.jdbcjobstore.JobStoreTX): This scheduler instance (ilstu-midpoint-dev-02) is still active but was recovered by another instance in the cluster. This may cause inconsistent behavior. 2022-10-03 21:12:18,516 [] [QuartzScheduler_midPointScheduler-ilstu-midpoint-dev-02_ClusterManager] INFO (org.quartz.impl.jdbcjobstore.JobStoreTX): ClusterManager: detected 1 failed or restarted instances. 2022-10-03 21:12:18,516 [] [QuartzScheduler_midPointScheduler-ilstu-midpoint-dev-02_ClusterManager] INFO (org.quartz.impl.jdbcjobstore.JobStoreTX): ClusterManager: Scanning for instance "ilstu-midpoint-dev-01"'s failed in-progress jobs.