[midPoint-git] [Evolveum/midpoint] 0a89ab: Make deletion-related recon. thresholds work

Thu Aug 26 10:01:10 CEST 2021

  Branch: refs/heads/master
  Home:   https://github.com/Evolveum/midpoint
  Commit: 0a89ab6ace86eecfc27ea7afc026de0f4314311e
      https://github.com/Evolveum/midpoint/commit/0a89ab6ace86eecfc27ea7afc026de0f4314311e
  Author: Pavol Mederly <mederly at evolveum.com>
  Date:   2021-08-25 (Wed, 25 Aug 2021)

  Changed paths:
    M infra/common/src/main/java/com/evolveum/midpoint/common/SynchronizationUtils.java
    M infra/schema/src/main/resources/xml/ns/public/common/common-core-3.xsd
    M infra/schema/src/main/resources/xml/ns/public/common/common-tasks-3.xsd
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/executor/LinkUpdater.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/sync/SynchronizationServiceImpl.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/sync/tasks/recon/RemainingShadowsActivityExecutionSpecifics.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/tasks/ShadowFetchingPreprocessor.java
    M model/model-intest/src/test/java/com/evolveum/midpoint/model/intest/tasks/TestThresholds.java
    M model/model-intest/src/test/resources/tasks/thresholds/task-reconciliation-simulate-execute-single-thread.xml
    M model/model-intest/src/test/resources/tasks/thresholds/task-reconciliation-simulate-single-thread.xml
    M provisioning/provisioning-impl/src/main/java/com/evolveum/midpoint/provisioning/impl/resourceobjects/EntitlementConverter.java
    M provisioning/provisioning-impl/src/main/java/com/evolveum/midpoint/provisioning/impl/resourceobjects/ResourceObjectConverter.java
    M provisioning/provisioning-impl/src/main/java/com/evolveum/midpoint/provisioning/impl/resourceobjects/ResourceObjectLiveSyncChange.java
    M provisioning/provisioning-impl/src/main/java/com/evolveum/midpoint/provisioning/impl/resourceobjects/ResourceObjectReferenceResolver.java
    M provisioning/provisioning-impl/src/main/java/com/evolveum/midpoint/provisioning/impl/shadows/GetHelper.java
    M provisioning/provisioning-impl/src/main/java/com/evolveum/midpoint/provisioning/impl/shadows/errors/ObjectNotFoundHandler.java
    M repo/repo-common/src/main/java/com/evolveum/midpoint/repo/common/activity/definition/ActivityDefinition.java
    M repo/repo-common/src/main/java/com/evolveum/midpoint/repo/common/activity/definition/ActivityMonitoringDefinition.java
    M repo/repo-common/src/main/java/com/evolveum/midpoint/repo/common/task/IterativeActivityExecution.java
    R repo/repo-common/src/main/java/com/evolveum/midpoint/repo/common/task/TaskExceptionHandlingUtil.java
    M repo/repo-common/src/main/java/com/evolveum/midpoint/repo/common/task/task/GenericTaskHandler.java
    M repo/repo-sql-impl-test/src/test/java/com/evolveum/midpoint/repo/sql/ModifyTest.java
    M repo/task-api/src/main/java/com/evolveum/midpoint/task/api/ExecutionSupport.java
    M repo/task-api/src/main/java/com/evolveum/midpoint/task/api/TaskUtil.java

  Log Message:
  -----------
  Make deletion-related recon. thresholds work

1. In order to detect exceptions (like threshold violations) in the
"remaining shadows" reconciliation activity, its approach was changed:
instead of relying on built-in ObjectNotFoundHandler we do handling
of the missing objects ourselves. This ensures that any exceptions
are not buried under layers of OperationResult instances, so they
can be processed in ItemProcessingGatekeeper.

2. In order to skip handling ObjectNotFoundExceptions in GET operations,
experimental "Forced exception" error reporting mode
in GetOperationOptions was introduced. This disables
ObjectNotFoundHandler.

3. When throwing ObjectNotFoundException for a get shadow op, we try
to provide an OID. This is to allow the client to distinguish the
unavailability of the requested shadow from unrelated problems.

!!! BEHAVIOR CHANGES in points 4-5 !!!

4. Full synchronization timestamp is NOT set for simulation mode.
Only in execution mode. This also means that when fetching remaining
shadows in recon, we use fullSynchronizationTimestamp in execution mode,
and synchronizationTimestamp in all other modes.

5. Shadows for missing objects are NOT marked as tombstone
(dead/not-existing) in simulation or dry run mode. This is to avoid
loss of this information when it comes to the real execution.

Other changes:

6. Monitoring configuration is now tailorable for sub-activities.

Work in progress. Some tests may fail.

  Commit: c8538334ea8e58a01a93d451598b573af3379fb1
      https://github.com/Evolveum/midpoint/commit/c8538334ea8e58a01a93d451598b573af3379fb1
  Author: Pavol Mederly <mederly at evolveum.com>
  Date:   2021-08-25 (Wed, 25 Aug 2021)

  Changed paths:
    M infra/schema/src/main/resources/xml/ns/public/common/common-tasks-3.xsd
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/projector/policy/PolicyRuleCounterUpdater.java
    M model/model-intest/src/test/java/com/evolveum/midpoint/model/intest/tasks/TestThresholds.java
    M model/model-intest/src/test/resources/tasks/thresholds/resource-dummy-source.xml
    R model/model-intest/src/test/resources/tasks/thresholds/role-modify-5.xml
    A model/model-intest/src/test/resources/tasks/thresholds/role-modify-cost-center-5.xml
    A model/model-intest/src/test/resources/tasks/thresholds/role-modify-full-name-5.xml
    M repo/repo-test-util/src/main/java/com/evolveum/midpoint/test/asserter/ActivityStateAsserter.java
    M repo/task-api/src/main/java/com/evolveum/midpoint/task/api/ExecutionSupport.java

  Log Message:
  -----------
  Separate simulation and execution rule counters

For composite activities with shared policy rule counters (like
reconciliation) we need to keep counters for simulation
and execution modes separate, to avoid their interference.

  Commit: faaed42d829f492cb8dabf15266b36749415f431
      https://github.com/Evolveum/midpoint/commit/faaed42d829f492cb8dabf15266b36749415f431
  Author: Pavol Mederly <mederly at evolveum.com>
  Date:   2021-08-25 (Wed, 25 Aug 2021)

  Changed paths:
    M model/model-impl/src/test/java/com/evolveum/midpoint/model/impl/lens/TestReconScript.java
    M provisioning/provisioning-impl/src/main/java/com/evolveum/midpoint/provisioning/impl/shadows/errors/ObjectNotFoundHandler.java

  Log Message:
  -----------
  Fix tombstone marking in reconciliation

The remaining shadows reconciliation activity (after recent changes)
did not mark deleted shadows as tombstones. This is now fixed.

  Commit: a5c38c873c9efa08d33ca32d78b5fdff6e89fe0a
      https://github.com/Evolveum/midpoint/commit/a5c38c873c9efa08d33ca32d78b5fdff6e89fe0a
  Author: Pavol Mederly <mederly at evolveum.com>
  Date:   2021-08-26 (Thu, 26 Aug 2021)

  Changed paths:
    M model/model-intest/src/test/java/com/evolveum/midpoint/model/intest/tasks/TestThresholds.java
    A model/model-intest/src/test/java/com/evolveum/midpoint/model/intest/tasks/TestThresholdsMultiNode.java
    A model/model-intest/src/test/java/com/evolveum/midpoint/model/intest/tasks/TestThresholdsSingleTask.java
    A model/model-intest/src/test/java/com/evolveum/midpoint/model/intest/tasks/TestThresholdsSingleTaskMultipleThreads.java
    A model/model-intest/src/test/java/com/evolveum/midpoint/model/intest/tasks/TestThresholdsSingleThread.java
    A model/model-intest/src/test/resources/tasks/thresholds/task-import-execute-multi.xml
    R model/model-intest/src/test/resources/tasks/thresholds/task-import-execute-single-thread.xml
    A model/model-intest/src/test/resources/tasks/thresholds/task-import-execute-single.xml
    A model/model-intest/src/test/resources/tasks/thresholds/task-import-simulate-execute-multi.xml
    R model/model-intest/src/test/resources/tasks/thresholds/task-import-simulate-execute-single-thread.xml
    A model/model-intest/src/test/resources/tasks/thresholds/task-import-simulate-execute-single.xml
    A model/model-intest/src/test/resources/tasks/thresholds/task-import-simulate-multi.xml
    R model/model-intest/src/test/resources/tasks/thresholds/task-import-simulate-single-thread.xml
    A model/model-intest/src/test/resources/tasks/thresholds/task-import-simulate-single.xml
    A model/model-intest/src/test/resources/tasks/thresholds/task-reconciliation-execute-multi.xml
    R model/model-intest/src/test/resources/tasks/thresholds/task-reconciliation-execute-single-thread.xml
    A model/model-intest/src/test/resources/tasks/thresholds/task-reconciliation-execute-single.xml
    A model/model-intest/src/test/resources/tasks/thresholds/task-reconciliation-simulate-execute-multi.xml
    R model/model-intest/src/test/resources/tasks/thresholds/task-reconciliation-simulate-execute-single-thread.xml
    A model/model-intest/src/test/resources/tasks/thresholds/task-reconciliation-simulate-execute-single.xml
    A model/model-intest/src/test/resources/tasks/thresholds/task-reconciliation-simulate-multi.xml
    R model/model-intest/src/test/resources/tasks/thresholds/task-reconciliation-simulate-single-thread.xml
    A model/model-intest/src/test/resources/tasks/thresholds/task-reconciliation-simulate-single.xml
    M repo/repo-common/src/main/java/com/evolveum/midpoint/repo/common/activity/state/ActivityState.java
    M repo/repo-common/src/main/java/com/evolveum/midpoint/repo/common/activity/state/CurrentActivityState.java
    M repo/repo-common/src/main/java/com/evolveum/midpoint/repo/common/task/IterativeActivityExecution.java
    M repo/repo-test-util/src/main/java/com/evolveum/midpoint/test/AbstractIntegrationTest.java
    M repo/repo-test-util/src/main/java/com/evolveum/midpoint/test/asserter/ActivityCounterGroupAsserter.java

  Log Message:
  -----------
  Add multi-threaded and multi-node threshold tests

We also fix thresholds for multi-node simple activities (like import).

  Commit: 1e9d19bb12776528f598b92bd9e9fb87f6003555
      https://github.com/Evolveum/midpoint/commit/1e9d19bb12776528f598b92bd9e9fb87f6003555
  Author: Pavol Mederly <mederly at evolveum.com>
  Date:   2021-08-26 (Thu, 26 Aug 2021)

  Changed paths:
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/sync/tasks/recon/RemainingShadowsActivityExecutionSpecifics.java
    M model/model-intest/src/test/resources/sync/task-reconcile-dummy-single.xml

  Log Message:
  -----------
  Fix remaining shadows reconciliation

"Deleted" shadows that are sent from the remaining shadows activity
to the synchronization service did not have tombstone status, causing
the clockwork to re-link them to their owners. (This bug was introduced
in recent rework of the activity.)

This is now fixed.

  Commit: 26912cc9ae42c75e239eab93b35c76735fa40ad2
      https://github.com/Evolveum/midpoint/commit/26912cc9ae42c75e239eab93b35c76735fa40ad2
  Author: Pavol Mederly <mederly at evolveum.com>
  Date:   2021-08-26 (Thu, 26 Aug 2021)

  Changed paths:
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/controller/DashboardServiceImpl.java
    M model/model-intest/src/test/java/com/evolveum/midpoint/model/intest/security/TestSecurityPrincipal.java
    M model/model-intest/testng-integration-sqale.xml
    M model/report-impl/src/main/java/com/evolveum/midpoint/report/impl/ReportFunctions.java
    M repo/audit-api/src/main/java/com/evolveum/midpoint/audit/api/AuditService.java
    M repo/audit-log-impl/src/main/java/com/evolveum/midpoint/audit/impl/LoggerAuditServiceImpl.java
    M repo/repo-sqale/sql/pgnew-repo-audit.sql
    M repo/repo-sqale/sql/pgnew-repo.sql
    M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/ExtUtils.java
    M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/ExtensionProcessor.java
    M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/SqaleRepositoryService.java
    A repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/audit/SqaleAuditService.java
    A repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/audit/qmodel/MAuditDelta.java
    A repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/audit/qmodel/MAuditEventRecord.java
    A repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/audit/qmodel/MAuditRefValue.java
    A repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/audit/qmodel/QAuditDelta.java
    A repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/audit/qmodel/QAuditDeltaMapping.java
    A repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/audit/qmodel/QAuditEventRecord.java
    A repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/audit/qmodel/QAuditEventRecordMapping.java
    A repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/audit/qmodel/QAuditRefValue.java
    A repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/audit/qmodel/QAuditRefValueMapping.java
    M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/mapping/SqaleTableMapping.java
    M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/qmodel/accesscert/QAccessCertificationCaseMapping.java
    M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/qmodel/ext/MExtItem.java
    M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/qmodel/object/QObjectMapping.java
    M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/qmodel/shadow/QShadowMapping.java
    M repo/repo-sqale/src/test/java/com/evolveum/midpoint/repo/sqale/func/SqaleRepoAddDeleteObjectTest.java
    M repo/repo-sqale/src/test/java/com/evolveum/midpoint/repo/sqale/func/SqaleRepoSearchTest.java
    M repo/repo-sql-impl/src/main/java/com/evolveum/midpoint/repo/sql/SqlAuditServiceImpl.java
    M repo/repo-sql-impl/src/main/java/com/evolveum/midpoint/repo/sql/audit/mapping/QAuditEventRecordMapping.java
    R repo/repo-sql-impl/src/main/java/com/evolveum/midpoint/repo/sql/data/SelectQueryBuilder.java
    R repo/repo-sql-impl/src/main/java/com/evolveum/midpoint/repo/sql/data/SingleSqlQuery.java
    R repo/repo-sql-impl/src/main/java/com/evolveum/midpoint/repo/sql/data/SqlQuery.java
    M repo/repo-sqlbase/src/main/java/com/evolveum/midpoint/repo/sqlbase/RepositoryObjectParseResult.java
    M repo/repo-test-util/src/main/java/com/evolveum/midpoint/test/DummyAuditService.java
    M repo/system-init/src/main/java/com/evolveum/midpoint/init/AuditServiceProxy.java

  Log Message:
  -----------
  Merge branch 'master' into feature/thresholds

Compare: https://github.com/Evolveum/midpoint/compare/4b763b4a1743...26912cc9ae42