[midPoint-git] [Evolveum/midpoint] 1f2391: Make TestLdapMassive work

mederly noreply at github.com
Sat Sep 25 00:38:24 CEST 2021 

  Branch: refs/heads/feature/mid-7053
  Home:   https://github.com/Evolveum/midpoint
  Commit: 1f239183882fd874352f555b19c0eda508de56a5
      https://github.com/Evolveum/midpoint/commit/1f239183882fd874352f555b19c0eda508de56a5
  Author: Pavol Mederly <mederly at evolveum.com>
  Date:   2021-09-23 (Thu, 23 Sep 2021)

  Changed paths:
    M testing/story/src/test/resources/ldap/sync-massive/resource-opendj-bad.xml

  Log Message:
  -----------
  Make TestLdapMassive work

This commit TEMPORARILY reverts a change that was made in
58315473ee865e950e4f96dd171d919d83ce35d1 (removing resource.namespace).

That change, although correct in itself, made TestLdapMassive.test210
start failing. Perhaps the test itself is broken, and its green status
was only by chance: because of an error in dj namespace declaration
in resource definition.

To do: analyze and maybe fix the test.


  Commit: 1ac7f7757cf07d3f3fe9c7bd98baf596b0cf7a11
      https://github.com/Evolveum/midpoint/commit/1ac7f7757cf07d3f3fe9c7bd98baf596b0cf7a11
  Author: Pavol Mederly <mederly at evolveum.com>
  Date:   2021-09-23 (Thu, 23 Sep 2021)

  Changed paths:
    M infra/schema/src/main/java/com/evolveum/midpoint/schema/util/ObjectTypeUtil.java
    M infra/schema/src/main/java/com/evolveum/midpoint/schema/util/ResourceTypeUtil.java
    M model/model-api/src/main/java/com/evolveum/midpoint/model/api/context/ModelContext.java
    M model/model-common/src/main/java/com/evolveum/midpoint/model/common/ArchetypeManager.java
    M model/model-common/src/test/java/com/evolveum/midpoint/model/common/mapping/TestMappingMetadata.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/ModelBeans.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/controller/ModelInteractionServiceImpl.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/expr/MidpointFunctionsImpl.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/AssignmentCollector.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/Clockwork.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/ClockworkAuditHelper.java
    A model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/ClockworkClick.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/ClockworkMedic.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/ContextFactory.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/EvaluatedPolicyRuleImpl.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/LensContext.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/LensContextPlaceholder.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/LensElementContext.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/LensFocusContext.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/LensObjectDeltaOperation.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/LensOwnerResolver.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/LensProjectionContext.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/LensUtil.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/ObjectDeltaWaves.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/OperationalDataManager.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/PersonaProcessor.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/executor/DeltaExecution.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/executor/FocusChangeExecution.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/executor/ProjectionChangeExecution.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/projector/ContextLoader.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/projector/DependencyProcessor.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/projector/ProjectionValuesProcessor.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/projector/Projector.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/projector/ReconciliationProcessor.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/projector/focus/AssignmentHolderProcessor.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/projector/focus/AssignmentProcessor.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/projector/focus/AssignmentTripleEvaluator.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/projector/focus/FocusLifecycleProcessor.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/projector/focus/InboundMappingsEvaluation.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/projector/focus/IterationHelper.java
    A model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/projector/loader/ContextLoadOperation.java
    A model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/projector/loader/FocusLoadOperation.java
    A model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/projector/loader/MissingShadowContextRefresher.java
    A model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/projector/loader/ProjectionFullLoadOperation.java
    A model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/projector/loader/ProjectionUpdateOperation.java
    A model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/projector/loader/ProjectionsLoadOperation.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/projector/policy/evaluators/ConstraintEvaluatorHelper.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/projector/util/ProcessorMethodRef.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/security/SecurityHelper.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/sync/SynchronizationServiceImpl.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/sync/action/AddFocusAction.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/util/ModelImplUtils.java
    M model/model-impl/src/test/java/com/evolveum/midpoint/model/impl/AbstractModelImplementationIntegrationTest.java
    M model/model-impl/src/test/java/com/evolveum/midpoint/model/impl/lens/TestAssignmentProcessor2.java
    M model/model-impl/src/test/java/com/evolveum/midpoint/model/impl/lens/TestClockwork.java
    M model/model-impl/src/test/java/com/evolveum/midpoint/model/impl/sync/TestSynchronizationService.java
    M model/model-intest/src/test/java/com/evolveum/midpoint/model/intest/AbstractConfiguredModelIntegrationTest.java
    M model/model-intest/src/test/java/com/evolveum/midpoint/model/intest/AbstractInitializedModelIntegrationTest.java
    M model/model-intest/src/test/java/com/evolveum/midpoint/model/intest/TestMemberRecompute.java
    M model/model-intest/src/test/java/com/evolveum/midpoint/model/intest/rbac/TestRbac.java
    M model/model-intest/src/test/java/com/evolveum/midpoint/model/intest/security/TestSecurityAdvanced.java
    M model/model-intest/src/test/java/com/evolveum/midpoint/model/intest/sync/TestRecomputeTask.java
    A model/model-intest/src/test/resources/common/metarole-recompute-members.xml
    M model/model-intest/src/test/resources/common/role-empty.xml
    M model/model-intest/src/test/resources/common/role-judge.xml
    M model/model-test/src/main/java/com/evolveum/midpoint/model/test/AbstractModelIntegrationTest.java
    M model/workflow-impl/src/main/java/com/evolveum/midpoint/wf/impl/execution/CaseOperationExecutionTaskHandler.java
    M model/workflow-impl/src/main/java/com/evolveum/midpoint/wf/impl/util/MiscHelper.java
    M repo/repo-common/src/main/java/com/evolveum/midpoint/repo/common/expression/ExpressionUtil.java
    M testing/story/src/test/java/com/evolveum/midpoint/testing/story/TestManyThreads.java
    M testing/story/src/test/resources/system-perf/task-import.vm.xml
    M testing/story/src/test/resources/system-perf/task-recomputation.vm.xml

  Log Message:
  -----------
  Merge branch 'feature/mid-7053'


  Commit: 9a94b45216e643afa9456e5a46adfe215d3c42ad
      https://github.com/Evolveum/midpoint/commit/9a94b45216e643afa9456e5a46adfe215d3c42ad
  Author: Pavol Mederly <mederly at evolveum.com>
  Date:   2021-09-25 (Sat, 25 Sep 2021)

  Changed paths:
    M infra/schema/src/main/java/com/evolveum/midpoint/schema/ObjectTreeDeltas.java
    M infra/schema/src/main/java/com/evolveum/midpoint/schema/ResourceShadowDiscriminator.java
    M model/certification-impl/src/main/java/com/evolveum/midpoint/certification/impl/AccCertTimedActionTriggerHandler.java
    M model/certification-impl/src/main/java/com/evolveum/midpoint/certification/impl/AccessCertificationCampaignReiterationTriggerHandler.java
    M model/certification-impl/src/main/java/com/evolveum/midpoint/certification/impl/AccessCertificationCloseStageApproachingTriggerHandler.java
    M model/certification-impl/src/main/java/com/evolveum/midpoint/certification/impl/AccessCertificationCloseStageTriggerHandler.java
    M model/model-api/src/main/java/com/evolveum/midpoint/model/api/context/ModelElementContext.java
    M model/model-api/src/main/java/com/evolveum/midpoint/model/api/context/ModelProjectionContext.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/controller/ModelController.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/Clockwork.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/ClockworkClick.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/ContextFactory.java
    A model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/DeltaModifier.java
    A model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/ElementState.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/LensContext.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/LensContextPlaceholder.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/LensElementContext.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/LensFocusContext.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/LensProjectionContext.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/LensUtil.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/ObjectDeltaWaves.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/PolicyRulesContext.java
    A model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/RememberedElementState.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/construction/ResourceObjectConstruction.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/executor/ProjectionChangeExecution.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/projector/ActivationProcessor.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/projector/ConsolidationProcessor.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/projector/ContextLoader.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/projector/ProjectionValuesProcessor.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/projector/ReconciliationProcessor.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/projector/ShadowConstraintsChecker.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/projector/focus/AssignmentHolderProcessor.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/projector/focus/InboundMappingsEvaluation.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/projector/focus/IterationHelper.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/projector/loader/ContextLoadOperation.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/projector/loader/FocusLoadOperation.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/projector/loader/MissingShadowContextRefresher.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/projector/loader/ProjectionFullLoadOperation.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/projector/loader/ProjectionUpdateOperation.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/projector/loader/ProjectionsLoadOperation.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/projector/mappings/MappingEvaluator.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/projector/mappings/ProjectionMappingLoader.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/sync/SynchronizationServiceImpl.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/sync/action/LinkAction.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/tasks/scanner/FocusValidityScanPartialExecution.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/trigger/CompletedTaskCleanupTriggerHandler.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/trigger/RecomputeTriggerHandler.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/trigger/ShadowReconcileTriggerHandler.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/trigger/SingleTriggerHandler.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/trigger/TriggerScanItemProcessor.java
    M model/model-impl/src/test/java/com/evolveum/midpoint/model/impl/AbstractModelImplementationIntegrationTest.java
    M model/model-impl/src/test/java/com/evolveum/midpoint/model/impl/lens/AbstractTestProjectorPersona.java
    M model/model-impl/src/test/java/com/evolveum/midpoint/model/impl/lens/TestAbstractAssignmentEvaluator.java
    M model/model-impl/src/test/java/com/evolveum/midpoint/model/impl/lens/TestAssignmentProcessor2.java
    M model/model-impl/src/test/java/com/evolveum/midpoint/model/impl/lens/TestClockwork.java
    M model/model-impl/src/test/java/com/evolveum/midpoint/model/impl/lens/TestPolicyRules.java
    M model/model-impl/src/test/java/com/evolveum/midpoint/model/impl/lens/TestPolicyRules2.java
    M model/model-impl/src/test/java/com/evolveum/midpoint/model/impl/lens/TestProjector.java
    M model/model-impl/src/test/java/com/evolveum/midpoint/model/impl/lens/TestProjectorRoleEntitlement.java
    M model/model-intest/src/test/java/com/evolveum/midpoint/model/intest/multi/TestMultiAccount.java
    M model/model-intest/src/test/java/com/evolveum/midpoint/model/intest/util/MockTriggerHandler.java
    M model/model-intest/src/test/resources/scripthooks/system-configuration-primary-delta-hook.xml
    M model/model-intest/src/test/resources/sync/task-dummy-emerald-livesync.xml
    M model/workflow-impl/src/main/java/com/evolveum/midpoint/wf/impl/execution/CaseOperationExecutionTaskHandler.java
    M model/workflow-impl/src/main/java/com/evolveum/midpoint/wf/impl/execution/LensContextHelper.java
    M model/workflow-impl/src/main/java/com/evolveum/midpoint/wf/impl/processors/primary/PrimaryChangeProcessor.java
    M model/workflow-impl/src/main/java/com/evolveum/midpoint/wf/impl/util/MiscHelper.java
    M provisioning/provisioning-impl/src/main/java/com/evolveum/midpoint/provisioning/impl/shadows/DefinitionsHelper.java

  Log Message:
  -----------
  Eliminate repeated delta/object recomputations

Each time current or summary delta (including relative and absolute ODO)
were obtained from LensElementContext, full re-creation of these deltas
from primary and secondary deltas occurred. This accounted for large
portion of time spent in the clockwork for some use cases.

This commit partly eliminates these recomputations: the computable
parts of the element state (mainly: current/summary delta, new object)
are cached, with invalidation bound to the changes in "source" data.

Work in progress. The performance can be improved further
by updating current/summary delta "on the fly", along with updates
to secondary deltas. It is also questionable if it is better to
recompute new object when accessed (and invalid), or to use original
approach based on LensElementContext#recompute method, or to update
it "on the fly" along with current/summary delta.

However, even this partial work saved 10-15% clockwork time
for scenarios with 100 assignments, and 30-35% time for 500 assignments.

Unrelated minor changes:

1. ConsolidationProcessor no longer drops primary and secondary deltas
for projections that were deleted.

2. Synchronization intent was moved from LensElementContext
to LensProjectionContext.

Related to MID-7053.


Compare: https://github.com/Evolveum/midpoint/compare/d98dea978b93...9a94b45216e6

More information about the midPoint-svn mailing list