[midPoint-git] [Evolveum/midpoint] d561ad: Improve performance reporting
mederly
noreply at github.com
Thu Feb 20 13:32:58 CET 2025
Branch: refs/heads/support-4.9
Home: https://github.com/Evolveum/midpoint
Commit: d561ad84a4357c5510c49f6af47340989c3528c6
https://github.com/Evolveum/midpoint/commit/d561ad84a4357c5510c49f6af47340989c3528c6
Author: Pavol Mederly <mederly at evolveum.com>
Date: 2025-02-20 (Thu, 20 Feb 2025)
Changed paths:
M infra/schema/src/main/java/com/evolveum/midpoint/schema/result/OperationConstants.java
M infra/schema/src/main/java/com/evolveum/midpoint/schema/result/OperationResult.java
M infra/schema/src/main/java/com/evolveum/midpoint/schema/statistics/AbstractStatisticsPrinter.java
A infra/schema/src/main/java/com/evolveum/midpoint/schema/statistics/BasicComponentStructure.java
A infra/schema/src/main/java/com/evolveum/midpoint/schema/statistics/ComponentsPerformanceComputer.java
A infra/schema/src/main/java/com/evolveum/midpoint/schema/statistics/ComponentsPerformanceInformationPrinter.java
A infra/schema/src/main/java/com/evolveum/midpoint/schema/statistics/ComponentsPerformanceInformationUtil.java
M infra/schema/src/main/java/com/evolveum/midpoint/schema/statistics/OperationsPerformanceInformationPrinter.java
M infra/schema/src/main/java/com/evolveum/midpoint/schema/statistics/OperationsPerformanceInformationUtil.java
M infra/schema/src/main/java/com/evolveum/midpoint/schema/statistics/RepositoryPerformanceInformationPrinter.java
M infra/schema/src/main/java/com/evolveum/midpoint/schema/traces/OpType.java
M infra/schema/src/main/java/com/evolveum/midpoint/schema/util/task/TaskOperationStatsUtil.java
M infra/schema/src/main/resources/xml/ns/public/common/common-core-3.xsd
M infra/schema/src/main/resources/xml/ns/public/common/common-tasks-3.xsd
M infra/test-util/src/main/java/com/evolveum/midpoint/test/util/TestReportUtil.java
M provisioning/provisioning-impl/src/main/java/com/evolveum/midpoint/provisioning/impl/ProvisioningServiceImpl.java
M provisioning/provisioning-impl/src/main/java/com/evolveum/midpoint/provisioning/impl/resourceobjects/ResourceObjectSearchOperation.java
M provisioning/provisioning-impl/src/main/java/com/evolveum/midpoint/provisioning/impl/resources/ConnectorManager.java
M provisioning/provisioning-impl/src/main/java/com/evolveum/midpoint/provisioning/impl/shadows/ShadowGetOperation.java
M testing/story/src/test/java/com/evolveum/midpoint/testing/story/sysperf/TestSystemPerformance.java
Log Message:
-----------
Improve performance reporting
1. "Own time" was added to individual operations performance
information.
2. Based on this information, basic component-level reporting was
introduced. See BasicComponentStructure class.
3. Some bugs in operation result handling in connector discovery methods
(in provisioning) were fixed. They led to obviously wrong (negative)
own times for the respective operations.
4. Operation name for ResourceObjectSearchOperation#handleObjectFound
was fixed, to be correctly categorized as belonging to provisioning.
Work in progress. To be tested.
(cherry picked from commit 0b54cec556912135bf67b68eceaa6e18558b43d9)
Commit: 0c5c7f45318e87761e43ccff5e465158b27cf73e
https://github.com/Evolveum/midpoint/commit/0c5c7f45318e87761e43ccff5e465158b27cf73e
Author: Pavol Mederly <mederly at evolveum.com>
Date: 2025-02-20 (Thu, 20 Feb 2025)
Changed paths:
M testing/story/src/test/resources/system-perf/system-configuration.xml
Log Message:
-----------
Temporarily disable MD in TestSystemPerformance
(cherry picked from commit 5e51e12d6756b7fa7a8a24361a3eee9682bce845)
Commit: febf997fe3ca8a512f1bb60ff227f5cb95d10265
https://github.com/Evolveum/midpoint/commit/febf997fe3ca8a512f1bb60ff227f5cb95d10265
Author: Pavol Mederly <mederly at evolveum.com>
Date: 2025-02-20 (Thu, 20 Feb 2025)
Changed paths:
M infra/schema/src/main/java/com/evolveum/midpoint/schema/processor/ShadowAssociationValue.java
M infra/schema/src/main/java/com/evolveum/midpoint/schema/processor/ShadowReferenceAttributeValue.java
M model/model-common/src/main/java/com/evolveum/midpoint/model/common/mapping/AbstractMappingImpl.java
Log Message:
-----------
Improve deleteOwnYieldFromNonNegativeValues perf
This method contributes to the performance penalty related to value
metadata (even if limited to assignments). Here we try to improve it.
However, the effect is little.
(cherry picked from commit 37b648a6d7fca11a2436baf277fbb8675c63b48f)
Commit: 5a50dcfe09aa4735434024c6f08d4ea7f3608a5b
https://github.com/Evolveum/midpoint/commit/5a50dcfe09aa4735434024c6f08d4ea7f3608a5b
Author: Pavol Mederly <mederly at evolveum.com>
Date: 2025-02-20 (Thu, 20 Feb 2025)
Changed paths:
M infra/schema/src/main/java/com/evolveum/midpoint/schema/ObjectHandler.java
M infra/schema/src/main/java/com/evolveum/midpoint/schema/ResultHandler.java
M infra/schema/src/main/java/com/evolveum/midpoint/schema/result/OperationResult.java
M infra/schema/src/main/java/com/evolveum/midpoint/schema/statistics/BasicComponentStructure.java
M infra/schema/src/main/java/com/evolveum/midpoint/schema/statistics/ComponentsPerformanceInformationUtil.java
M infra/test-util/src/main/java/com/evolveum/midpoint/test/util/OperationResultAssert.java
M model/model-api/src/main/java/com/evolveum/midpoint/model/api/hooks/HookRegistry.java
M model/model-common/src/main/java/com/evolveum/midpoint/model/common/mapping/metadata/MetadataItemProcessingSpecImpl.java
M model/model-common/src/main/java/com/evolveum/midpoint/model/common/stringpolicy/AbstractValuePolicyOriginResolver.java
M model/model-common/src/main/java/com/evolveum/midpoint/model/common/stringpolicy/ValueChecker.java
M model/model-common/src/main/java/com/evolveum/midpoint/model/common/util/ObjectTemplateIncludeProcessor.java
M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/ModelObjectResolver.java
M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/controller/ModelController.java
M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/correlator/items/ItemsCorrelator.java
M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/LensOwnerResolver.java
M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/projector/focus/AutoAssignMappingCollector.java
M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/projector/policy/scriptExecutor/LinkSourceFinder.java
M model/model-intest/src/test/java/com/evolveum/midpoint/model/intest/misc/AbstractMiscTest.java
M model/model-intest/src/test/java/com/evolveum/midpoint/model/intest/misc/TestMisc.java
M provisioning/provisioning-impl/src/main/java/com/evolveum/midpoint/provisioning/impl/operations/ProvisioningSearchLikeOperation.java
M provisioning/provisioning-impl/src/main/java/com/evolveum/midpoint/provisioning/impl/resourceobjects/ResourceObjectSearchOperation.java
M provisioning/provisioning-impl/src/main/java/com/evolveum/midpoint/provisioning/impl/shadows/ShadowSearchLikeOperation.java
M provisioning/provisioning-impl/src/main/java/com/evolveum/midpoint/provisioning/impl/shadows/ShadowsFacade.java
M provisioning/provisioning-impl/src/main/java/com/evolveum/midpoint/provisioning/impl/shadows/manager/ShadowFinder.java
M provisioning/provisioning-impl/src/main/java/com/evolveum/midpoint/provisioning/impl/shadows/task/MultiPropagationActivityRun.java
M provisioning/ucf-api/src/main/java/com/evolveum/midpoint/provisioning/ucf/api/ConnectorInstance.java
M provisioning/ucf-impl-connid/src/main/java/com/evolveum/midpoint/provisioning/ucf/impl/connid/ConnectorInstanceConnIdImpl.java
M provisioning/ucf-impl-connid/src/main/java/com/evolveum/midpoint/provisioning/ucf/impl/connid/SearchExecutor.java
M repo/repo-api/src/main/java/com/evolveum/midpoint/repo/api/RepositoryService.java
M repo/repo-cache/src/main/java/com/evolveum/midpoint/repo/cache/RepositoryCache.java
M repo/repo-cache/src/main/java/com/evolveum/midpoint/repo/cache/handlers/SearchOpHandler.java
M repo/repo-common/src/main/java/com/evolveum/midpoint/repo/common/RepoObjectResolver.java
M repo/repo-common/src/main/java/com/evolveum/midpoint/repo/common/activity/run/processing/ContainerableProcessingRequest.java
M repo/repo-common/src/main/java/com/evolveum/midpoint/repo/common/activity/run/processing/ItemProcessingGatekeeper.java
M repo/repo-common/src/main/java/com/evolveum/midpoint/repo/common/activity/run/processing/ProcessingCoordinator.java
M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/SqaleRepositoryService.java
M repo/repo-sql-impl/src/main/java/com/evolveum/midpoint/repo/sql/SqlRepositoryServiceImpl.java
M repo/repo-sql-impl/src/main/java/com/evolveum/midpoint/repo/sql/helpers/ObjectRetriever.java
M repo/repo-test-util/src/main/java/com/evolveum/midpoint/test/AbstractIntegrationTest.java
A repo/repo-test-util/src/main/java/com/evolveum/midpoint/test/asserter/ComponentsPerformanceInformationAsserter.java
A repo/repo-test-util/src/main/java/com/evolveum/midpoint/test/asserter/OperationsPerformanceInformationAsserter.java
M repo/task-quartz-impl/src/main/java/com/evolveum/midpoint/task/quartzimpl/quartz/TaskSynchronizer.java
Log Message:
-----------
Fix performance reporting by components
This commit fixes the reporting of performance by components, especially
concentrating on handling objects found in the iterative search methods.
See also the docs added to OperationResult (principle #4) and
ResultHandler#providingOwnOperationResult.
Work in progress.
Related to MID-10446.
(cherry picked from commit 5f21b43764d317fe3d29d691e9b30d260b2456d3)
Commit: 7fe99817a4fa406ae37a620cb87fc6630949ec4a
https://github.com/Evolveum/midpoint/commit/7fe99817a4fa406ae37a620cb87fc6630949ec4a
Author: Pavol Mederly <mederly at evolveum.com>
Date: 2025-02-20 (Thu, 20 Feb 2025)
Changed paths:
M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/controller/ModelController.java
M model/model-intest/src/test/java/com/evolveum/midpoint/model/intest/misc/TestMisc.java
M model/model-intest/src/test/java/com/evolveum/midpoint/model/intest/sync/TestImportRecon.java
A model/model-intest/src/test/resources/misc/resource-dummy-perf.xml
M provisioning/provisioning-api/src/main/java/com/evolveum/midpoint/provisioning/api/ProvisioningService.java
M provisioning/provisioning-impl/src/main/java/com/evolveum/midpoint/provisioning/impl/ProvisioningServiceImpl.java
M provisioning/provisioning-impl/src/main/java/com/evolveum/midpoint/provisioning/impl/resourceobjects/ResourceObjectConverter.java
M provisioning/provisioning-impl/src/main/java/com/evolveum/midpoint/provisioning/impl/resourceobjects/ResourceObjectSearchOperation.java
M provisioning/provisioning-impl/src/main/java/com/evolveum/midpoint/provisioning/impl/shadows/ShadowSearchLikeOperation.java
M provisioning/provisioning-impl/src/main/java/com/evolveum/midpoint/provisioning/impl/shadows/ShadowsFacade.java
M provisioning/provisioning-impl/src/main/java/com/evolveum/midpoint/provisioning/impl/shadows/manager/ShadowFinder.java
M provisioning/ucf-impl-connid/src/main/java/com/evolveum/midpoint/provisioning/ucf/impl/connid/ConnectorInstanceConnIdImpl.java
M provisioning/ucf-impl-connid/src/main/java/com/evolveum/midpoint/provisioning/ucf/impl/connid/SearchExecutor.java
M repo/repo-cache/src/main/java/com/evolveum/midpoint/repo/cache/RepositoryCache.java
M repo/repo-cache/src/main/java/com/evolveum/midpoint/repo/cache/handlers/GetObjectOpHandler.java
M repo/repo-test-util/src/main/java/com/evolveum/midpoint/test/asserter/OperationsPerformanceInformationAsserter.java
Log Message:
-----------
Add more performance reporting tests
Now we test also searchObjectsIteratively and getObject operations
that are related to shadows (both fetch and noFetch mode).
Also, this commit fixes failing TestImportRecon by removing
assertions on operation results that were out of date (and not
strictly necessary).
Related to MID-10446.
(cherry picked from commit 2ba5b01bf4c62268c4d9dd5796888e29a8467649)
Commit: 1466944133c5219890bd210e0757389bb65ca84c
https://github.com/Evolveum/midpoint/commit/1466944133c5219890bd210e0757389bb65ca84c
Author: Pavol Mederly <mederly at evolveum.com>
Date: 2025-02-20 (Thu, 20 Feb 2025)
Changed paths:
M testing/story/src/test/java/com/evolveum/midpoint/testing/story/sysperf/TestSystemPerformance.java
Log Message:
-----------
Add dumping representative shadows in perf. test
(cherry picked from commit 80c7357f0b149587c2c000b8d459a3fad3ed3e5d)
Commit: 8aa06888751746b497916701c741b3ebf2c24716
https://github.com/Evolveum/midpoint/commit/8aa06888751746b497916701c741b3ebf2c24716
Author: Pavol Mederly <mederly at evolveum.com>
Date: 2025-02-20 (Thu, 20 Feb 2025)
Changed paths:
M testing/story/src/test/java/com/evolveum/midpoint/testing/story/sysperf/ImportConfiguration.java
A testing/story/src/test/java/com/evolveum/midpoint/testing/story/sysperf/RandomSource.java
M testing/story/src/test/java/com/evolveum/midpoint/testing/story/sysperf/ReconciliationConfiguration.java
M testing/story/src/test/java/com/evolveum/midpoint/testing/story/sysperf/RolesConfiguration.java
M testing/story/src/test/java/com/evolveum/midpoint/testing/story/sysperf/SourceInitializer.java
M testing/story/src/test/java/com/evolveum/midpoint/testing/story/sysperf/SourcesConfiguration.java
M testing/story/src/test/java/com/evolveum/midpoint/testing/story/sysperf/TargetsConfiguration.java
Log Message:
-----------
Make TestSystemPerformance less random
In order to compare performance between various runs more exactly,
this commit reduces the variability of generated data.
Previously, the number of operations varied mainly because the number
of technical roles (and therefore, entitlements) could be different
amount of overlap between randomly-generated and randomly-assigned
business roles.
Now we reduce the randomness by using a generator with a fixed seed.
(cherry picked from commit f6d077c464c921b3b125092580cb906aecad6354)
Commit: faefd8cd375932ef7c55a8ac0a1a8f5c7fff203f
https://github.com/Evolveum/midpoint/commit/faefd8cd375932ef7c55a8ac0a1a8f5c7fff203f
Author: Pavol Mederly <mederly at evolveum.com>
Date: 2025-02-20 (Thu, 20 Feb 2025)
Changed paths:
M testing/story/src/test/resources/system-perf/system-configuration.xml
Log Message:
-----------
Revert "Temporarily disable MD in TestSystemPerformance"
This reverts commit 0c5c7f45318e87761e43ccff5e465158b27cf73e.
Compare: https://github.com/Evolveum/midpoint/compare/cdf1e60fe457...faefd8cd3759
To unsubscribe from these emails, change your notification settings at https://github.com/Evolveum/midpoint/settings/notifications
More information about the midPoint-svn
mailing list