[midPoint-git] [Evolveum/midpoint] ada053: Add basic task bucket performance test
mederly
noreply at github.com
Thu Apr 15 13:35:06 CEST 2021
Branch: refs/heads/master
Home: https://github.com/Evolveum/midpoint
Commit: ada05385dd0fda4cc66bef274ec4a3f36a41dde7
https://github.com/Evolveum/midpoint/commit/ada05385dd0fda4cc66bef274ec4a3f36a41dde7
Author: Pavol Mederly <mederly at evolveum.com>
Date: 2021-04-09 (Fri, 09 Apr 2021)
Changed paths:
M infra/schema/src/main/java/com/evolveum/midpoint/schema/statistics/TaskWorkBucketManagementPerformanceInformationUtil.java
M infra/schema/src/main/java/com/evolveum/midpoint/schema/util/task/TaskOperationStatsUtil.java
A infra/schema/src/test/java/com/evolveum/midpoint/schema/misc/TestMeasuringSizes.java
M repo/repo-test-util/src/main/resources/test-config.xml
M repo/task-quartz-impl/src/main/java/com/evolveum/midpoint/task/quartzimpl/statistics/Statistics.java
A testing/story/src/test/java/com/evolveum/midpoint/testing/story/buckets/TestBucketsPerformance.java
A testing/story/src/test/resources/buckets/role-slow.xml
A testing/story/src/test/resources/buckets/system-configuration.xml
A testing/story/src/test/resources/buckets/task-recompute-16-04.xml
A testing/story/src/test/resources/buckets/task-recompute-256-30.xml
Log Message:
-----------
Add basic task bucket performance test
Also:
- added summarization of bucket performance information through trees;
- increased the number of task manager threads in tests to 50;
- added simple code that measures objects parts sizes.
Commit: 75dab94d3fa25654df149cb8fdbbd01ee9f4b39a
https://github.com/Evolveum/midpoint/commit/75dab94d3fa25654df149cb8fdbbd01ee9f4b39a
Author: Pavol Mederly <mederly at evolveum.com>
Date: 2021-04-14 (Wed, 14 Apr 2021)
Changed paths:
M infra/schema/src/main/java/com/evolveum/midpoint/schema/util/task/TaskOperationStatsUtil.java
M repo/repo-api/src/main/java/com/evolveum/midpoint/repo/api/ModifyObjectResult.java
M repo/repo-api/src/main/java/com/evolveum/midpoint/repo/api/RepositoryService.java
M repo/repo-sql-impl-test/src/test/java/com/evolveum/midpoint/repo/sql/ConcurrencyTest.java
M repo/repo-sql-impl-test/src/test/java/com/evolveum/midpoint/repo/sql/ModifyTest.java
M repo/repo-sql-impl/src/main/java/com/evolveum/midpoint/repo/sql/OperationLogger.java
M repo/repo-sql-impl/src/main/java/com/evolveum/midpoint/repo/sql/SqlRepositoryServiceImpl.java
A repo/repo-sql-impl/src/main/java/com/evolveum/midpoint/repo/sql/helpers/ExplicitAccessLock.java
M repo/repo-sql-impl/src/main/java/com/evolveum/midpoint/repo/sql/helpers/ObjectUpdater.java
M repo/repo-sqlbase/src/main/java/com/evolveum/midpoint/repo/sqlbase/perfmon/SqlPerformanceMonitorImpl.java
M repo/task-quartz-impl/src/main/java/com/evolveum/midpoint/task/quartzimpl/work/WorkStateManager.java
M repo/task-quartz-impl/src/main/java/com/evolveum/midpoint/task/quartzimpl/work/segmentation/WorkSegmentationStrategy.java
M testing/story/src/test/resources/buckets/task-recompute-16-04.xml
M testing/story/src/test/resources/buckets/task-recompute-256-30.xml
Log Message:
-----------
Improve bucket management efficiency
Implemented experimental modifyObjectDynamically method
in the repository API to avoid costly get -> compute delta ->
modify-with-precondition -> retry cycles.
This also changes the semantics of 'conflicts' in bucket management
statistics. The conflicts now represent repo-level conflicts, that
were not measured there previously. (This also required adding repo
perf statistics into ModifyObjectResult.)
Unrelated changes:
- added aggregation of repo performance information in task trees.
Commit: 49da8d7c9884905e436ff2f713621b353bbec7f7
https://github.com/Evolveum/midpoint/commit/49da8d7c9884905e436ff2f713621b353bbec7f7
Author: Pavol Mederly <mederly at evolveum.com>
Date: 2021-04-14 (Wed, 14 Apr 2021)
Changed paths:
R repo/repo-sql-impl/src/main/java/com/evolveum/midpoint/repo/sql/helpers/ExplicitAccessLock.java
M repo/repo-sql-impl/src/main/java/com/evolveum/midpoint/repo/sql/helpers/ObjectUpdater.java
Log Message:
-----------
Fix modifyObjectDynamically
The object was retrieved without being locked for update. This affected
some databases like H2, MySQL, and Oracle.
Along with this fix the ExplicitAccessLock class could be deleted.
Commit: 303168ef415c1e295e30d81f91ae5c1f2fe09836
https://github.com/Evolveum/midpoint/commit/303168ef415c1e295e30d81f91ae5c1f2fe09836
Author: Pavol Mederly <mederly at evolveum.com>
Date: 2021-04-15 (Thu, 15 Apr 2021)
Changed paths:
M pom.xml
M repo/repo-sql-impl/src/main/java/com/evolveum/midpoint/repo/sql/helpers/ObjectUpdater.java
M repo/repo-sqlbase/src/main/java/com/evolveum/midpoint/repo/sqlbase/DataSourceFactory.java
Log Message:
-----------
Fix connection leak in modifyObjectDynamically
Also:
1. DB connections in HikariCP are now stored with autocommit = false
(avoiding unnecessary resetting of this flag each time a connection
is returned to the pool).
2. Updated HikariCP to 3.4.5 (the latest 3.x version).
Commit: 20c958050dd813ee40c7a60f8470f522102fbffc
https://github.com/Evolveum/midpoint/commit/20c958050dd813ee40c7a60f8470f522102fbffc
Author: Pavol Mederly <mederly at evolveum.com>
Date: 2021-04-15 (Thu, 15 Apr 2021)
Changed paths:
M infra/prism-api/src/main/java/com/evolveum/midpoint/prism/polystring/PolyString.java
M infra/util/src/main/java/com/evolveum/midpoint/util/exception/ObjectNotFoundException.java
M repo/repo-api/src/main/java/com/evolveum/midpoint/repo/api/RepositoryService.java
M repo/repo-sqale/sql/pgnew-experiments.sql
M repo/repo-sqale/sql/pgnew-repo.sql
R repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/InOidFilterProcessor.java
R repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/ObjectRefTableItemFilterProcessor.java
R repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/RefItemFilterProcessor.java
M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/SqaleQueryContext.java
M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/SqaleRepoContext.java
M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/SqaleRepositoryConfiguration.java
M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/SqaleRepositoryService.java
M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/SqaleTransformerSupport.java
M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/SqaleUpdateContext.java
M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/UriCache.java
R repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/UriItemFilterProcessor.java
A repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/delta/DelegatingItemDeltaProcessor.java
M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/delta/ItemDeltaProcessor.java
R repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/delta/SimpleItemDeltaProcessor.java
R repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/delta/SinglePathItemDeltaProcessor.java
R repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/delta/SqaleItemSqlMapper.java
R repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/delta/TimestampItemDeltaProcessor.java
A repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/delta/item/EnumItemDeltaProcessor.java
A repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/delta/item/ItemDeltaValueProcessor.java
A repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/delta/item/PolyStringItemDeltaProcessor.java
A repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/delta/item/RefItemDeltaProcessor.java
A repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/delta/item/SimpleItemDeltaProcessor.java
A repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/delta/item/SinglePathItemDeltaProcessor.java
A repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/delta/item/TimestampItemDeltaProcessor.java
A repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/delta/item/UriItemDeltaProcessor.java
A repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/filtering/InOidFilterProcessor.java
A repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/filtering/ObjectRefTableItemFilterProcessor.java
A repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/filtering/RefItemFilterProcessor.java
A repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/filtering/UriItemFilterProcessor.java
A repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/mapping/NestedMappingResolver.java
A repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/mapping/SqaleItemRelationResolver.java
A repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/mapping/SqaleItemSqlMapper.java
A repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/mapping/TableRelationResolver.java
A repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/qmodel/EmbeddedContainerDeltaProcessor.java
M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/qmodel/SqaleNestedMapping.java
M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/qmodel/SqaleTableMapping.java
M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/qmodel/SqaleTransformerBase.java
M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/qmodel/accesscert/QAccessCertificationDefinitionMapping.java
M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/qmodel/assignment/QAssignmentMapping.java
M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/qmodel/cases/QCaseMapping.java
M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/qmodel/common/MContainerType.java
M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/qmodel/connector/QConnectorMapping.java
M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/qmodel/focus/QFocusMapping.java
M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/qmodel/focus/QUserMapping.java
M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/qmodel/lookuptable/LookupTableRowSqlTransformer.java
A repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/qmodel/lookuptable/LookupTableSqlTransformer.java
M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/qmodel/lookuptable/MLookupTableRow.java
M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/qmodel/lookuptable/QLookupTableMapping.java
M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/qmodel/lookuptable/QLookupTableRow.java
M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/qmodel/lookuptable/QLookupTableRowMapping.java
M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/qmodel/object/MObjectType.java
M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/qmodel/object/ObjectSqlTransformer.java
M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/qmodel/object/QObjectMapping.java
M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/qmodel/object/QOperationExecutionMapping.java
M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/qmodel/object/QTriggerMapping.java
M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/qmodel/ref/QReferenceMapping.java
M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/qmodel/report/QReportData.java
M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/qmodel/report/QReportDataMapping.java
M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/qmodel/report/QReportMapping.java
M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/qmodel/resource/QResourceMapping.java
M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/qmodel/role/QAbstractRoleMapping.java
M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/qmodel/shadow/QShadow.java
M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/qmodel/shadow/QShadowMapping.java
M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/qmodel/task/QTask.java
M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/qmodel/task/QTaskMapping.java
A repo/repo-sqale/src/test/java/com/evolveum/midpoint/repo/sqale/SqaleRepoAddDeleteObjectTest.java
R repo/repo-sqale/src/test/java/com/evolveum/midpoint/repo/sqale/SqaleRepoAddObjectTest.java
M repo/repo-sqale/src/test/java/com/evolveum/midpoint/repo/sqale/SqaleRepoBaseTest.java
M repo/repo-sqale/src/test/java/com/evolveum/midpoint/repo/sqale/SqaleRepoModifyObjectTest.java
M repo/repo-sqale/src/test/java/com/evolveum/midpoint/repo/sqale/SqaleRepoSmokeTest.java
M repo/repo-sql-impl/src/main/java/com/evolveum/midpoint/repo/sql/SqlAuditServiceImpl.java
M repo/repo-sql-impl/src/main/java/com/evolveum/midpoint/repo/sql/audit/mapping/AuditCustomColumnItemFilterProcessor.java
M repo/repo-sql-impl/src/main/java/com/evolveum/midpoint/repo/sql/audit/mapping/AuditRefItemFilterProcessor.java
M repo/repo-sql-impl/src/main/java/com/evolveum/midpoint/repo/sql/helpers/ObjectRetriever.java
M repo/repo-sql-impl/src/main/java/com/evolveum/midpoint/repo/sql/query/matcher/PolyStringMatcher.java
M repo/repo-sqlbase/src/main/java/com/evolveum/midpoint/repo/sqlbase/SqlRepoContext.java
M repo/repo-sqlbase/src/main/java/com/evolveum/midpoint/repo/sqlbase/SystemConfigurationChangeDispatcherImpl.java
M repo/repo-sqlbase/src/main/java/com/evolveum/midpoint/repo/sqlbase/filtering/ValueFilterProcessor.java
M repo/repo-sqlbase/src/main/java/com/evolveum/midpoint/repo/sqlbase/filtering/item/CanonicalItemPathItemFilterProcessor.java
M repo/repo-sqlbase/src/main/java/com/evolveum/midpoint/repo/sqlbase/filtering/item/DetailTableItemFilterProcessor.java
M repo/repo-sqlbase/src/main/java/com/evolveum/midpoint/repo/sqlbase/filtering/item/EnumItemFilterProcessor.java
M repo/repo-sqlbase/src/main/java/com/evolveum/midpoint/repo/sqlbase/filtering/item/EnumOrdinalItemFilterProcessor.java
M repo/repo-sqlbase/src/main/java/com/evolveum/midpoint/repo/sqlbase/filtering/item/PolyStringItemFilterProcessor.java
M repo/repo-sqlbase/src/main/java/com/evolveum/midpoint/repo/sqlbase/filtering/item/SimpleItemFilterProcessor.java
M repo/repo-sqlbase/src/main/java/com/evolveum/midpoint/repo/sqlbase/filtering/item/SinglePathItemFilterProcessor.java
A repo/repo-sqlbase/src/main/java/com/evolveum/midpoint/repo/sqlbase/mapping/ItemRelationResolver.java
A repo/repo-sqlbase/src/main/java/com/evolveum/midpoint/repo/sqlbase/mapping/ItemSqlMapper.java
M repo/repo-sqlbase/src/main/java/com/evolveum/midpoint/repo/sqlbase/mapping/QueryModelMapping.java
M repo/repo-sqlbase/src/main/java/com/evolveum/midpoint/repo/sqlbase/mapping/QueryTableMapping.java
M repo/repo-sqlbase/src/main/java/com/evolveum/midpoint/repo/sqlbase/mapping/SqlTransformer.java
R repo/repo-sqlbase/src/main/java/com/evolveum/midpoint/repo/sqlbase/mapping/item/ItemRelationResolver.java
R repo/repo-sqlbase/src/main/java/com/evolveum/midpoint/repo/sqlbase/mapping/item/ItemSqlMapper.java
R repo/repo-sqlbase/src/main/java/com/evolveum/midpoint/repo/sqlbase/mapping/item/NestedMappingResolver.java
R repo/repo-sqlbase/src/main/java/com/evolveum/midpoint/repo/sqlbase/mapping/item/TableRelationResolver.java
Log Message:
-----------
Merge remote-tracking branch 'origin/master'
# Conflicts:
# repo/repo-api/src/main/java/com/evolveum/midpoint/repo/api/RepositoryService.java
Compare: https://github.com/Evolveum/midpoint/compare/40e014ee5ca8...20c958050dd8
More information about the midPoint-svn
mailing list