[midPoint-git] [Evolveum/midpoint] 032c94: Add better access to repo statistics from tests

Wed May 15 17:06:07 CEST 2019

  Branch: refs/heads/feature/optimizing-repo-access
  Home:   https://github.com/Evolveum/midpoint
  Commit: 032c948d519fd501bfa32b5a41fe24beb59176b9
      https://github.com/Evolveum/midpoint/commit/032c948d519fd501bfa32b5a41fe24beb59176b9
  Author: Pavol Mederly <mederly at evolveum.com>
  Date:   2019-05-14 (Tue, 14 May 2019)

  Changed paths:
    M repo/repo-api/src/main/java/com/evolveum/midpoint/repo/api/RepositoryService.java
    M repo/repo-sql-impl/src/main/java/com/evolveum/midpoint/repo/sql/SqlPerformanceMonitor.java
    M repo/repo-sql-impl/src/main/java/com/evolveum/midpoint/repo/sql/SqlRepositoryServiceImpl.java

  Log Message:
  -----------
  Add better access to repo statistics from tests

  Commit: 93ee7ce93da553b23baa31ae053cf52614b1af24
      https://github.com/Evolveum/midpoint/commit/93ee7ce93da553b23baa31ae053cf52614b1af24
  Author: Pavol Mederly <mederly at evolveum.com>
  Date:   2019-05-14 (Tue, 14 May 2019)

  Changed paths:
    M infra/util/src/main/java/com/evolveum/midpoint/util/caching/AbstractCache.java
    A infra/util/src/main/java/com/evolveum/midpoint/util/caching/CachePerformanceCollector.java
    M model/model-common/src/main/java/com/evolveum/midpoint/model/common/expression/evaluator/AbstractSearchExpressionEvaluator.java
    M model/model-common/src/main/java/com/evolveum/midpoint/model/common/expression/evaluator/AssociationTargetSearchExpressionEvaluator.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/projector/focus/FocusConstraintsChecker.java
    M provisioning/provisioning-impl/src/main/java/com/evolveum/midpoint/provisioning/impl/ConstraintsChecker.java
    M repo/repo-cache/src/main/java/com/evolveum/midpoint/repo/cache/RepositoryCache.java

  Log Message:
  -----------
  Provide a better cache statistics info for tests

Note that CachePerformanceCollector is a temporary implementation.
Something more universal is needed.

  Commit: f0aeb05477a918669b5abc554e461a4ab682b97f
      https://github.com/Evolveum/midpoint/commit/f0aeb05477a918669b5abc554e461a4ab682b97f
  Author: Pavol Mederly <mederly at evolveum.com>
  Date:   2019-05-14 (Tue, 14 May 2019)

  Changed paths:
    M testing/story/src/test/java/com/evolveum/midpoint/testing/story/ldap/AbstractLdapTest.java
    A testing/story/src/test/java/com/evolveum/midpoint/testing/story/ldap/TestLdapAssociationPerformance.java
    A testing/story/src/test/resources/ldap/assoc-perf/resource-opendj.xml
    A testing/story/src/test/resources/ldap/assoc-perf/role-ldap.xml
    A testing/story/src/test/resources/ldap/assoc-perf/role-meta.xml
    A testing/story/src/test/resources/ldap/assoc-perf/role-test.xml
    A testing/story/src/test/resources/ldap/assoc-perf/task-recompute-1.xml
    A testing/story/src/test/resources/ldap/assoc-perf/user-template.xml
    A testing/story/src/test/resources/ldap/assoc-perf/user-test.xml
    M testing/story/src/test/resources/schema/ldap.xsd

  Log Message:
  -----------
  Add LDAP association performance test (MID-5341)

  Commit: bca156105760d0a6a82cf2a334fcb450f53cf632
      https://github.com/Evolveum/midpoint/commit/bca156105760d0a6a82cf2a334fcb450f53cf632
  Author: Pavol Mederly <mederly at evolveum.com>
  Date:   2019-05-14 (Tue, 14 May 2019)

  Changed paths:
    M model/model-common/src/main/java/com/evolveum/midpoint/model/common/expression/evaluator/AbstractSearchExpressionEvaluator.java
    A model/model-common/src/main/java/com/evolveum/midpoint/model/common/expression/evaluator/caching/DefaultSearchExpressionEvaluatorCache.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/lens/Clockwork.java

  Log Message:
  -----------
  Add cache for search expression evaluators

This is EXPERIMENTAL feature! The cache is not invalidated
properly now. (Probably OK, because it is operation-scoped anyway.)

  Commit: efb4275e8a96272dac69e88b863734b6bdba3269
      https://github.com/Evolveum/midpoint/commit/efb4275e8a96272dac69e88b863734b6bdba3269
  Author: Pavol Mederly <mederly at evolveum.com>
  Date:   2019-05-15 (Wed, 15 May 2019)

  Changed paths:
    M infra/schema/src/main/java/com/evolveum/midpoint/schema/SelectorOptions.java
    M infra/schema/src/test/java/com/evolveum/midpoint/schema/TestQueryConverter.java
    M infra/util/src/main/java/com/evolveum/midpoint/util/caching/CachePerformanceCollector.java
    M model/model-common/src/main/java/com/evolveum/midpoint/model/common/expression/evaluator/AbstractSearchExpressionEvaluator.java
    M model/model-common/src/main/java/com/evolveum/midpoint/model/common/expression/evaluator/AssociationFromLinkExpressionEvaluator.java
    M model/model-impl/src/main/java/com/evolveum/midpoint/model/impl/ModelObjectResolver.java
    M provisioning/provisioning-impl/src/main/java/com/evolveum/midpoint/provisioning/impl/ConstraintsChecker.java
    M provisioning/provisioning-impl/src/main/java/com/evolveum/midpoint/provisioning/impl/ProvisioningServiceImpl.java
    M provisioning/provisioning-impl/src/main/java/com/evolveum/midpoint/provisioning/impl/ShadowCache.java
    M provisioning/provisioning-impl/src/main/java/com/evolveum/midpoint/provisioning/impl/ShadowManager.java
    M provisioning/provisioning-impl/src/main/java/com/evolveum/midpoint/provisioning/impl/errorhandling/ObjectAlreadyExistHandler.java
    M repo/repo-cache/src/main/java/com/evolveum/midpoint/repo/cache/RepositoryCache.java
    M repo/repo-common/src/main/java/com/evolveum/midpoint/repo/common/DirectoryFileObjectResolver.java
    M repo/repo-common/src/main/java/com/evolveum/midpoint/repo/common/ObjectResolver.java
    M repo/repo-common/src/main/java/com/evolveum/midpoint/repo/common/RepoObjectResolver.java
    M repo/repo-sql-impl/src/main/java/com/evolveum/midpoint/repo/sql/SqlRepositoryServiceImpl.java
    M testing/story/src/test/java/com/evolveum/midpoint/testing/story/ldap/TestLdapAssociationPerformance.java

  Log Message:
  -----------
  Reduce usage of searchObjectsIterative

This call was frequently used in provisioning and in search-based
expression evaluators. Its disadvantage is that it is not nicely
cacheable. So we replaced it here by searchObjects calls. The back
side is that we lost the possibility to stop search after sufficient
entries are fetched; but we didn't use this option so far anyway.

Fixes in RepositoryCache's null-or-harmless option checking are
here as well.

  Commit: c63d4d1fc834aba998bc9e741ef2fbdb38c4c0a4
      https://github.com/Evolveum/midpoint/commit/c63d4d1fc834aba998bc9e741ef2fbdb38c4c0a4
  Author: Pavol Mederly <mederly at evolveum.com>
  Date:   2019-05-15 (Wed, 15 May 2019)

  Changed paths:
    M infra/schema/src/main/java/com/evolveum/midpoint/schema/util/ObjectQueryUtil.java
    A repo/repo-api/src/main/java/com/evolveum/midpoint/repo/api/DeleteObjectResult.java
    A repo/repo-api/src/main/java/com/evolveum/midpoint/repo/api/ModifyObjectResult.java
    M repo/repo-api/src/main/java/com/evolveum/midpoint/repo/api/RepositoryService.java
    A repo/repo-cache/src/main/java/com/evolveum/midpoint/repo/cache/AddObjectResult.java
    M repo/repo-cache/src/main/java/com/evolveum/midpoint/repo/cache/Cache.java
    A repo/repo-cache/src/main/java/com/evolveum/midpoint/repo/cache/ChangeDescription.java
    M repo/repo-cache/src/main/java/com/evolveum/midpoint/repo/cache/QueryKey.java
    M repo/repo-cache/src/main/java/com/evolveum/midpoint/repo/cache/RepositoryCache.java
    M repo/repo-sql-impl-test/pom.xml
    M repo/repo-sql-impl-test/src/main/java/com/evolveum/midpoint/repo/sql/testing/TestSqlRepositoryServiceImpl.java
    M repo/repo-sql-impl/src/main/java/com/evolveum/midpoint/repo/sql/SqlRepositoryServiceImpl.java
    M repo/repo-sql-impl/src/main/java/com/evolveum/midpoint/repo/sql/helpers/ObjectUpdater.java

  Log Message:
  -----------
  Implement elaborate query cache eviction

Originally we removed all queries related to given object type.
However, we should be more selective; otherwise e.g. after account
shadow is modified, all entitlement shadow queries are thrown away.

  Commit: 9a631d1292aa98fd3f449243950f234c377f2fa1
      https://github.com/Evolveum/midpoint/commit/9a631d1292aa98fd3f449243950f234c377f2fa1
  Author: Pavol Mederly <mederly at evolveum.com>
  Date:   2019-05-15 (Wed, 15 May 2019)

  Changed paths:
    M model/model-test/src/main/java/com/evolveum/midpoint/model/test/AbstractModelIntegrationTest.java
    A repo/repo-api/src/main/java/com/evolveum/midpoint/repo/api/PerformanceMonitor.java
    M repo/repo-api/src/main/java/com/evolveum/midpoint/repo/api/RepositoryService.java
    M repo/repo-cache/src/main/java/com/evolveum/midpoint/repo/cache/RepositoryCache.java
    M repo/repo-sql-impl-test/src/test/java/com/evolveum/midpoint/repo/sql/SearchIterativeTest.java
    M repo/repo-sql-impl/src/main/java/com/evolveum/midpoint/repo/sql/SqlAuditServiceImpl.java
    M repo/repo-sql-impl/src/main/java/com/evolveum/midpoint/repo/sql/SqlBaseService.java
    R repo/repo-sql-impl/src/main/java/com/evolveum/midpoint/repo/sql/SqlPerformanceMonitor.java
    A repo/repo-sql-impl/src/main/java/com/evolveum/midpoint/repo/sql/SqlPerformanceMonitorImpl.java
    M repo/repo-sql-impl/src/main/java/com/evolveum/midpoint/repo/sql/SqlRepositoryFactory.java
    M repo/repo-sql-impl/src/main/java/com/evolveum/midpoint/repo/sql/SqlRepositoryServiceImpl.java
    M repo/repo-sql-impl/src/main/java/com/evolveum/midpoint/repo/sql/data/common/dictionary/ExtItemDictionary.java
    M testing/story/src/test/java/com/evolveum/midpoint/testing/story/ldap/TestLdapAssociationPerformance.java

  Log Message:
  -----------
  Pull up performance monitor to repo-api

This is needed to access the monitor even through RepositoryCache.

Note that this is probably a temporary measure, until serious
performance monitoring is in place.

Compare: https://github.com/Evolveum/midpoint/compare/032c948d519f%5E...9a631d1292aa