[midPoint-git] [Evolveum/midpoint] 569fec: Fix serializability of some auxiliary classes

Viliam Repan noreply at github.com
Mon Apr 11 13:28:28 CEST 2022 

  Branch: refs/heads/feature/adminlte-upgrade-merged
  Home:   https://github.com/Evolveum/midpoint
  Commit: 569fec12c7ddc1bab9aa515fcd5176e68d3399aa
      https://github.com/Evolveum/midpoint/commit/569fec12c7ddc1bab9aa515fcd5176e68d3399aa
  Author: Pavol Mederly <mederly at evolveum.com>
  Date:   2022-04-04 (Mon, 04 Apr 2022)

  Changed paths:
    M infra/schema/src/main/java/com/evolveum/midpoint/schema/internals/ThreadLocalOperationsMonitor.java
    M infra/schema/src/main/java/com/evolveum/midpoint/schema/result/OperationMonitoringConfiguration.java

  Log Message:
  -----------
  Fix serializability of some auxiliary classes

These are sometimes part of OperationResult, which is Serializable.


  Commit: bbadfcf7dae267722e1b248d59241bf6e34693df
      https://github.com/Evolveum/midpoint/commit/bbadfcf7dae267722e1b248d59241bf6e34693df
  Author: Pavol Mederly <mederly at evolveum.com>
  Date:   2022-04-06 (Wed, 06 Apr 2022)

  Changed paths:
    M repo/repo-common/src/main/java/com/evolveum/midpoint/repo/common/activity/run/IterativeActivityRun.java
    M repo/task-quartz-impl/src/main/java/com/evolveum/midpoint/task/quartzimpl/run/TaskCycleExecutor.java
    M repo/task-quartz-impl/src/main/java/com/evolveum/midpoint/task/quartzimpl/tasks/CloseHelper.java

  Log Message:
  -----------
  Prune operation result after each bucket

This is to avoid uncontrollable growth of task operation result,
resulting in OOMs, extra large repo objects, and/or long task execution.

Should resolve MID-7830 (at least partially).


  Commit: f6d3260fa323a79e9948a604352bf1f831e524a6
      https://github.com/Evolveum/midpoint/commit/f6d3260fa323a79e9948a604352bf1f831e524a6
  Author: Pavol Mederly <mederly at evolveum.com>
  Date:   2022-04-06 (Wed, 06 Apr 2022)

  Changed paths:
    M .travis.yml
    M gui/admin-gui/src/main/java/com/evolveum/midpoint/gui/impl/prism/panel/ItemRefinedPanel.java
    M gui/admin-gui/src/main/java/com/evolveum/midpoint/gui/impl/prism/panel/ObjectTemplateItemPanel.java
    M infra/schema/src/main/resources/xml/ns/public/common/common-core-3.xsd
    M infra/schema/src/main/resources/xml/ns/public/common/common-notifications-3.xsd
    M infra/schema/src/main/resources/xml/ns/public/common/common-tasks-3.xsd
    M model/model-impl/src/test/resources/controller/account-with-pwd.xml
    M model/model-impl/src/test/resources/controller/account-without-pwd.xml
    M model/model-impl/src/test/resources/expr/user-new.xml
    M model/model-intest/src/test/java/com/evolveum/midpoint/model/intest/AbstractConfiguredModelIntegrationTest.java
    M model/model-intest/src/test/java/com/evolveum/midpoint/model/intest/sync/AbstractSynchronizationStoryTest.java
    M model/model-intest/src/test/java/com/evolveum/midpoint/model/intest/sync/TestSyncStoryUsingLiveSync.java
    M model/model-intest/src/test/java/com/evolveum/midpoint/model/intest/sync/TestSyncStoryUsingReconciliation.java
    M model/model-intest/src/test/java/com/evolveum/midpoint/model/intest/sync/TestSyncStoryUsingReconciliationMultiNode.java
    M model/model-intest/src/test/java/com/evolveum/midpoint/model/intest/sync/TestSyncStoryUsingReconciliationMultithreaded.java
    R model/model-intest/src/test/java/com/evolveum/midpoint/model/intest/sync/TestSyncStoryUsingReconciliationPartitioned.java
    A model/model-intest/src/test/java/com/evolveum/midpoint/model/intest/sync/TestSyncStoryUsingReconciliationPlain.java
    A model/model-intest/src/test/java/com/evolveum/midpoint/model/intest/sync/TestSyncStoryUsingReconciliationWithSubtasks.java
    M model/model-intest/src/test/resources/common/resource-dummy-blue.xml
    M model/model-intest/src/test/resources/common/resource-dummy-green.xml
    M model/model-intest/src/test/resources/common/resource-dummy.xml
    R model/model-intest/src/test/resources/common/task-delete-not-updated-shadows.xml
    R model/model-intest/src/test/resources/common/task-dumy-blue-livesync.xml
    R model/model-intest/src/test/resources/common/task-dumy-green-livesync.xml
    R model/model-intest/src/test/resources/common/task-dumy-livesync.xml
    R model/model-intest/src/test/resources/common/task-reconcile-dummy-blue-multinode.xml
    R model/model-intest/src/test/resources/common/task-reconcile-dummy-blue-multithreaded.xml
    R model/model-intest/src/test/resources/common/task-reconcile-dummy-blue-partitioned.xml
    R model/model-intest/src/test/resources/common/task-reconcile-dummy-blue.xml
    R model/model-intest/src/test/resources/common/task-reconcile-dummy-green-multinode.xml
    R model/model-intest/src/test/resources/common/task-reconcile-dummy-green-multithreaded.xml
    R model/model-intest/src/test/resources/common/task-reconcile-dummy-green-partitioned.xml
    R model/model-intest/src/test/resources/common/task-reconcile-dummy-green.xml
    R model/model-intest/src/test/resources/common/task-reconcile-dummy-multinode.xml
    R model/model-intest/src/test/resources/common/task-reconcile-dummy-multithreaded.xml
    R model/model-intest/src/test/resources/common/task-reconcile-dummy-partitioned.xml
    R model/model-intest/src/test/resources/common/user-template-sync.xml
    A model/model-intest/src/test/resources/sync-story/task-live-sync-dummy-blue.xml
    A model/model-intest/src/test/resources/sync-story/task-live-sync-dummy-green.xml
    A model/model-intest/src/test/resources/sync-story/task-live-sync-dummy.xml
    A model/model-intest/src/test/resources/sync-story/task-reconcile-dummy-blue-multinode.xml
    A model/model-intest/src/test/resources/sync-story/task-reconcile-dummy-blue-multithreaded.xml
    A model/model-intest/src/test/resources/sync-story/task-reconcile-dummy-blue-with-subtasks.xml
    A model/model-intest/src/test/resources/sync-story/task-reconcile-dummy-blue.xml
    A model/model-intest/src/test/resources/sync-story/task-reconcile-dummy-green-multinode.xml
    A model/model-intest/src/test/resources/sync-story/task-reconcile-dummy-green-multithreaded.xml
    A model/model-intest/src/test/resources/sync-story/task-reconcile-dummy-green-with-subtasks.xml
    A model/model-intest/src/test/resources/sync-story/task-reconcile-dummy-green.xml
    A model/model-intest/src/test/resources/sync-story/task-reconcile-dummy-multinode.xml
    A model/model-intest/src/test/resources/sync-story/task-reconcile-dummy-multithreaded.xml
    A model/model-intest/src/test/resources/sync-story/task-reconcile-dummy-with-subtasks.xml
    A model/model-intest/src/test/resources/sync-story/task-reconcile-dummy.xml
    A model/model-intest/src/test/resources/sync-story/task-shadow-cleanup-green.xml
    A model/model-intest/src/test/resources/sync-story/user-template-sync.xml
    M model/model-intest/testng-db-specific.xml
    M model/model-intest/testng-integration-full.xml
    M model/notifications-impl/src/main/java/com/evolveum/midpoint/notifications/impl/notifiers/AccountActivationNotifier.java
    M model/notifications-impl/src/main/java/com/evolveum/midpoint/notifications/impl/notifiers/PasswordResetNotifier.java
    M provisioning/ucf-impl-connid/src/main/java/com/evolveum/midpoint/provisioning/ucf/impl/connid/ConnectorFactoryConnIdImpl.java
    M repo/repo-common/src/main/java/com/evolveum/midpoint/repo/common/activity/run/IterativeActivityRun.java
    M repo/repo-sqale/src/test/java/com/evolveum/midpoint/repo/sqale/func/SqaleRepoModifyObjectTest.java

  Log Message:
  -----------
  Merge branch 'master' into tmp/fixes-after-4.5

# Conflicts:
#	repo/repo-common/src/main/java/com/evolveum/midpoint/repo/common/activity/run/IterativeActivityRun.java


  Commit: 06df862e9f03f42ab94bd7128bac854b96ebb55b
      https://github.com/Evolveum/midpoint/commit/06df862e9f03f42ab94bd7128bac854b96ebb55b
  Author: Pavol Mederly <mederly at evolveum.com>
  Date:   2022-04-06 (Wed, 06 Apr 2022)

  Changed paths:
    M model/report-impl/src/main/java/com/evolveum/midpoint/report/impl/activity/ClassicCollectionReportExportActivityRun.java
    M model/report-impl/src/main/java/com/evolveum/midpoint/report/impl/activity/ClassicDashboardReportExportActivityRun.java
    M model/report-impl/src/main/java/com/evolveum/midpoint/report/impl/activity/ClassicReportImportActivityRun.java
    M repo/repo-common/src/test/java/com/evolveum/midpoint/repo/common/tasks/handlers/iterative/IterativeMockActivityRun.java

  Log Message:
  -----------
  Fix suspension of report-related activities

Report-related activities (both export and import) ignored signals
to suspend that come from the activity framework. So even if the task
was suspended, report processes continued to iterate through input
objects.

This is now fixed. The iteration should stop when the task goes down.

Related to MID-7839.


  Commit: 9d859850565604f3e81deac95d9a4592d12cbfe6
      https://github.com/Evolveum/midpoint/commit/9d859850565604f3e81deac95d9a4592d12cbfe6
  Author: Pavol Mederly <mederly at evolveum.com>
  Date:   2022-04-06 (Wed, 06 Apr 2022)

  Changed paths:
    M model/report-impl/src/main/java/com/evolveum/midpoint/report/impl/activity/ClassicCollectionReportExportActivityRun.java
    M model/report-impl/src/main/java/com/evolveum/midpoint/report/impl/activity/ClassicDashboardReportExportActivityRun.java
    M model/report-impl/src/main/java/com/evolveum/midpoint/report/impl/activity/ReportDataAggregationActivityRun.java
    M model/report-impl/src/main/java/com/evolveum/midpoint/report/impl/activity/ReportDataCreationActivityRun.java

  Log Message:
  -----------
  Stop writing operationExecution data for reports

It harms performance (a lot).

Related to MID-7857.


  Commit: 977fcf3be104bb390df705fe359eb82d9b072fda
      https://github.com/Evolveum/midpoint/commit/977fcf3be104bb390df705fe359eb82d9b072fda
  Author: Pavol Mederly <mederly at evolveum.com>
  Date:   2022-04-06 (Wed, 06 Apr 2022)

  Changed paths:
    M model/report-impl/src/main/java/com/evolveum/midpoint/report/impl/activity/SaveReportFileSupport.java
    M model/report-impl/src/main/java/com/evolveum/midpoint/report/impl/controller/CollectionDistributedExportController.java

  Log Message:
  -----------
  Do not write incomplete report output on suspend

When the task goes down, we must not write incomplete (partial) data
to the report output file.

Work in progress: This solves the problem for classical reports.
What remains are the distributed reports. (They may be safe, but it
needs to be verified.)

Related to MID-7839.


  Commit: 2bc083aca2f39140268c8c837160f23c6a4cf0f6
      https://github.com/Evolveum/midpoint/commit/2bc083aca2f39140268c8c837160f23c6a4cf0f6
  Author: Pavol Mederly <mederly at evolveum.com>
  Date:   2022-04-07 (Thu, 07 Apr 2022)

  Changed paths:
    M model/report-impl/src/main/java/com/evolveum/midpoint/report/impl/activity/ReportDataAggregationActivityRun.java
    M repo/repo-common/src/main/java/com/evolveum/midpoint/repo/common/activity/run/AbstractCompositeActivityRun.java
    M repo/repo-common/src/main/java/com/evolveum/midpoint/repo/common/activity/run/IterativeActivityRun.java
    M repo/repo-common/src/main/java/com/evolveum/midpoint/repo/common/activity/run/LocalActivityRun.java

  Log Message:
  -----------
  Stop distribution of non-distributable activities

This commit introduced safety checks for activities that are known
to fail in multi-workers or multithreaded mode:

- report data aggregation (any parallelism is forbidden),
- any composite activity (multi-workers are forbidden, see the source).

This is an immediate fix for MID-7861.

However, it's not a complete one: we should check for multi-node
distributability _before_ the worker tasks are created, not in the tasks
themselves. Otherwise, the ultimate harm is avoided, but the workers
are already there - and suspended with a fatal error.


  Commit: 704291c73a5f1372fc9577126e7dc17325e70264
      https://github.com/Evolveum/midpoint/commit/704291c73a5f1372fc9577126e7dc17325e70264
  Author: Pavol Mederly <mederly at evolveum.com>
  Date:   2022-04-11 (Mon, 11 Apr 2022)

  Changed paths:
    M infra/schema/src/main/java/com/evolveum/midpoint/schema/internals/ThreadLocalOperationsMonitor.java
    M infra/schema/src/main/java/com/evolveum/midpoint/schema/result/OperationMonitoringConfiguration.java
    M model/report-impl/src/main/java/com/evolveum/midpoint/report/impl/activity/ClassicCollectionReportExportActivityRun.java
    M model/report-impl/src/main/java/com/evolveum/midpoint/report/impl/activity/ClassicDashboardReportExportActivityRun.java
    M model/report-impl/src/main/java/com/evolveum/midpoint/report/impl/activity/ClassicReportImportActivityRun.java
    M model/report-impl/src/main/java/com/evolveum/midpoint/report/impl/activity/ReportDataAggregationActivityRun.java
    M model/report-impl/src/main/java/com/evolveum/midpoint/report/impl/activity/ReportDataCreationActivityRun.java
    M model/report-impl/src/main/java/com/evolveum/midpoint/report/impl/activity/SaveReportFileSupport.java
    M model/report-impl/src/main/java/com/evolveum/midpoint/report/impl/controller/CollectionDistributedExportController.java
    M repo/repo-common/src/main/java/com/evolveum/midpoint/repo/common/activity/run/AbstractCompositeActivityRun.java
    M repo/repo-common/src/main/java/com/evolveum/midpoint/repo/common/activity/run/IterativeActivityRun.java
    M repo/repo-common/src/main/java/com/evolveum/midpoint/repo/common/activity/run/LocalActivityRun.java
    M repo/repo-common/src/test/java/com/evolveum/midpoint/repo/common/tasks/handlers/iterative/IterativeMockActivityRun.java

  Log Message:
  -----------
  Merge branch 'tmp/fixes-after-4.5'


  Commit: d3d3f3ee38bb20513d5d0b504f2c9fff70698a22
      https://github.com/Evolveum/midpoint/commit/d3d3f3ee38bb20513d5d0b504f2c9fff70698a22
  Author: Pavol Mederly <mederly at evolveum.com>
  Date:   2022-04-11 (Mon, 11 Apr 2022)

  Changed paths:
    M model/report-impl/src/main/java/com/evolveum/midpoint/report/impl/activity/ClassicReportImportActivityRun.java

  Log Message:
  -----------
  Simplify ClassicReportImportActivityRun

Due to some past refactorings, iterateOverItemsInBucket method in
ClassicReportImportActivityRun got unnecessarily complex. This got
clearly visible in 06df862e9f03f42ab94bd7128bac854b96ebb55b.

So, this commit simplifies that method.


  Commit: 431af54aad0076bae6af28a25a1583365ec48d85
      https://github.com/Evolveum/midpoint/commit/431af54aad0076bae6af28a25a1583365ec48d85
  Author: Pavol Mederly <mederly at evolveum.com>
  Date:   2022-04-11 (Mon, 11 Apr 2022)

  Changed paths:
    M model/report-impl/src/main/java/com/evolveum/midpoint/report/impl/activity/ReportDataAggregationActivityRun.java
    M model/report-impl/src/main/java/com/evolveum/midpoint/report/impl/activity/ReportDataCreationActivityRun.java
    M model/report-impl/src/main/java/com/evolveum/midpoint/report/impl/controller/CollectionDistributedExportController.java

  Log Message:
  -----------
  Add task suspension check for distributed reports

Now we skip writing per-bucket report data if the data-creation task
is going down. This is to ensure that the resulting data are clean.

In data-aggregation activity we now also check the sequence of partial
report data objects, to ensure nothing is missing and nothing is
included twice.

Notes:

1. There is (still) no check for the total expected number of partial
data objects in data aggregation activity. We rely on the activity
framework to ensure the aggregator is started only after the preceding
activity (data creation) is 100% complete.

2. In general, one should not resume report-creation activity after
it has been suspended (at least not after too much time). There is non-
negligible chance that the resulting report will be inconsistent because
of the time gap.

Related to MID-7839 (suspension of report tasks).


  Commit: 75a96658a759e332a60e48aa28e95cb82e77d765
      https://github.com/Evolveum/midpoint/commit/75a96658a759e332a60e48aa28e95cb82e77d765
  Author: Viliam Repan <vilo.repan at evolveum.com>
  Date:   2022-04-11 (Mon, 11 Apr 2022)

  Changed paths:
    M infra/schema/src/main/java/com/evolveum/midpoint/schema/internals/ThreadLocalOperationsMonitor.java
    M infra/schema/src/main/java/com/evolveum/midpoint/schema/result/OperationMonitoringConfiguration.java
    M model/report-impl/src/main/java/com/evolveum/midpoint/report/impl/activity/ClassicCollectionReportExportActivityRun.java
    M model/report-impl/src/main/java/com/evolveum/midpoint/report/impl/activity/ClassicDashboardReportExportActivityRun.java
    M model/report-impl/src/main/java/com/evolveum/midpoint/report/impl/activity/ClassicReportImportActivityRun.java
    M model/report-impl/src/main/java/com/evolveum/midpoint/report/impl/activity/ReportDataAggregationActivityRun.java
    M model/report-impl/src/main/java/com/evolveum/midpoint/report/impl/activity/ReportDataCreationActivityRun.java
    M model/report-impl/src/main/java/com/evolveum/midpoint/report/impl/activity/SaveReportFileSupport.java
    M model/report-impl/src/main/java/com/evolveum/midpoint/report/impl/controller/CollectionDistributedExportController.java
    M repo/repo-common/src/main/java/com/evolveum/midpoint/repo/common/activity/run/AbstractCompositeActivityRun.java
    M repo/repo-common/src/main/java/com/evolveum/midpoint/repo/common/activity/run/IterativeActivityRun.java
    M repo/repo-common/src/main/java/com/evolveum/midpoint/repo/common/activity/run/LocalActivityRun.java
    M repo/repo-common/src/test/java/com/evolveum/midpoint/repo/common/tasks/handlers/iterative/IterativeMockActivityRun.java

  Log Message:
  -----------
  Merge remote-tracking branch 'origin/master' into feature/adminlte-upgrade-merged


Compare: https://github.com/Evolveum/midpoint/compare/b7ad64b865ce...75a96658a759

More information about the midPoint-svn mailing list