[midPoint-git] [Evolveum/midpoint] 163175: Added export statistics action skeletons to Ninja

Andrej noreply at github.com
Mon Oct 13 19:34:45 CEST 2025 

  Branch: refs/heads/feature/ninja-stats-export-4.8
  Home:   https://github.com/Evolveum/midpoint
  Commit: 16317561b928194899106b51a605cc8b60847268
      https://github.com/Evolveum/midpoint/commit/16317561b928194899106b51a605cc8b60847268
  Author: Tony Tkáčik <tonydamage at gmail.com>
  Date:   2025-10-13 (Mon, 13 Oct 2025)

  Changed paths:
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/AbstractRepositorySearchAction.java
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/CountRepositoryAction.java
    A tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/ExportConfigurationAction.java
    A tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/ExportFocusStatisticsAction.java
    A tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/ExportResourceSchemaAction.java
    A tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/ExportShadowStatisticsAction.java
    A tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/ExportShadowStatisticsOptions.java
    A tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/worker/ExportConfigurationWorker.java
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/worker/ExportConsumerWorker.java
    A tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/worker/ExportFocusStatisticsWorker.java
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/impl/Command.java
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/util/NinjaUtils.java

  Log Message:
  -----------
  Added export statistics action skeletons to Ninja

(cherry picked from commit df12803d7112acb5592c3966586e237a8b92c246)


  Commit: 9f735b1e369c9406e5f59b8e0e632425e850d696
      https://github.com/Evolveum/midpoint/commit/9f735b1e369c9406e5f59b8e0e632425e850d696
  Author: Andrej Zan <andrej.zan at evolveum.com>
  Date:   2025-10-13 (Mon, 13 Oct 2025)

  Changed paths:
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/ExportShadowStatisticsAction.java
    A tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/CategoriesDistribution.java
    A tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/ExportFocusStatisticsAction.java
    A tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/ExportFocusStatisticsWorker.java
    A tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/FocusStats.java
    A tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/FocusTypeCounter.java
    A tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/FocusTypeStats.java
    A tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/PrismTransformableRecord.java
    A tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/PropertyStats.java
    A tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/PropertyStatsCounter.java
    A tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/StatsCounter.java
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/impl/Command.java

  Log Message:
  -----------
  Implement export of focus statistics.

**What**

Implement calculation and export of various statistics related to focus
objects.

**Why**

This statistics can be very useful for future development of midPoint.

**Notes**

Current implementation does not consider complex structures inside of
focus objects. For example the `activation` is not currently counted to
any statistics, because it is a complex structure.

This version is not yet fully optimized in terms of memory usage.

(cherry picked from commit 6e10a2a7b7a894e5f669420063c957b813fad604)


  Commit: 4c2844f16b8fce1450c3510a334c683a06e7519f
      https://github.com/Evolveum/midpoint/commit/4c2844f16b8fce1450c3510a334c683a06e7519f
  Author: Michal Zelencik <michal.zelencik at evolveum.com>
  Date:   2025-10-13 (Mon, 13 Oct 2025)

  Changed paths:
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/ExportOptions.java
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/ExportRepositoryAction.java
    A tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/worker/ExportPerObjectWorker.java

  Log Message:
  -----------
  Add per-object export mode with type subdirectories and filenames

- Add ExportPerObjectWorker to write each object into a separate file
- Add --split-files (-sf) flag to ExportOptions

Route export to the appropriate worker in ExportRepositoryAction
- ExportPerObjectWorker when --split-files is enabled
- ExportConsumerWorker otherwise (existing single-file behavior)

Signed-off-by: Michal Zelencik <michal.zelencik at evolveum.com>

(cherry picked from commit b18aabe9f5a064632e82611e8877ec87914cc8d2)


  Commit: d03e439e00a31ffcad1799aa46967b50ab7b594d
      https://github.com/Evolveum/midpoint/commit/d03e439e00a31ffcad1799aa46967b50ab7b594d
  Author: Michal Zelencik <michal.zelencik at evolveum.com>
  Date:   2025-10-13 (Mon, 13 Oct 2025)

  Changed paths:
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/ExportConfigurationAction.java
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/ExportRepositoryAction.java
    M tools/ninja/src/main/resources/messages.properties

  Log Message:
  -----------
  Fix: Move Split into files possibility from export repository to export
configuration
Fix: Add messages properties to be able to use --split-files flag

(cherry picked from commit 96e255541d40742de586b33f9c1d7b54a0b4993e)


  Commit: 820e233cffd95f0fbb0ef042e2bdc350ff108375
      https://github.com/Evolveum/midpoint/commit/820e233cffd95f0fbb0ef042e2bdc350ff108375
  Author: Michal Zelencik <michal.zelencik at evolveum.com>
  Date:   2025-10-13 (Mon, 13 Oct 2025)

  Changed paths:
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/ExportConfigurationAction.java
    A tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/worker/ExportConfigurationSeparateWorker.java
    R tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/worker/ExportPerObjectWorker.java

  Log Message:
  -----------
  Remake configuration exporter to inherit from
ExportConfigurationSeparateWorker to include getExcludeItemsPaths()
and editObject() methods.

Signed-off-by: Michal Zelencik <michal.zelencik at evolveum.com>

(cherry picked from commit ef5e44fc7766d50b7e219de8d67ddeab4e948625)


  Commit: 3119707f370dd5f8e499a18ed5941c4e037b031f
      https://github.com/Evolveum/midpoint/commit/3119707f370dd5f8e499a18ed5941c4e037b031f
  Author: Andrej Zan <andrej.zan at evolveum.com>
  Date:   2025-10-13 (Mon, 13 Oct 2025)

  Changed paths:
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/CategoriesDistribution.java
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/FocusTypeCounter.java
    A tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/MagnitudeCounter.java
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/PropertyStats.java
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/PropertyStatsCounter.java

  Log Message:
  -----------
  Focus statistics improvements and changes

**What**

- Collect statistics also for properties in custom schema extensions.
- Change all counts to ratios (except objects count).
- Add new stats like ratio of multi valued properties.
- Use order of magnitudes for object counts instead of exact counts.
- Remove "floating" threshold for distribution threshold.
- Do not calculate cardinality if distribution is shown.
- Do not consider null values (basically missing property) as a
  "category" (each set of identical values is one category). That means:
  - Missing properties are not counted in cardinality.
  - Missing properties are not part of the distribution array.

**Why**

Most of the changes were done from compliance reasons. We do not want to
expose potentially sensitive statistics.

(cherry picked from commit 691c9ccf24e4486e3a83efc22bf6eaebf9a65c46)


  Commit: ec08efb3e516e4e5e4f226801ab45e112eaa0363
      https://github.com/Evolveum/midpoint/commit/ec08efb3e516e4e5e4f226801ab45e112eaa0363
  Author: Andrej Zan <andrej.zan at evolveum.com>
  Date:   2025-10-13 (Mon, 13 Oct 2025)

  Changed paths:
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/ExportFocusStatisticsAction.java
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/FocusTypeCounter.java
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/StatsCounter.java

  Log Message:
  -----------
  Support items exclusion in focus statistics export

**What**

Support the `-ei` option to exclude particular items from the
statistics.

**Why**

To allow user decide, what items are too much sensitive to count any
statistics on them.

(cherry picked from commit 13f773331164c6172947418c71682748a6aa6390)


  Commit: 8c1eb2b742b912fe5d15e1abb6ae3026fdb6f873
      https://github.com/Evolveum/midpoint/commit/8c1eb2b742b912fe5d15e1abb6ae3026fdb6f873
  Author: Andrej Zan <andrej.zan at evolveum.com>
  Date:   2025-10-13 (Mon, 13 Oct 2025)

  Changed paths:
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/ExportFocusStatisticsAction.java
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/FocusTypeCounter.java
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/StatsCounter.java

  Log Message:
  -----------
  Hard-code items allow list to Ninja's focus stats export

**What**

Add hard-coded list of items, which are by default included in focus
stats export.

**Why**

Not all items are interesting for our needs, and some items may be also
sensitive. Because of that we decided to include just some particular
items. Only these hard-coded items will be included in the export.

Items can still be explicitly excluded by user with the use of `-ei`
option.

Explicitly excluding items that are not by default included has no
effect, since they would not be exported anyway.

(cherry picked from commit eba3e89200b522559b7a88bfa5c7cb1407db267a)


  Commit: 02b4aae05d8192cf423f4d45c8e6de0bee36a3f6
      https://github.com/Evolveum/midpoint/commit/02b4aae05d8192cf423f4d45c8e6de0bee36a3f6
  Author: Andrej Zan <andrej.zan at evolveum.com>
  Date:   2025-10-13 (Mon, 13 Oct 2025)

  Changed paths:
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/PropertyStatsCounter.java

  Log Message:
  -----------
  Add limit on size of map with properties values

**What**

Add a limit on a size of map which holds "categories" of properties
values across all objects of the same type.

**Why**

To lower potentially high memory usage.

(cherry picked from commit 820e7ed748b6a4c46e92c16c5caf884eb136ac06)


  Commit: fc3a163e282218ba4875cf29c1e1e0aeb13a46b2
      https://github.com/Evolveum/midpoint/commit/fc3a163e282218ba4875cf29c1e1e0aeb13a46b2
  Author: Andrej Zan <andrej.zan at evolveum.com>
  Date:   2025-10-13 (Mon, 13 Oct 2025)

  Changed paths:
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/ExportShadowStatisticsAction.java
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/ExportFocusStatisticsAction.java
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/MagnitudeCounter.java

  Log Message:
  -----------
  Change shadow statistics to approximations

**What**

Change the exact counts in shadow statistics to approximations in the
order of magnitude precision.

**Why**

Because of compliance reasons, we don't want to expose exact counts, as
they could be seen as sensitive data.

(cherry picked from commit 63c84cee815b5d9c35ad283b89bd2a655c7ed1db)


  Commit: ce953403edd5cd10ef2e12ca1f86babef9b048d9
      https://github.com/Evolveum/midpoint/commit/ce953403edd5cd10ef2e12ca1f86babef9b048d9
  Author: Andrej Zan <andrej.zan at evolveum.com>
  Date:   2025-10-13 (Mon, 13 Oct 2025)

  Changed paths:
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/ExportFocusStatisticsAction.java

  Log Message:
  -----------
  Remove "riskLevel" from the focus statistics

**What**

Remove the `riskLevel` property from the focus statistics export.

**Why**

It was included by mistake. We consider it to be sensitive, thus we
don't want to export any stats for it.

(cherry picked from commit 15ceaba090e8cc5f78c26f09d2a7415e62ce89b5)


  Commit: a05f4a1008a81090e66522c80a387a78f202dab5
      https://github.com/Evolveum/midpoint/commit/a05f4a1008a81090e66522c80a387a78f202dab5
  Author: Andrej Zan <andrej.zan at evolveum.com>
  Date:   2025-10-13 (Mon, 13 Oct 2025)

  Changed paths:
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/ExportShadowStatisticsAction.java

  Log Message:
  -----------
  Add synchronization situation to the shadow statistics

**What**

Instead of correlation situation use synchronization situation in
aggregate query.

(cherry picked from commit 70bec637bf1161f91cd886510629e13db0eb963b)


  Commit: e7b72999ec65b557e26004cea560c2bd90c977c6
      https://github.com/Evolveum/midpoint/commit/e7b72999ec65b557e26004cea560c2bd90c977c6
  Author: Andrej Zan <andrej.zan at evolveum.com>
  Date:   2025-10-13 (Mon, 13 Oct 2025)

  Changed paths:
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/ExportConfigurationAction.java
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/worker/ExportConfigurationSeparateWorker.java
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/worker/ExportConfigurationWorker.java

  Log Message:
  -----------
  Exclude of unnecessary items from configuration export

**What**

Remove items, which could be considered as sensitive or are not so
interesting.

**Why**

Mainly because of compliance.

(cherry picked from commit 9fb3cf974ed6446fba7d15df33f90785d9b337b6)


  Commit: afe24cfe6cb6107ffdacc1ce84f24175e37c5ca2
      https://github.com/Evolveum/midpoint/commit/afe24cfe6cb6107ffdacc1ce84f24175e37c5ca2
  Author: Andrej Zan <andrej.zan at evolveum.com>
  Date:   2025-10-13 (Mon, 13 Oct 2025)

  Changed paths:
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/ExportFocusStatisticsAction.java
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/FocusTypeCounter.java
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/StatsCounter.java

  Log Message:
  -----------
  Fix inclusion of extensions in focus stats export

**What**

Fix the issue with unmatched extensions properties to include in focus
stats export.

**Note**

The main problem was, that the definition does not contain whole path,
but just the last segment (item name). When the include precondition was
applied, it did not matched any extension property, because the
precondition expected that the path will start with `extension/`. The
fix was to use `Item` instead of `ItemDefinition` in the precondition.
Item is aware of it's parents, so it contains whole path, not just item
name.

(cherry picked from commit df10f572fc2e3060b3641d8f8a3956c23887b1d1)


  Commit: 57bc79df50f68510519559e485c14d6e74f11e46
      https://github.com/Evolveum/midpoint/commit/57bc79df50f68510519559e485c14d6e74f11e46
  Author: Andrej Zan <andrej.zan at evolveum.com>
  Date:   2025-10-13 (Mon, 13 Oct 2025)

  Changed paths:
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/ExportConfigurationAction.java
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/FocusTypeCounter.java
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/worker/ExportConfigurationWorker.java
    M tools/ninja/src/main/resources/messages.properties

  Log Message:
  -----------
  Fix missing code after cherry-picks

**What**

Fix errors caused by missing code used by functionality cherry picked
from new versions.


Compare: https://github.com/Evolveum/midpoint/compare/16317561b928%5E...57bc79df50f6

To unsubscribe from these emails, change your notification settings at https://github.com/Evolveum/midpoint/settings/notifications

More information about the midPoint-svn mailing list