[midPoint-git] [Evolveum/midpoint] 15fc48: Added support to aggregate queries to return enums...

Andrej noreply at github.com
Mon Oct 13 14:17:45 CEST 2025 

  Branch: refs/heads/feature/ninja-stats-export-4.9
  Home:   https://github.com/Evolveum/midpoint
  Commit: 15fc480de4d6b5540c460e934f710ae58325e7c4
      https://github.com/Evolveum/midpoint/commit/15fc480de4d6b5540c460e934f710ae58325e7c4
  Author: Tony Tkáčik <tonydamage at gmail.com>
  Date:   2025-10-13 (Mon, 13 Oct 2025)

  Changed paths:
    M repo/repo-sqale/src/main/java/com/evolveum/midpoint/repo/sqale/AggregateSearchContext.java

  Log Message:
  -----------
  Added support to aggregate queries to return enums and QNames

(cherry picked from commit 20f03d55f61925ef21819a3135502ed1d7afe1ea)


  Commit: c26f8d28b3969104b4aa0979784d87a7c5ee0ed0
      https://github.com/Evolveum/midpoint/commit/c26f8d28b3969104b4aa0979784d87a7c5ee0ed0
  Author: Tony Tkáčik <tonydamage at gmail.com>
  Date:   2025-10-13 (Mon, 13 Oct 2025)

  Changed paths:
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/AbstractRepositorySearchAction.java
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/CountRepositoryAction.java
    A tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/ExportConfigurationAction.java
    A tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/ExportFocusStatisticsAction.java
    A tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/ExportResourceSchemaAction.java
    A tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/ExportShadowStatisticsAction.java
    A tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/ExportShadowStatisticsOptions.java
    A tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/worker/ExportConfigurationWorker.java
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/worker/ExportConsumerWorker.java
    A tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/worker/ExportFocusStatisticsWorker.java
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/impl/Command.java
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/util/NinjaUtils.java

  Log Message:
  -----------
  Added export statistics action skeletons to Ninja

(cherry picked from commit df12803d7112acb5592c3966586e237a8b92c246)


  Commit: 5d690356d84ba545adc680c812fd8c9e59977198
      https://github.com/Evolveum/midpoint/commit/5d690356d84ba545adc680c812fd8c9e59977198
  Author: Andrej Zan <andrej.zan at evolveum.com>
  Date:   2025-10-13 (Mon, 13 Oct 2025)

  Changed paths:
    R tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/ExportFocusStatisticsAction.java
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/ExportShadowStatisticsAction.java
    A tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/CategoriesDistribution.java
    A tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/ExportFocusStatisticsAction.java
    A tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/ExportFocusStatisticsWorker.java
    A tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/FocusStats.java
    A tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/FocusTypeCounter.java
    A tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/FocusTypeStats.java
    A tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/PrismTransformableRecord.java
    A tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/PropertyStats.java
    A tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/PropertyStatsCounter.java
    A tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/StatsCounter.java
    R tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/worker/ExportFocusStatisticsWorker.java
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/impl/Command.java

  Log Message:
  -----------
  Implement export of focus statistics.

**What**

Implement calculation and export of various statistics related to focus
objects.

**Why**

This statistics can be very useful for future development of midPoint.

**Notes**

Current implementation does not consider complex structures inside of
focus objects. For example the `activation` is not currently counted to
any statistics, because it is a complex structure.

This version is not yet fully optimized in terms of memory usage.

(cherry picked from commit 6e10a2a7b7a894e5f669420063c957b813fad604)


  Commit: c4352dd9d9ac744017e7fc115393c8084fc99706
      https://github.com/Evolveum/midpoint/commit/c4352dd9d9ac744017e7fc115393c8084fc99706
  Author: Michal Zelencik <michal.zelencik at evolveum.com>
  Date:   2025-10-13 (Mon, 13 Oct 2025)

  Changed paths:
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/ExportOptions.java
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/ExportRepositoryAction.java
    A tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/worker/ExportPerObjectWorker.java

  Log Message:
  -----------
  Add per-object export mode with type subdirectories and filenames

- Add ExportPerObjectWorker to write each object into a separate file
- Add --split-files (-sf) flag to ExportOptions

Route export to the appropriate worker in ExportRepositoryAction
- ExportPerObjectWorker when --split-files is enabled
- ExportConsumerWorker otherwise (existing single-file behavior)

Signed-off-by: Michal Zelencik <michal.zelencik at evolveum.com>

(cherry picked from commit b18aabe9f5a064632e82611e8877ec87914cc8d2)


  Commit: 6be5462b83a625c9900241898450333a68bdf5bf
      https://github.com/Evolveum/midpoint/commit/6be5462b83a625c9900241898450333a68bdf5bf
  Author: Michal Zelencik <michal.zelencik at evolveum.com>
  Date:   2025-10-13 (Mon, 13 Oct 2025)

  Changed paths:
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/ExportConfigurationAction.java
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/ExportRepositoryAction.java
    M tools/ninja/src/main/resources/messages.properties

  Log Message:
  -----------
  Fix: Move Split into files possibility from export repository to export
configuration
Fix: Add messages properties to be able to use --split-files flag

(cherry picked from commit 96e255541d40742de586b33f9c1d7b54a0b4993e)


  Commit: d1ccdc3f212a4b5107f6f09683165f8e61f926c0
      https://github.com/Evolveum/midpoint/commit/d1ccdc3f212a4b5107f6f09683165f8e61f926c0
  Author: Michal Zelencik <michal.zelencik at evolveum.com>
  Date:   2025-10-13 (Mon, 13 Oct 2025)

  Changed paths:
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/ExportConfigurationAction.java
    A tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/worker/ExportConfigurationSeparateWorker.java
    R tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/worker/ExportPerObjectWorker.java

  Log Message:
  -----------
  Remake configuration exporter to inherit from
ExportConfigurationSeparateWorker to include getExcludeItemsPaths()
and editObject() methods.

Signed-off-by: Michal Zelencik <michal.zelencik at evolveum.com>

(cherry picked from commit ef5e44fc7766d50b7e219de8d67ddeab4e948625)


  Commit: de8e21e10a7aa0d7e31c865e6c973b22cbe9a967
      https://github.com/Evolveum/midpoint/commit/de8e21e10a7aa0d7e31c865e6c973b22cbe9a967
  Author: Andrej Zan <andrej.zan at evolveum.com>
  Date:   2025-10-13 (Mon, 13 Oct 2025)

  Changed paths:
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/CategoriesDistribution.java
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/FocusTypeCounter.java
    A tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/MagnitudeCounter.java
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/PropertyStats.java
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/PropertyStatsCounter.java

  Log Message:
  -----------
  Focus statistics improvements and changes

**What**

- Collect statistics also for properties in custom schema extensions.
- Change all counts to ratios (except objects count).
- Add new stats like ratio of multi valued properties.
- Use order of magnitudes for object counts instead of exact counts.
- Remove "floating" threshold for distribution threshold.
- Do not calculate cardinality if distribution is shown.
- Do not consider null values (basically missing property) as a
  "category" (each set of identical values is one category). That means:
  - Missing properties are not counted in cardinality.
  - Missing properties are not part of the distribution array.

**Why**

Most of the changes were done from compliance reasons. We do not want to
expose potentially sensitive statistics.

(cherry picked from commit 691c9ccf24e4486e3a83efc22bf6eaebf9a65c46)


  Commit: e5be041fc455862381fcc9bcac6b61ea740e6175
      https://github.com/Evolveum/midpoint/commit/e5be041fc455862381fcc9bcac6b61ea740e6175
  Author: Andrej Zan <andrej.zan at evolveum.com>
  Date:   2025-10-13 (Mon, 13 Oct 2025)

  Changed paths:
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/ExportFocusStatisticsAction.java
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/FocusTypeCounter.java
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/StatsCounter.java

  Log Message:
  -----------
  Support items exclusion in focus statistics export

**What**

Support the `-ei` option to exclude particular items from the
statistics.

**Why**

To allow user decide, what items are too much sensitive to count any
statistics on them.

(cherry picked from commit 13f773331164c6172947418c71682748a6aa6390)


  Commit: 8865ba7d6ca11899dd8996e2257cfa95ea7a1785
      https://github.com/Evolveum/midpoint/commit/8865ba7d6ca11899dd8996e2257cfa95ea7a1785
  Author: Andrej Zan <andrej.zan at evolveum.com>
  Date:   2025-10-13 (Mon, 13 Oct 2025)

  Changed paths:
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/ExportFocusStatisticsAction.java
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/FocusTypeCounter.java
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/StatsCounter.java

  Log Message:
  -----------
  Hard-code items allow list to Ninja's focus stats export

**What**

Add hard-coded list of items, which are by default included in focus
stats export.

**Why**

Not all items are interesting for our needs, and some items may be also
sensitive. Because of that we decided to include just some particular
items. Only these hard-coded items will be included in the export.

Items can still be explicitly excluded by user with the use of `-ei`
option.

Explicitly excluding items that are not by default included has no
effect, since they would not be exported anyway.

(cherry picked from commit eba3e89200b522559b7a88bfa5c7cb1407db267a)


  Commit: a4a0b95dcc1301f843188c3329f158a4afaadbc2
      https://github.com/Evolveum/midpoint/commit/a4a0b95dcc1301f843188c3329f158a4afaadbc2
  Author: Andrej Zan <andrej.zan at evolveum.com>
  Date:   2025-10-13 (Mon, 13 Oct 2025)

  Changed paths:
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/PropertyStatsCounter.java

  Log Message:
  -----------
  Add limit on size of map with properties values

**What**

Add a limit on a size of map which holds "categories" of properties
values across all objects of the same type.

**Why**

To lower potentially high memory usage.

(cherry picked from commit 820e7ed748b6a4c46e92c16c5caf884eb136ac06)


  Commit: 6468a58a6a177f2d258ed7c83502079614f23211
      https://github.com/Evolveum/midpoint/commit/6468a58a6a177f2d258ed7c83502079614f23211
  Author: Andrej Zan <andrej.zan at evolveum.com>
  Date:   2025-10-13 (Mon, 13 Oct 2025)

  Changed paths:
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/ExportShadowStatisticsAction.java
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/ExportFocusStatisticsAction.java
    M tools/ninja/src/main/java/com/evolveum/midpoint/ninja/action/stats/MagnitudeCounter.java

  Log Message:
  -----------
  Change shadow statistics to approximations

**What**

Change the exact counts in shadow statistics to approximations in the
order of magnitude precision.

**Why**

Because of compliance reasons, we don't want to expose exact counts, as
they could be seen as sensitive data.

(cherry picked from commit 63c84cee815b5d9c35ad283b89bd2a655c7ed1db)


Compare: https://github.com/Evolveum/midpoint/compare/15fc480de4d6%5E...6468a58a6a17

To unsubscribe from these emails, change your notification settings at https://github.com/Evolveum/midpoint/settings/notifications

More information about the midPoint-svn mailing list