<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Dear midPoint community,</p>
<p><span class="byline"><span class="author vcard"></span></span></p>
<div class="entry-content">
<p>The development of <a
href="https://docs.evolveum.com/midpoint/midprivacy/phases/01-data-provenance-prototype/">data
provenance prototype</a> is finished. The prototype will be a
part of midPoint 4.2 release. This concludes the first phase of
<a href="https://docs.evolveum.com/midpoint/midprivacy/">midPrivacy</a>
initiative. There are interesting results, both practical and
theoretical.<span id="more-6909"></span></p>
<p>Metadata, data about data. That is the core of data provenance.
However, metadata have their structure similar to the structure
of ordinary data. The first problem was how to express that
structure. None of the existing popular data modeling languages
had any support for metadata. Therefore we had to invent our own
language: <a
href="https://docs.evolveum.com/midpoint/midprivacy/phases/01-data-provenance-prototype/axiom/spec/">Axiom</a>.
Creating new language is a major task and we have considered all
the options to avoid reinventing the wheel. But in the end,
Axiom was the right way to go.</p>
<p>We have used Axiom to create metadata schemas. We have updated
all of the midPoint core to support metadata. Metadata are
stored in the repository, there are metadata mappings and value
consolidation and reconciliation algorithms are fully
metadata-aware. MidPoint user interface was extended to display
value metadata.</p>
<p>If you want to see the results of our work, there is an <a
href="https://docs.evolveum.com/media/2020-09-10-data-provenance-workshop.mp4">recording
from our workshop</a> (and <a
href="https://docs.evolveum.com/talks/files/2020-09-data-provenance-workshop.pdf">slides</a>)
that also includes the demo of metadata functionality. All the
other details can be found on <a
href="https://docs.evolveum.com/midpoint/midprivacy/phases/01-data-provenance-prototype/">project
page under the midPrivacy initiative</a>. If the concept of
metadata is new to you, then perhaps the <a
href="https://docs.evolveum.com/midpoint/midprivacy/phases/01-data-provenance-prototype/identity-metadata-in-a-nutshell/">Identity
Metadata In A Nutshell</a> story is a good place to start.</p>
<p>This project was really interesting and enlightening. Metadata
are one of the fundamental building blocks for data protection
functionality. But it is also an area that was not completely
explored yet. We have encountered a lot of challenges during the
project. Some of them were very expected, such as the difficulty
to design Axiom. But other challenges came entirely out of the
blue, such as <a
href="https://docs.evolveum.com/midpoint/midprivacy/phases/01-data-provenance-prototype/metadata-multiplicity-problem/">metadata
multiplicity problem</a>. Some of these challenges may perhaps
be even classified as discoveries. Anyway, we have dealt with
them in one way or another. The prototype was a success in both
ways: it uncovered hidden problems and we have a working code in
the end.</p>
<p>The prototype code is now integral part of midPoint. It will be
released in midPoint 4.2, which is planned to happen soon.
However, this is still a prototype. Entire metadata
functionality is marked as <i>experimental</i>. The new
implicit <i>value metadata</i> live alongside the old explicit
metadata. The old metadata as we know them from midPoint 3.x are
still there and they are fully supported. We have preferred
compatibility and decided not to use the new experimental code
until it is sufficiently stable. The new metadata functionality
is part of midPoint, but it is turned off by default.</p>
<p>Most of the costs of this project were covered by European
community funding, in the form of NGI_TRUST initiative. We are
more than thankful for this opportunity. I would like to thank
the mentors which were very helpful, especially given that this
was our first “Europroject”. However, we felt that we have to go
beyond the scope of original project proposal and therefore we
have also invested our own resources into the project.</p>
<p>Phase 1 of <a
href="https://docs.evolveum.com/midpoint/midprivacy/">midPrivacy
initiative</a> is done. But we are still far from our ultimate
goal. There is still a lot to work on to develop the data
protection and privacy functionality that we need. However, data
protection is quite a special field in many ways. One of the
characteristics of data protection is that it is very difficult
to secure commercial funding for data protection and privacy
features. We all know that data protection is needed, but it is
hard to get anyone to actually pay for it. Therefore the major
obstacle to continue midPrivacy initiative is, of course, the
funding. We have tried to follow-up by submitting several
proposals for European community funding. But sadly, none of the
proposals to continue midPrivacy was successful. Therefore the
future of midPrivacy is not certain yet. But one thing is
certain: data protection and privacy is absolutely necessary and
we are not giving up!</p>
</div>
<span class="byline"><span class="author vcard"></span></span>
<div class="entry-content"><span class="byline"><span class="author
vcard"></span></span>This project has received funding from
the European Union’s Horizon 2020 research and innovation
programme under the NGI_TRUST grant agreement no 825618. </div>
<p>(Reposted from <a moz-do-not-send="true"
href="https://evolveum.com/data-provenance-prototype-is-finished/">Evolveum
blog</a>)</p>
<pre class="moz-signature" cols="72">--
Radovan Semancik
Software Architect
evolveum.com</pre>
</body>
</html>