<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Dear midPoint community,</p>
<p><span class="byline"><span class="author vcard"></span></span></p>
<div class="entry-content">
<p><a
href="https://docs.evolveum.com/midpoint/midprivacy/phases/01-data-provenance-prototype/">Data
provenance</a> development in midPoint has reached its second
milestone. While we are not at the end yet, there is already a
pile of interesting materials to have a look at. There are good
news, but there are also not so good news.<span id="more-6721"></span></p>
<p>First of all, some of you might be wondering what that <i>provenance</i>
thing is and why it is so important. You are not alone. Data
protection may look easy, but it is not an easy thing to
understand. Therefore I have put together <a
href="https://docs.evolveum.com/midpoint/midprivacy/phases/01-data-provenance-prototype/identity-metadata-in-a-nutshell/">“Identity
Metadata In A Nutshell”</a>, a document that explains the
metadata concepts and the way how we are going to implement it
in midPoint.</p>
<p>There is good news from the implementation effort. <a
href="https://docs.evolveum.com/midpoint/midprivacy/phases/01-data-provenance-prototype/axiom/spec/">Axiom</a>,
our data modeling language, is taking shape very nicely. As
usual, designing the language was much harder than we have
anticipated (even though we have expected it won’t be easy). But
I’m very pleased with the results so far. Most aspects of the
language are designed and it looks it works well for metadata
modeling. Significant part of Axiom-processing code is developed
and integrated into midPoint. The code works for metadata
modeling. There is still a long way to go to make Axiom a
universal data modeling language for midPoint (and other uses),
but the first results look more than promising.</p>
<p>Having a modeling language is one thing, but <a
href="https://docs.evolveum.com/midpoint/midprivacy/phases/01-data-provenance-prototype/metadata-usecases/">designing
actual metadata model</a> is quite a different thing. There is
no metadata standard and there seems to be no general agreement
how identity metadata should look like. Therefore we have done
our best to create a reasonable set of metadata and <a
href="https://github.com/Evolveum/midpoint/blob/master/infra/schema/src/main/resources/xml/ns/public/common/common-metadata-3.axiom">express
them in Axiom</a>. It is quite likely that this schema is not
final yet, but it allows us to go on to the next step of testing
and validation.</p>
<p>We have modeling language and metadata models now. But we still
need to set up midPoint to use the metadata. For all of you that
know us, it is perhaps no bit surprise that we have reused an
existing mechanism. Enter metadata mappings. As ordinary
mappings are applied to ordinary data, metadata mappings are
applied to metadata. The documentation is not yet completely up
to date, but the <a
href="https://docs.evolveum.com/midpoint/midprivacy/phases/01-data-provenance-prototype/identity-metadata-in-a-nutshell/">“Nutshell”
document</a> has some nice examples of metadata mappings.</p>
<p>As usual, we have made the system a bit more generic than
strictly necessary. Goal of this project phase was identity
provenance, but we have created a system that can handle almost
any kind of metadata. There are several built-in metadata types
in midPoint schemas and you can extend the system with a
completely custom metadata. Nevertheless, we have still kept our
primary goal in mind and identity provenance metadata play a
primary role in the solution. There is a robust schema for
identity provenance metadata and we have invested a lot of
design time into that. We especially focused on making the
provenance schema “future proof”, to make sure it can be
extended in the future to support advanced data protection
functionality.</p>
<p>Of course, everything is <a
href="https://docs.evolveum.com/midpoint/midprivacy/phases/01-data-provenance-prototype/challenges/">harder
that it seems</a>. Data protection may seem simple enough.
Yet, it is everything but simple. Identity management was all
about moving data around. But data protection adds a completely
new <i>dimension</i> to that. Data protection is all about
reasoning <i>behind</i> the data: how the data got here, how we
can process them, where we can send them, when to delete them.
We were aware of most of the data protection difficulties, but
work on identity provenance exposed even deeper issues. We had
to make a <a
href="https://docs.evolveum.com/midpoint/midprivacy/phases/01-data-provenance-prototype/provenance-origin-basis/">rough
design of future data protection functionality</a> to make
sure that our identity provenance functionality design was
right.</p>
<p>This is our second milestone and there is still one final part
of the project to finish. The final part is mostly focused on
testing, bugfixing and overall validation of the results. We
still have to improve user interface and experiment with user
experience, which may also lead to adjustments of metadata
schemas. Final weeks will be focused on demonstration of the
result and gathering user feedback.</p>
<p>We are very excited about the development that this project
brings. It is not just about the metadata. Axiom brings a
radical change and we have high hopes about the future. However,
please keep in mind that the goal of this project is identity
provenance <i>prototype</i>. Being a prototype, we are not yet
sure how useful this is going to be for practical deployments.
That is exactly what the prototype has to find out. You are more
than welcome to test the functionality. Just please keep in mind
that there may be limitations.</p>
<p>What we are not so much happy about is the immediate future
after this “provenance” phase of midPrivacy is finished. We
would absolutely love to move data protection functionality out
of prototype stage and make it production-ready. We have spent a
lot of time during the past six months to secure funding for
future development. We have submitted several proposals, mostly
to NGI open calls. Sadly, none of the proposals were successful.
Therefore it looks like we will have to put our data protection
efforts aside, at least for a while. It is a real pity to
suspend this project, especially after such a promising start.
We strongly believe that data protection is absolutely necessary
for the safety of our digital future. Yet it is almost
impossible to get funding for data protection feature
development from our commercial engagements. Therefore we will
be more than grateful to anyone willing to sponsor our data
protection efforts or anyone that knows about any other form of
funding that we could use. We keep a strong hope that we would
be able to resume working on <a
href="https://docs.evolveum.com/midpoint/midprivacy/">midPrivacy</a>
as soon as possible.</p>
</div>
<span class="byline"><span class="author vcard"></span></span>
<div class="entry-content"><span class="byline"><span class="author
vcard"></span></span>This project has received funding from
the European Union’s Horizon 2020 research and innovation
programme under the NGI_TRUST grant agreement no 825618. </div>
<p>(Reposted from <a moz-do-not-send="true"
href="https://evolveum.com/data-provenance-milestone-2/">Evolveum
blog</a>)</p>
<pre class="moz-signature" cols="72">--
Radovan Semancik
Software Architect
evolveum.com</pre>
</body>
</html>