<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>Hello Florin,</p>
<p>it seems that either midPoint (i.e. hibernate), Quartz scheduler,
or Activiti BPMS use connections without properly closing them. <br>
</p>
<p>First step to diagnose it would be perhaps to setup things like
this:</p>
<ol>
<li>Let midPoint and Activiti use their own data source (by
configuring them with a JDBC URL): see <a
href="https://wiki.evolveum.com/display/midPoint/Repository+Configuration">Repository
configuration</a> and <a
href="https://wiki.evolveum.com/display/midPoint/Workflow+configuration">Workflow
configuration</a> wiki pages. (If you don't use approvals you
can disable workflows altogether by setting "enabled" to
"false".)</li>
<li>Keep Quartz schedule using tomcat data source: something like</li>
</ol>
<div class="line number13 index12 alt2"><code class="html plain"><</code><code
class="html keyword">taskManager</code><code class="html plain">></code></div>
<div class="line number14 index13 alt1"><code class="html spaces"> </code><code
class="html plain"><</code><code class="html keyword">clustered</code><code
class="html plain">>true</</code><code class="html
keyword">clustered</code><code class="html plain">><br>
<dataSource>...your data source name...</dataSource><br>
</code></div>
...<br>
<div class="line number18 index17 alt1"><code class="html plain"></</code><code
class="html keyword">taskManager</code><code class="html plain">><br>
</code><br>
And you'll see if the problem would appear again. (Maybe you could
even configure three Tomcat data sources; one for midPoint, second
for Quartz, and the third for Activiti, if needed. But I have not
done this yet; these are pure speculations of mine...)<br>
<br>
Best regards,<br>
</div>
<pre class="moz-signature" cols="72">Pavol Mederly
Software developer
evolveum.com
</pre>
<div class="moz-cite-prefix">On 24.08.2016 22:41, Florin. Stingaciu
wrote:<br>
</div>
<blockquote
cite="mid:CAMQHPY24srqDBgC+vri16kZ3b7UWV8Q8Se5RUwGGY0ATKquEvw@mail.gmail.com"
type="cite">
<div dir="ltr"><font face="Source Sans Pro, Helvetica Neue,
Helvetica, Arial, sans-serif" color="#333333"><span
style="font-size:14px;line-height:20px">I just rebuilt the
latest snapshot (git-v3.5devel-221-g9744fa0) and am
experiencing the same problem. </span></font>
<div><font face="Source Sans Pro, Helvetica Neue, Helvetica,
Arial, sans-serif" color="#333333"><span
style="font-size:14px;line-height:20px"><br>
</span></font></div>
<div><font face="Source Sans Pro, Helvetica Neue, Helvetica,
Arial, sans-serif" color="#333333"><span
style="font-size:14px;line-height:20px">Thanks,</span></font></div>
<div><font face="Source Sans Pro, Helvetica Neue, Helvetica,
Arial, sans-serif" color="#333333"><span
style="font-size:14px;line-height:20px">-F </span></font></div>
<div>
<div><br>
</div>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed, Aug 24, 2016 at 1:24 PM,
Florin. Stingaciu <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:fstingaciu@mirantis.com" target="_blank">fstingaciu@mirantis.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Upon further investigation I found that this
error only occurs when I try to access the shadow
repository via Configuration->Repository
Objects->Shadows
<div><br>
</div>
<div>It seems that something triggers the creation of DB
connections until it runs out of available connections.
For this particular instance, I'm at <span
style="color:rgb(51,51,51);font-family:'Source Sans
Pro','Helvetica
Neue',Helvetica,Arial,sans-serif;font-size:14px;line-height:20px">git-v3.5devel-170-g26b0644.<wbr> </span></div>
<div><span style="color:rgb(51,51,51);font-family:'Source
Sans Pro','Helvetica
Neue',Helvetica,Arial,sans-serif;font-size:14px;line-height:20px"><br>
</span></div>
<div><span style="color:rgb(51,51,51);font-family:'Source
Sans Pro','Helvetica
Neue',Helvetica,Arial,sans-serif;font-size:14px;line-height:20px">Thanks, </span></div>
<span class="HOEnZb"><font color="#888888">
<div><span
style="color:rgb(51,51,51);font-family:'Source
Sans Pro','Helvetica
Neue',Helvetica,Arial,sans-serif;font-size:14px;line-height:20px">-F </span></div>
</font></span></div>
<div class="HOEnZb">
<div class="h5">
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed, Aug 24, 2016 at 1:13
PM, Florin. Stingaciu <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:fstingaciu@mirantis.com"
target="_blank">fstingaciu@mirantis.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hello Pavol,
<div><br>
</div>
<div>After running this configuration my dev
environment with two instances up for a couple
of days, I started to get the following error
messages: <a moz-do-not-send="true"
href="http://pastebin.com/nkxE9LNA"
target="_blank">http://pastebin.com/<wbr>nkxE9LNA</a></div>
<div><br>
</div>
<div>It seems as though the connection pool is
not properly cleaned up. The database logs
indicate no failure and in fact my production
environment is hooked up to the same DB
instance and is working fine. </div>
<div><br>
</div>
<div>Any recommendations would be very
appreciated. </div>
<div><br>
</div>
<div>Thanks, </div>
<span><font color="#888888">
<div>-F</div>
</font></span></div>
<div>
<div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Aug 15,
2016 at 3:29 PM, Florin. Stingaciu <span
dir="ltr"><<a moz-do-not-send="true"
href="mailto:fstingaciu@mirantis.com"
target="_blank">fstingaciu@mirantis.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
<div dir="ltr">Awesome! Thanks for all
your help today Pavol. I will try and
test in my environment and let you
know how that goes.
<div><br>
</div>
<div>Thanks again!</div>
<span><font color="#888888">
<div>-F </div>
</font></span></div>
<div>
<div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon,
Aug 15, 2016 at 3:00 PM, Pavol
Mederly <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:mederly@evolveum.com"
target="_blank">mederly@evolveum.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div bgcolor="#FFFFFF"
text="#000000">
<p>Florin,</p>
<p>I think I've fixed it: <a
moz-do-not-send="true"
href="https://github.com/Evolveum/midpoint/commit/c00f2979cea6f6ada0a1194b13bad51c0ecec2e3"
target="_blank">https://github.com/Evolveum/mi<wbr>dpoint/commit/c00f2979cea6f6ad<wbr>a0a1194b13bad51c0ecec2e3</a><br>
</p>
<p>Cannot verify as the
issue isn't visible on my
computer, but I am quite
convinced it should be
fixed now. The suspected
cause is that some
query-only transactions
weren't closed properly.
C3P0 closes them
automatically but <a
moz-do-not-send="true"
href="http://stackoverflow.com/questions/19396563/does-hibernates-session-close-automatically-rollback-uncommitted-transactions"
target="_blank">other
connection providers
probably not</a>.</p>
<p>You can safely use the
master, or you can apply
the above patch to 3.4
yourself. The master is
something like
"3.4-support" these days
(preparing for 3.4.1), so
it should be of
production-level quality,
or at least near it. <br>
(I'm going to run full
test suite during the
night, but I don't expect
any failures. Or you can
check after approx 3 hours
<a moz-do-not-send="true"
href="https://travis-ci.org/Evolveum/midpoint" target="_blank">here</a>.)</p>
<span>
<p>Best regards,<br>
</p>
<pre cols="72">Pavol Mederly
Software developer
<a moz-do-not-send="true" href="http://evolveum.com" target="_blank">evolveum.com</a>
</pre>
</span>
<div>
<div>
<div>On 15.08.2016
22:57, Florin.
Stingaciu wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hey
Pavol,
<div><br>
</div>
<div>I uploaded the
log file you
requested to my
gdrive (<a
moz-do-not-send="true"
href="https://drive.google.com/file/d/0B9f4-vWBsztoZy1zWThGYnpGeGs/view?usp=sharing"
target="_blank">https://drive.google.com/file<wbr>/d/0B9f4-vWBsztoZy1zWThGYnpGeG<wbr>s/view?usp=sharing</a>)
It's a trace level
debug so the file
is quite large.
Let me know if
anything jumps out
at you. </div>
<div><br>
Thanks, </div>
<div>-F </div>
</div>
<div
class="gmail_extra"><br>
<div
class="gmail_quote">On
Mon, Aug 15, 2016
at 1:07 PM, Pavol
Mederly <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:mederly@evolveum.com"
target="_blank">mederly@evolveum.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0
0 0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
<div
bgcolor="#FFFFFF"
text="#000000">
<p>Strange
enough. I've
replicated
your setup on
my computer
(using
PostgreSQL, as
I have no
MySQL
currently
installed),
and it works:
midPoint
starts
cleanly. I am
able to
add/delete
repository
objects.<br>
</p>
<p>Attached
are my
config.xml and
server.xml;
they are
pretty much
the same as
yours.</p>
<p>So, I have
no idea. You
could try
adding
<database>mysql</database>
into
<repository>
element in
your
config.xml,
but it would
most probably
make no
difference.</p>
<p>As last
resort, you
could turn on
the most
detailed
debugging for
the repository
on startup by
adding a line
like this</p>
<p><logger
name="<a
moz-do-not-send="true"
href="http://com.evolveum.midpoint.re" target="_blank">com.evolveum.midpoint.re</a><wbr>po"
level="TRACE"
/></p>
<p>into
webapps\midpoint\WEB-INF\class<wbr>es\logback.xml
file and start
midPoint. In
idm.log there
should be
quite a
detailed
record of
what's going
on.<br>
</p>
<span>
<pre cols="72">Pavol Mederly
Software developer
<a moz-do-not-send="true" href="http://evolveum.com" target="_blank">evolveum.com</a>
</pre>
</span>
<div>
<div>
<div>On
15.08.2016
21:03, Florin.
Stingaciu
wrote:<br>
</div>
<blockquote
type="cite">
<div dir="ltr">Here's
my datasource
config:<br>
<br>
<div>
<Resource
name="jdbc/mysql"
auth="Container" type="javax.sql.DataSource"</div>
<div>
username="midpoint"
password="pass"</div>
<div>
url="jdbc:mysql://SERVER:3306/<wbr>midpoint_dev"</div>
<div>
driverClassName="com.mysql.jdb<wbr>c.Driver"</div>
<div>
accessToUnderlyingConnectionAl<wbr>lowed="true"</div>
<div>
initialSize="5"
maxWait="5000"</div>
<div>
maxActive="30"
maxIdle="5"</div>
<div>
validationQuery="select
1"</div>
<div>
poolPreparedStatements="true"/<wbr>></div>
<div><br>
</div>
<div>And
here's my
config.xml:<br>
<br>
<div>
<repository></div>
<div>
<embedded>false</embedded></div>
<div>
<repositoryServiceFactoryClass<wbr>>com.evolveum.midpoint.repo.sq<wbr>l.SqlRepositoryFactory</reposi<wbr>toryServiceFactoryClass></div>
<div>
<hibernateHbm2ddl>validate</hi<wbr>bernateHbm2ddl></div>
<div>
<hibernateDialect>com.evolveum<wbr>.midpoint.repo.sql.util.MidPoi<wbr>ntMySQLDialect</hibernateDiale<wbr>ct></div>
<div>
<dataSource>java:comp/env/jdbc<wbr>/mysql</dataSource></div>
<div>
</repository></div>
<div>
<taskManager></div>
<div>
<clustered>true</clustered></div>
<div>
<jmxUsername>midpoint</jmxUser<wbr>name></div>
<div>
<jmxPassword>password</jmxPass<wbr>word></div>
<div>
</taskManager></div>
</div>
</div>
<div
class="gmail_extra"><br>
<div
class="gmail_quote">On
Mon, Aug 15,
2016 at 11:44
AM, Pavol
Mederly <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:mederly@evolveum.com"
target="_blank">mederly@evolveum.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div
bgcolor="#FFFFFF"
text="#000000">
<p>Hello
Florin,</p>
<p>this is
really
interesting.
Please, could
you also share
your midPoint
config.xml, as
well as your
data source
configuration?
(except
credentials,
of course)</p>
<span>
<p>Best
regards,<br>
</p>
<pre cols="72">Pavol Mederly
Software developer
<a moz-do-not-send="true" href="http://evolveum.com" target="_blank">evolveum.com</a>
</pre>
</span>
<div>
<div>
<div>On
15.08.2016
19:39, Florin.
Stingaciu
wrote:<br>
</div>
<blockquote
type="cite">
<div dir="ltr">Hello
Pavol,
<div><br>
</div>
<div>Thanks
for your
detailed
response. I
tried setting
up the
datasource
with
validationQuery
set up
properly for
the mySQL
backed I have.
However, upon
service
restart I get
the following
errors: <a
moz-do-not-send="true"
href="http://pastebin.com/8dpGN0JC" target="_blank">http://pastebin.com/8d<wbr>pGN0JC</a><br>
<br>
To save you a
click, it
seems as
though the
connection is
set up in
readonly mode
or some other
strange things
happen. I've
tried setting <span>readonly</span><span>=</span><span>"false"</span><span> in
the resource
definition in
server.xml but
that didn't
help. I will
continue
researching
this, however
any guidance
would be quite
appreciated. </span></div>
<div><span><br>
</span></div>
<div><span>Thanks!</span></div>
<div><span>-F
</span></div>
</div>
<div
class="gmail_extra"><br>
<div
class="gmail_quote">On
Mon, Aug 15,
2016 at 2:29
AM, Pavol
Mederly <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:mederly@evolveum.com"
target="_blank">mederly@evolveum.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div
bgcolor="#FFFFFF"
text="#000000">
<p>Hello
Florin,</p>
<p>having
looked at your
logs, it seems
that maybe
explicit
setting of
validationQuery
in Quartz data
source setup
would help.
(See <a
moz-do-not-send="true"
href="http://www.quartz-scheduler.org/documentation/quartz-2.x/configuration/ConfigDataSources.html"
target="_blank">http://www.quartz-scheduler.or<wbr>g/documentation/quartz-2.x/con<wbr>figuration/ConfigDataSources.h<wbr>tml</a>.)</p>
<p>Unfortunately,
current
midPoint
implementation
does not allow
to configure
Quartz data
source
parameters.
So, there are
the following
three
possibilities:</p>
<ol>
<li>Take an
alternative
route, and use
application
server-defined
data source
(with
validationQuery
set up).<br>
</li>
<li>Patch Task
Manager
implementation
by adding
appropriate
lines to
Quartz
configuration
(see
LocalNodeManager.java:87-90).</li>
<li>Wait until
we implement
it - I've
created an
issue <a
moz-do-not-send="true"
href="https://jira.evolveum.com/browse/MID-3347" target="_blank">MID-3347</a>
for this.<br>
</li>
</ol>
<p>As for the
first one
(externally
defined data
source):
Please see <a
moz-do-not-send="true"
href="https://wiki.evolveum.com/display/midPoint/Repository+Configuration#RepositoryConfiguration-Datasourceconfiguration"
target="_blank">https://wiki.evolveum.com/disp<wbr>lay/midPoint/Repository+Config<wbr>uration#RepositoryConfiguratio<wbr>n-Datasourceconfiguration</a>
on how to
configure
midPoint
repository
with the data
source. This
data source
will be used
also by
Quartz, if not
overriden in
<taskManager>
section. It
should work
but I don't
remember if
someone
actually
tested this.</p>
<p>Concerning
autoReconnect:
I have no
experiences
with this
setting. I
agree with you
that about the
hesitation of
using it in
production
environment.
If really
needed, I'd
recommend to
separate
midPoint
repository
configuration
from Quartz
configuration
by using
different JDBC
URLs for the
two: the
standard one
for the
repository and
the one with
"autoReconnect=true"
for Quartz. In
this way,
potential
negative
effects should
be restricted
to task
management
functionality
only. But,
overall, I'd
suggest trying
to eliminate
the problem by
setting
validationQuery
first.<br>
</p>
<p>Best
regards,<br>
</p>
<pre cols="72">Pavol Mederly
Software developer
<a moz-do-not-send="true" href="http://evolveum.com" target="_blank">evolveum.com</a>
</pre>
<div>
<div>
<div>On
14.08.2016
22:40, Florin.
Stingaciu
wrote:<br>
</div>
</div>
</div>
<blockquote
type="cite">
<div>
<div>
<div dir="ltr">Hello,
<div><br>
</div>
<div>I'm
trying to
configure an
active/active
configuration,
and
experiencing
some issues
with the
Quartz
scheduler. The
SQL connection
seems to
timeout quite
often and
result in many
warning
messages. I'm
also
experiencing
some errors --
as the timeout
closes the
connection,
some processes
are still
trying to
commit using
that stale
handler. </div>
<div><br>
</div>
<div>Here are
some relevant
logs from both
midPoint
instances in
this cluster: <a
moz-do-not-send="true" href="http://pastebin.com/JFjqrLnT"
target="_blank">http://pastebin.com/J<wbr>FjqrLnT</a> <a
moz-do-not-send="true" href="http://pastebin.com/1vy4iYPY"
target="_blank">http://pastebin.com/1v<wbr>y4iYPY</a><br>
<br>
Also here's
the relevant
portion of my
config.xml
file: <a
moz-do-not-send="true"
href="http://pastebin.com/WbydcCC8" target="_blank">http://pastebin.com/Wbyd<wbr>cCC8</a></div>
<div><br>
Following the
suggestions in
the warnings,
I've started
to look at
autoReconnect
propriety of
the JDBC
connector and
applied it to
my configs
like so:</div>
<div><br>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span><jdbcUrl>jdbc:mysql://SERVER:3<wbr>306/midpoint_dev?autoReconnect<wbr>=true</jdbcUrl></span></blockquote>
<div><br>
</div>
<div>Since
implementing
this change,
the errors and
warnings seem
to have
disappeared. I
will continue
to monitor the
logs and
ensure this
actually the
case. </div>
<div><br>
</div>
<div>Reading
the mysql
docs, I found
that this is
not
recommended as
this may cause
data
inconsistency
issues and
that stale
connection
exceptions
should be
properly
caught within
the
application.
Namely:</div>
<div><br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">The
use of this
feature is not
recommended,
because it has
side effects
related to
session state
and data
consistency
when
applications
don't handle
SQLExceptions
properly, and
is only
designed to be
used when you
are unable to
configure your
application to
handle
SQLExceptions
resulting from
dead and stale
connections
properly.</blockquote>
<div><br>
Do you have
any
recommended
configuration
for this
scenario? I'd
like to move
my current
production
environment in
an active
active
configuration,
however as of
right now I'm
hesitant to do
so in order to
avoid any data
corruption.
Especially
since it's
quite
difficult to
test for data
consistency
issues that
may arise from
using
autoReconnect
in my dev
environment. <br>
<br>
Thanks, </div>
<div>-F </div>
<div> </div>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
</div>
</div>
<pre>______________________________<wbr>_________________
midPoint mailing list
<a moz-do-not-send="true" href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>
<a moz-do-not-send="true" href="http://lists.evolveum.com/mailman/listinfo/midpoint" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a>
</pre>
</blockquote>
</div>
______________________________<wbr>_________________
midPoint mailing list
<a moz-do-not-send="true" href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>
<a moz-do-not-send="true" href="http://lists.evolveum.com/mailman/listinfo/midpoint" rel="noreferrer" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a>
</blockquote></div>
</div>
<fieldset></fieldset>
<pre>______________________________<wbr>_________________
midPoint mailing list
<a moz-do-not-send="true" href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>
<a moz-do-not-send="true" href="http://lists.evolveum.com/mailman/listinfo/midpoint" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a>
</pre>
</blockquote>
</div></div></div>
______________________________<wbr>_________________
midPoint mailing list
<a moz-do-not-send="true" href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>
<a moz-do-not-send="true" href="http://lists.evolveum.com/mailman/listinfo/midpoint" rel="noreferrer" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a>
</blockquote></div>
</div>
<fieldset></fieldset>
<pre>______________________________<wbr>_________________
midPoint mailing list
<a moz-do-not-send="true" href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>
<a moz-do-not-send="true" href="http://lists.evolveum.com/mailman/listinfo/midpoint" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a>
</pre>
</blockquote>
</div></div></div>
______________________________<wbr>_________________
midPoint mailing list
<a moz-do-not-send="true" href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>
<a moz-do-not-send="true" href="http://lists.evolveum.com/mailman/listinfo/midpoint" rel="noreferrer" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a>
</blockquote></div>
</div>
<fieldset></fieldset>
<pre>______________________________<wbr>_________________
midPoint mailing list
<a moz-do-not-send="true" href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>
<a moz-do-not-send="true" href="http://lists.evolveum.com/mailman/listinfo/midpoint" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a>
</pre>
</blockquote>
</div></div></div>
______________________________<wbr>_________________
midPoint mailing list
<a moz-do-not-send="true" href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>
<a moz-do-not-send="true" href="http://lists.evolveum.com/mailman/listinfo/midpoint" rel="noreferrer" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a>
</blockquote></div>
</div>
</div></div></blockquote></div>
</div>
</div></div></blockquote></div>
</div>
</div></div></blockquote></div>
</div>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre wrap="">_______________________________________________
midPoint mailing list
<a class="moz-txt-link-abbreviated" href="mailto:midPoint@lists.evolveum.com">midPoint@lists.evolveum.com</a>
<a class="moz-txt-link-freetext" href="http://lists.evolveum.com/mailman/listinfo/midpoint">http://lists.evolveum.com/mailman/listinfo/midpoint</a>
</pre>
</blockquote>
</body></html>