<div dir="ltr"><font color="#333333" face="Source Sans Pro, Helvetica Neue, Helvetica, Arial, sans-serif"><span style="font-size:14px;line-height:20px">I just rebuilt the latest snapshot (git-v3.5devel-221-g9744fa0) and am experiencing the same problem. </span></font><div><font color="#333333" face="Source Sans Pro, Helvetica Neue, Helvetica, Arial, sans-serif"><span style="font-size:14px;line-height:20px"><br></span></font></div><div><font color="#333333" face="Source Sans Pro, Helvetica Neue, Helvetica, Arial, sans-serif"><span style="font-size:14px;line-height:20px">Thanks,</span></font></div><div><font color="#333333" face="Source Sans Pro, Helvetica Neue, Helvetica, Arial, sans-serif"><span style="font-size:14px;line-height:20px">-F </span></font></div><div><div><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Aug 24, 2016 at 1:24 PM, Florin. Stingaciu <span dir="ltr"><<a href="mailto:fstingaciu@mirantis.com" target="_blank">fstingaciu@mirantis.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Upon further investigation I found that this error only occurs when I try to access the shadow repository via Configuration->Repository Objects->Shadows <div><br></div><div>It seems that something triggers the creation of DB connections until it runs out of available connections. For this particular instance, I'm at <span style="color:rgb(51,51,51);font-family:'Source Sans Pro','Helvetica Neue',Helvetica,Arial,sans-serif;font-size:14px;line-height:20px">git-v3.5devel-170-g26b0644.<wbr> </span></div><div><span style="color:rgb(51,51,51);font-family:'Source Sans Pro','Helvetica Neue',Helvetica,Arial,sans-serif;font-size:14px;line-height:20px"><br></span></div><div><span style="color:rgb(51,51,51);font-family:'Source Sans Pro','Helvetica Neue',Helvetica,Arial,sans-serif;font-size:14px;line-height:20px">Thanks, </span></div><span class="HOEnZb"><font color="#888888"><div><span style="color:rgb(51,51,51);font-family:'Source Sans Pro','Helvetica Neue',Helvetica,Arial,sans-serif;font-size:14px;line-height:20px">-F </span></div></font></span></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Aug 24, 2016 at 1:13 PM, Florin. Stingaciu <span dir="ltr"><<a href="mailto:fstingaciu@mirantis.com" target="_blank">fstingaciu@mirantis.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hello Pavol, <div><br></div><div>After running this configuration my dev environment with two instances up for a couple of days, I started to get the following error messages: <a href="http://pastebin.com/nkxE9LNA" target="_blank">http://pastebin.com/<wbr>nkxE9LNA</a></div><div><br></div><div>It seems as though the connection pool is not properly cleaned up. The database logs indicate no failure and in fact my production environment is hooked up to the same DB instance and is working fine. </div><div><br></div><div>Any recommendations would be very appreciated. </div><div><br></div><div>Thanks, </div><span><font color="#888888"><div>-F</div></font></span></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Aug 15, 2016 at 3:29 PM, Florin. Stingaciu <span dir="ltr"><<a href="mailto:fstingaciu@mirantis.com" target="_blank">fstingaciu@mirantis.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Awesome! Thanks for all your help today Pavol. I will try and test in my environment and let you know how that goes. <div><br></div><div>Thanks again!</div><span><font color="#888888"><div>-F </div></font></span></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Aug 15, 2016 at 3:00 PM, Pavol Mederly <span dir="ltr"><<a href="mailto:mederly@evolveum.com" target="_blank">mederly@evolveum.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<p>Florin,</p>
<p>I think I've fixed it: <a href="https://github.com/Evolveum/midpoint/commit/c00f2979cea6f6ada0a1194b13bad51c0ecec2e3" target="_blank">https://github.com/Evolveum/mi<wbr>dpoint/commit/c00f2979cea6f6ad<wbr>a0a1194b13bad51c0ecec2e3</a><br>
</p>
<p>Cannot verify as the issue isn't visible on my computer, but I am
quite convinced it should be fixed now. The suspected cause is
that some query-only transactions weren't closed properly. C3P0
closes them automatically but <a href="http://stackoverflow.com/questions/19396563/does-hibernates-session-close-automatically-rollback-uncommitted-transactions" target="_blank">other
connection providers probably not</a>.</p>
<p>You can safely use the master, or you can apply the above patch
to 3.4 yourself. The master is something like "3.4-support" these
days (preparing for 3.4.1), so it should be of production-level
quality, or at least near it. <br>
(I'm going to run full test suite during the night, but I don't
expect any failures. Or you can check after approx 3 hours <a href="https://travis-ci.org/Evolveum/midpoint" target="_blank">here</a>.)</p><span>
<p>Best regards,<br>
</p>
<pre cols="72">Pavol Mederly
Software developer
<a href="http://evolveum.com" target="_blank">evolveum.com</a>
</pre>
</span><div><div><div>On 15.08.2016 22:57, Florin. Stingaciu
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hey Pavol,
<div><br>
</div>
<div>I uploaded the log file you requested to my gdrive (<a href="https://drive.google.com/file/d/0B9f4-vWBsztoZy1zWThGYnpGeGs/view?usp=sharing" target="_blank">https://drive.google.com/file<wbr>/d/0B9f4-vWBsztoZy1zWThGYnpGeG<wbr>s/view?usp=sharing</a>)
It's a trace level debug so the file is quite large. Let me
know if anything jumps out at you. </div>
<div><br>
Thanks, </div>
<div>-F </div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Aug 15, 2016 at 1:07 PM, Pavol
Mederly <span dir="ltr"><<a href="mailto:mederly@evolveum.com" target="_blank">mederly@evolveum.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<p>Strange enough. I've replicated your setup on my
computer (using PostgreSQL, as I have no MySQL currently
installed), and it works: midPoint starts cleanly. I am
able to add/delete repository objects.<br>
</p>
<p>Attached are my config.xml and server.xml; they are
pretty much the same as yours.</p>
<p>So, I have no idea. You could try adding
<database>mysql</database> into
<repository> element in your config.xml, but it
would most probably make no difference.</p>
<p>As last resort, you could turn on the most detailed
debugging for the repository on startup by adding a line
like this</p>
<p><logger name="<a href="http://com.evolveum.midpoint.re" target="_blank">com.evolveum.midpoint.re</a><wbr>po"
level="TRACE" /></p>
<p>into webapps\midpoint\WEB-INF\class<wbr>es\logback.xml
file and start midPoint. In idm.log there should be
quite a detailed record of what's going on.<br>
</p>
<span>
<pre cols="72">Pavol Mederly
Software developer
<a href="http://evolveum.com" target="_blank">evolveum.com</a>
</pre>
</span>
<div>
<div>
<div>On 15.08.2016 21:03, Florin. Stingaciu wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Here's my datasource config:<br>
<br>
<div> <Resource name="jdbc/mysql"
auth="Container" type="javax.sql.DataSource"</div>
<div> username="midpoint" password="pass"</div>
<div> url="jdbc:mysql://SERVER:3306/<wbr>midpoint_dev"</div>
<div> driverClassName="com.mysql.jdb<wbr>c.Driver"</div>
<div> accessToUnderlyingConnectionAl<wbr>lowed="true"</div>
<div> initialSize="5" maxWait="5000"</div>
<div> maxActive="30" maxIdle="5"</div>
<div> validationQuery="select 1"</div>
<div> poolPreparedStatements="true"/<wbr>></div>
<div><br>
</div>
<div>And here's my config.xml:<br>
<br>
<div> <repository></div>
<div>
<embedded>false</embedded></div>
<div>
<repositoryServiceFactoryClass<wbr>>com.evolveum.midpoint.repo.sq<wbr>l.SqlRepositoryFactory</reposi<wbr>toryServiceFactoryClass></div>
<div>
<hibernateHbm2ddl>validate</hi<wbr>bernateHbm2ddl></div>
<div>
<hibernateDialect>com.evolveum<wbr>.midpoint.repo.sql.util.MidPoi<wbr>ntMySQLDialect</hibernateDiale<wbr>ct></div>
<div>
<dataSource>java:comp/env/jdbc<wbr>/mysql</dataSource></div>
<div> </repository></div>
<div> <taskManager></div>
<div>
<clustered>true</clustered></div>
<div>
<jmxUsername>midpoint</jmxUser<wbr>name></div>
<div>
<jmxPassword>password</jmxPass<wbr>word></div>
<div> </taskManager></div>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Aug 15, 2016 at
11:44 AM, Pavol Mederly <span dir="ltr"><<a href="mailto:mederly@evolveum.com" target="_blank">mederly@evolveum.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<p>Hello Florin,</p>
<p>this is really interesting. Please, could
you also share your midPoint config.xml,
as well as your data source configuration?
(except credentials, of course)</p>
<span>
<p>Best regards,<br>
</p>
<pre cols="72">Pavol Mederly
Software developer
<a href="http://evolveum.com" target="_blank">evolveum.com</a>
</pre>
</span>
<div>
<div>
<div>On 15.08.2016 19:39, Florin.
Stingaciu wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hello Pavol,
<div><br>
</div>
<div>Thanks for your detailed
response. I tried setting up the
datasource with validationQuery
set up properly for the mySQL
backed I have. However, upon
service restart I get the
following errors: <a href="http://pastebin.com/8dpGN0JC" target="_blank">http://pastebin.com/8d<wbr>pGN0JC</a><br>
<br>
To save you a click, it seems as
though the connection is set up in
readonly mode or some other
strange things happen. I've tried
setting <span>readonly</span><span>=</span><span>"false"</span><span> in
the resource definition in
server.xml but that didn't help.
I will continue researching
this, however any guidance would
be quite appreciated. </span></div>
<div><span><br>
</span></div>
<div><span>Thanks!</span></div>
<div><span>-F </span></div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Aug
15, 2016 at 2:29 AM, Pavol Mederly
<span dir="ltr"><<a href="mailto:mederly@evolveum.com" target="_blank">mederly@evolveum.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<p>Hello Florin,</p>
<p>having looked at your logs,
it seems that maybe explicit
setting of validationQuery
in Quartz data source setup
would help. (See <a href="http://www.quartz-scheduler.org/documentation/quartz-2.x/configuration/ConfigDataSources.html" target="_blank">http://www.quartz-scheduler.or<wbr>g/documentation/quartz-2.x/con<wbr>figuration/ConfigDataSources.h<wbr>tml</a>.)</p>
<p>Unfortunately, current
midPoint implementation does
not allow to configure
Quartz data source
parameters. So, there are
the following three
possibilities:</p>
<ol>
<li>Take an alternative
route, and use application
server-defined data source
(with validationQuery set
up).<br>
</li>
<li>Patch Task Manager
implementation by adding
appropriate lines to
Quartz configuration (see
LocalNodeManager.java:87-90).</li>
<li>Wait until we implement
it - I've created an issue
<a href="https://jira.evolveum.com/browse/MID-3347" target="_blank">MID-3347</a>
for this.<br>
</li>
</ol>
<p>As for the first one
(externally defined data
source): Please see <a href="https://wiki.evolveum.com/display/midPoint/Repository+Configuration#RepositoryConfiguration-Datasourceconfiguration" target="_blank">https://wiki.evolveum.com/disp<wbr>lay/midPoint/Repository+Config<wbr>uration#RepositoryConfiguratio<wbr>n-Datasourceconfiguration</a>
on how to configure midPoint
repository with the data
source. This data source
will be used also by Quartz,
if not overriden in
<taskManager> section.
It should work but I don't
remember if someone actually
tested this.</p>
<p>Concerning autoReconnect: I
have no experiences with
this setting. I agree with
you that about the
hesitation of using it in
production environment. If
really needed, I'd recommend
to separate midPoint
repository configuration
from Quartz configuration by
using different JDBC URLs
for the two: the standard
one for the repository and
the one with
"autoReconnect=true" for
Quartz. In this way,
potential negative effects
should be restricted to task
management functionality
only. But, overall, I'd
suggest trying to eliminate
the problem by setting
validationQuery first.<br>
</p>
<p>Best regards,<br>
</p>
<pre cols="72">Pavol Mederly
Software developer
<a href="http://evolveum.com" target="_blank">evolveum.com</a>
</pre>
<div>
<div>
<div>On 14.08.2016 22:40,
Florin. Stingaciu wrote:<br>
</div>
</div>
</div>
<blockquote type="cite">
<div>
<div>
<div dir="ltr">Hello,
<div><br>
</div>
<div>I'm trying to
configure an
active/active
configuration, and
experiencing some
issues with the
Quartz scheduler.
The SQL connection
seems to timeout
quite often and
result in many
warning messages.
I'm also
experiencing some
errors -- as the
timeout closes the
connection, some
processes are still
trying to commit
using that stale
handler. </div>
<div><br>
</div>
<div>Here are some
relevant logs from
both midPoint
instances in this
cluster: <a href="http://pastebin.com/JFjqrLnT" target="_blank">http://pastebin.com/J<wbr>FjqrLnT</a> <a href="http://pastebin.com/1vy4iYPY" target="_blank">http://pastebin.com/1v<wbr>y4iYPY</a><br>
<br>
Also here's the
relevant portion of
my config.xml file: <a href="http://pastebin.com/WbydcCC8" target="_blank">http://pastebin.com/Wbyd<wbr>cCC8</a></div>
<div><br>
Following the
suggestions in the
warnings, I've
started to look at
autoReconnect
propriety of the
JDBC connector and
applied it to my
configs like so:</div>
<div><br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span><jdbcUrl>jdbc:mysql://SERVER:3<wbr>306/midpoint_dev?autoReconnect<wbr>=true</jdbcUrl></span></blockquote>
<div><br>
</div>
<div>Since
implementing this
change, the errors
and warnings seem
to have
disappeared. I
will continue to
monitor the logs
and ensure this
actually the case.
</div>
<div><br>
</div>
<div>Reading the
mysql docs, I
found that this is
not recommended as
this may cause
data inconsistency
issues and that
stale connection
exceptions should
be properly caught
within the
application.
Namely:</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">The
use of this
feature is not
recommended,
because it has
side effects
related to session
state and data
consistency when
applications don't
handle
SQLExceptions
properly, and is
only designed to
be used when you
are unable to
configure your
application to
handle
SQLExceptions
resulting from
dead and stale
connections
properly.</blockquote>
<div><br>
Do you have any
recommended
configuration for
this scenario? I'd
like to move my
current production
environment in an
active active
configuration,
however as of
right now I'm
hesitant to do so
in order to avoid
any data
corruption.
Especially since
it's quite
difficult to test
for data
consistency issues
that may arise
from using
autoReconnect in
my dev
environment. <br>
<br>
Thanks, </div>
<div>-F </div>
<div> </div>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
</div>
</div>
<pre>______________________________<wbr>_________________
midPoint mailing list
<a href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>
<a href="http://lists.evolveum.com/mailman/listinfo/midpoint" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a>
</pre>
</blockquote>
</div>
______________________________<wbr>_________________
midPoint mailing list
<a href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>
<a href="http://lists.evolveum.com/mailman/listinfo/midpoint" rel="noreferrer" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a>
</blockquote></div>
</div>
<fieldset></fieldset>
<pre>______________________________<wbr>_________________
midPoint mailing list
<a href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>
<a href="http://lists.evolveum.com/mailman/listinfo/midpoint" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a>
</pre>
</blockquote>
</div></div></div>
______________________________<wbr>_________________
midPoint mailing list
<a href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>
<a href="http://lists.evolveum.com/mailman/listinfo/midpoint" rel="noreferrer" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a>
</blockquote></div>
</div>
<fieldset></fieldset>
<pre>______________________________<wbr>_________________
midPoint mailing list
<a href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>
<a href="http://lists.evolveum.com/mailman/listinfo/midpoint" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a>
</pre>
</blockquote>
</div></div></div>
______________________________<wbr>_________________
midPoint mailing list
<a href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>
<a href="http://lists.evolveum.com/mailman/listinfo/midpoint" rel="noreferrer" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a>
</blockquote></div>
</div>
<fieldset></fieldset>
<pre>______________________________<wbr>_________________
midPoint mailing list
<a href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>
<a href="http://lists.evolveum.com/mailman/listinfo/midpoint" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a>
</pre>
</blockquote>
</div></div></div><br>______________________________<wbr>_________________<br>
midPoint mailing list<br>
<a href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a><br>
<a href="http://lists.evolveum.com/mailman/listinfo/midpoint" rel="noreferrer" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a><br>
<br></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>