<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>Florin,</p>
<p>I see...</p>
<p>I am out of ideas here ... so, logged JIRA issue for this (<a
href="https://jira.evolveum.com/browse/MID-3378">MID-3378</a>).<br>
</p>
<p>Maybe someone here uses clustering + MySQL and could help you
better. Or maybe some very detailed logging, or reviewing the
parameters (e.g. is "SELECT 1" reliable way of determining
connection validity? Probably yes, but...). <br>
</p>
<p>If you would have any more hints, we can discuss them. I just
don't know at the moment. :(<br>
</p>
<pre class="moz-signature" cols="72">Pavol Mederly
Software developer
evolveum.com
</pre>
<div class="moz-cite-prefix">On 30.08.2016 20:28, Florin. Stingaciu
wrote:<br>
</div>
<blockquote
cite="mid:CAMQHPY0XrM1ASShd0pw7=ZZqAvduAsiesZdaBRWYLUZ_gBnKFA@mail.gmail.com"
type="cite">
<div dir="ltr">Hey Pavol,
<div><br>
</div>
<div>Here's the relevant lines:</div>
<div><br>
</div>
<div>
<div>2016-08-30 18:18:44,443 [] [localhost-startStop-1] INFO
(com.evolveum.midpoint.task.quartzimpl.Initializer): Task
Manager initialization.</div>
<div>2016-08-30 18:18:44,452 [] [localhost-startStop-1] INFO
(com.evolveum.midpoint.task.quartzimpl.Initializer): Task
Manager: Quartz Job Store: JDBC, clustered. Threads: 10</div>
<div>2016-08-30 18:18:44,452 [] [localhost-startStop-1] INFO
(com.evolveum.midpoint.task.quartzimpl.TaskManagerConfiguration):
Quartz database is at java:comp/env/jdbc/mysql (a data
source)</div>
<div>2016-08-30 18:18:44,461 [] [localhost-startStop-1] INFO
(com.evolveum.midpoint.task.quartzimpl.cluster.NodeRegistrar):
Registering this node in the repository as 001 at <a
moz-do-not-send="true" href="http://100.73.170.188:20001">100.73.170.188:20001</a></div>
<div>2016-08-30 18:18:44,945 [] [localhost-startStop-1] INFO
(com.evolveum.midpoint.task.quartzimpl.execution.TaskSynchronizer):
Synchronizing Quartz job store with midPoint repository.</div>
<div>2016-08-30 18:18:46,036 [] [localhost-startStop-1] INFO
(com.evolveum.midpoint.task.quartzimpl.execution.TaskSynchronizer):
Synchronization of midpoint and Quartz task store finished.
26 task(s) existing in midPoint repository successfully
processed, resulting in 0 updated Quartz job(s). 0 task(s)
removed from Quartz job store. Processing of 0 task(s)
failed.</div>
<div>2016-08-30 18:18:46,037 [] [localhost-startStop-1] INFO
(com.evolveum.midpoint.task.quartzimpl.Initializer): Task
Manager initialized</div>
</div>
<div><br>
</div>
<div>Thanks, </div>
<div>-F </div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Aug 30, 2016 at 11:17 AM, Pavol
Mederly <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:mederly@evolveum.com" target="_blank">mederly@evolveum.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<p>Florin,</p>
<p>this is really interesting. The exception of <br>
</p>
<p><tt>Caused by: com.mysql.jdbc.exceptions.<wbr>jdbc4.CommunicationsException:
The last packet successfully received from the server
was 64,841,997 milliseconds ago. The last packet sent
successfully to the server was 64,841,998 milliseconds
ago. is longer than the server configured value of
'wait_timeout'. You should consider either expiring
and/or testing connection validity before use in your
application, increasing the server configured values
for client timeouts, or using the Connector/J
connection property 'autoReconnect=true' to avoid this
problem.</tt></p>
<p>occurs in the context of Quartz scheduler, where the
data source that checks connection validity is used!</p>
<p>When you start midPoint there should be messages like
these:</p>
<p><tt>2016-08-30 20:14:31,388 [] [RMI TCP
Connection(5)-127.0.0.1] INFO
(com.evolveum.midpoint.task.<wbr>quartzimpl.Initializer):
Task Manager: Quartz Job Store: JDBC, NOT clustered.
Threads: 10</tt><tt><br>
</tt><tt>2016-08-30 20:14:31,388 [] [RMI TCP
Connection(5)-127.0.0.1] INFO
(com.evolveum.midpoint.task.<wbr>quartzimpl.<wbr>TaskManagerConfiguration):
Quartz database is at jdbc:postgresql://localhost/<wbr>midpoint
(a JDBC URL)</tt></p>
<p>In your case there should be a reference to the data
source. Could you please verify it is OK, and perhaps
paste relevant lines here?<br>
</p>
<span class="">
<pre cols="72">Pavol Mederly
Software developer
<a moz-do-not-send="true" href="http://evolveum.com" target="_blank">evolveum.com</a>
</pre>
</span>
<div>
<div class="h5">
<div>On 30.08.2016 19:21, Florin. Stingaciu wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hey Pavol,
<div><br>
</div>
<div>Over night, I now have the same error
messages I initially had: <a
moz-do-not-send="true"
href="http://pastebin.com/iWeSaswD"
target="_blank">http://pastebin.com/<wbr>iWeSaswD</a></div>
<div><br>
</div>
<div>Thank you for all your help so far. Any
further recommendations would be really
appreciated. </div>
<div><br>
</div>
<div>Thanks, </div>
<div>-F </div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Aug 29, 2016 at
3:54 PM, Florin. Stingaciu <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:fstingaciu@mirantis.com"
target="_blank">fstingaciu@mirantis.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div dir="ltr">Hey Pavol,
<div><br>
</div>
<div>I switched the repository to the
configuration you mentioned, and I could
now access the shadow page. However (minus
the task manager config), this is the same
configuration as I previously had where I
ran into errors and then made this thread
because of it. For now I'm not seeing any
errors in the log files. </div>
<div><br>
</div>
<div>I'll update this thread once I run into
issues. As of right now, I'm still
definitely not comfortable moving to
active active considering all these
issues. Have you guys seen similar issues
before? Or do you believe I've just
misconfigured something?</div>
<div><br>
</div>
<div>Thanks,</div>
<div>-F </div>
<div><br>
</div>
<div><br>
</div>
</div>
<div>
<div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Aug 29,
2016 at 3:39 PM, Pavol Mederly <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:mederly@evolveum.com"
target="_blank">mederly@evolveum.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div bgcolor="#FFFFFF"
text="#000000">
<p>Hello Florin,</p>
<p>I've tried to use your setup in
my environment (slightly
modified), but I was not able to
replicate the exceptions. I've
tried in particular to play with
the listing of shadow objects.</p>
<p>So we perhaps have to continue
diagnosing on your side. <br>
</p>
<p>I suggest to try to isolate the
problem. If you could use the
following config:</p>
<p><span><span> </span></span></p>
<div> <repository></div>
<div>
<embedded>false</embedded></div>
<div>
<repositoryServiceFactoryClass<wbr>>com.evolveum.midpoint.repo.sq<wbr>l.SqlRepositoryFactory</reposi<wbr>toryServiceFactoryClass></div>
<div>
<hibernateHbm2ddl>validate</hi<wbr>bernateHbm2ddl></div>
<div>
<database>mysql</database><br>
</div>
<div>
<jdbcUsername>...</jdbcUsernam<wbr>e><br>
<jdbcPassword>...</jdbcPasswor<wbr>d><br>
<jdbcUrl>jdbc:mysql://servder:<wbr>3306/midpoint_dev</jdbcUrl><br>
</repository></div>
<span>
<div> <taskManager></div>
<div>
<clustered>true</clustered></div>
<div>
<jmxUsername>midpoint</jmxUser<wbr>name></div>
</span>
<div>
<jmxPassword>password</jmxPass<wbr>word><br>
<dataSource><span>java:comp/env/jdbc<wbr>/mysql</span></dataSource><br>
</div>
<div> </taskManager><br>
<workflow><br>
<enabled>false</enabled><br>
<span></span>
<div> </workflow><br>
</div>
<div><br>
</div>
...and try the testing. The
configuration simply makes
midPoint use its own connection
mechanism (c3p0) and Quartz use
the tomcat's connection pool.<br>
</div>
<span>
<div>Best regards,<br>
</div>
<pre cols="72">Pavol Mederly
Software developer
<a moz-do-not-send="true" href="http://evolveum.com" target="_blank">evolveum.com</a>
</pre>
</span>
<div>
<div>
<div>On 29.08.2016 20:17,
Florin. Stingaciu wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">As an update,
after having the instance
up for half an hour or so
with the above config, I
got the following errors
again: <a
moz-do-not-send="true"
href="http://pastebin.com/tJJVFVsu"
target="_blank">http://pastebin.com/tJJ<wbr>VFVsu</a>
<div><br>
</div>
<div>Also, in the
deployment, there's no
workflows at all and
very little user load as
it's just a dev env.</div>
<div><br>
</div>
<div>Thanks,</div>
<div>-F </div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Mon, Aug 29, 2016 at
10:43 AM, Florin.
Stingaciu <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:fstingaciu@mirantis.com" target="_blank">fstingaciu@mirantis.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
<div dir="ltr">Hey
Pavol,
<div><br>
</div>
<div>Currently, for
Quartz and
miPoint, I'm using
a tomcat7 defined
datasource with
the following
config:<span><br>
<br>
<div>
<Resource
name="jdbc/mysql"
auth="Container" type="javax.sql.DataSource"</div>
</span>
<div>
username="midpoint"
password="password"</div>
<div>
url="jdbc:mysql://servder:3306<wbr>/midpoint_dev"</div>
<span>
<div>
driverClassName="com.mysql.jdb<wbr>c.Driver"</div>
<div>
accessToUnderlyingConnectionAl<wbr>lowed="true"</div>
<div>
initialSize="5"
maxWait="5000"</div>
<div>
maxActive="30"
maxIdle="5"</div>
<div>
validationQuery="select
1"</div>
</span>
<div>
readonly="false"
</div>
<div>
poolPreparedStatements="true"/<wbr>><br>
<br>
And here's my
relevant
config.xml:<span><br>
<br>
<div>
<repository></div>
<div>
<embedded>false</embedded></div>
<div>
<repositoryServiceFactoryClass<wbr>>com.evolveum.midpoint.repo.sq<wbr>l.SqlRepositoryFactory</reposi<wbr>toryServiceFactoryClass></div>
<div>
<hibernateHbm2ddl>validate</hi<wbr>bernateHbm2ddl></div>
<div>
<hibernateDialect>com.evolveum<wbr>.midpoint.repo.sql.util.MidPoi<wbr>ntMySQLDialect</hibernateDiale<wbr>ct></div>
<div>
<dataSource>java:comp/env/jdbc<wbr>/mysql</dataSource></div>
<div>
</repository></div>
<div>
<taskManager></div>
<div>
<clustered>true</clustered></div>
<div>
<jmxUsername>midpoint</jmxUser<wbr>name></div>
<div>
<jmxPassword>password</jmxPass<wbr>word></div>
<div>
</taskManager></div>
</span></div>
<div><br>
</div>
<div>With this
configuration,
when I was
trying to access
the Shadows
page, I got the
errors mentioned
in my previous
message. </div>
<div><br>
</div>
<div>Following
your last email,
I added the
following
<dataSource>java:comp/env/jdbc<wbr>/mysql</dataSource>
under
<taskManager>.
Now when I try
to access the
Shadows Page I
get the
following
error: <a
moz-do-not-send="true"
href="http://imgur.com/a/391CU" target="_blank">http://imgur.com/a/391C<wbr>U</a>.
Please note
there are four
photos in that
album. Also the
reason, I chose
an image is due
to the fact that
this error does
not appear
anywhere in the
logs and it's in
the GUI only. </div>
<div><br>
</div>
<div>I would also
like to add that
even though this
error occurs,
midPoint still
continues to
function. Before
I added the
<dataSource>
in the
taskManager
section, even
though I was
using a
datasource for
the Repository,
when I tried to
access the
shadows page,
midPoint would
not work anymore
(due to lack of
mysql
connections)
until I
restarted
tomcat. </div>
<div><br>
</div>
<div>The last
point I want to
make is that
from what I've
seen so far,
this issue only
happens when I
try to access
the shadow page.
I don't see this
issue while
trying to access
any other type
of repository
objects. </div>
<div><br>
</div>
<div>Thanks, </div>
<span><font
color="#888888">
<div>-F</div>
<br>
</font></span></div>
</div>
<div>
<div>
<div
class="gmail_extra"><br>
<div
class="gmail_quote">On
Fri, Aug 26,
2016 at 3:41
PM, Pavol
Mederly <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:mederly@evolveum.com"
target="_blank">mederly@evolveum.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div
bgcolor="#FFFFFF"
text="#000000">
<p>Hello
Florin,</p>
<p>it seems
that either
midPoint (i.e.
hibernate),
Quartz
scheduler, or
Activiti BPMS
use
connections
without
properly
closing them.
<br>
</p>
<p>First step
to diagnose it
would be
perhaps to
setup things
like this:</p>
<ol>
<li>Let
midPoint and
Activiti use
their own data
source (by
configuring
them with a
JDBC URL): see
<a
moz-do-not-send="true"
href="https://wiki.evolveum.com/display/midPoint/Repository+Configuration"
target="_blank">Repository configuration</a> and <a
moz-do-not-send="true"
href="https://wiki.evolveum.com/display/midPoint/Workflow+configuration"
target="_blank">Workflow configuration</a> wiki pages. (If you don't use
approvals you
can disable
workflows
altogether by
setting
"enabled" to
"false".)</li>
<li>Keep
Quartz
schedule using
tomcat data
source:
something like</li>
</ol>
<div><code><</code><code>taskManager</code><code>></code></div>
<div><code> </code><code><</code><code>clustered</code><code>>true</</code><code>clustered</code><code>><br>
<dataSource>...your
data source
name...</dataSource><br>
</code></div>
...<br>
<div><code></</code><code>taskManager</code><code>><br>
</code><br>
And you'll see
if the problem
would appear
again. (Maybe
you could even
configure
three Tomcat
data sources;
one for
midPoint,
second for
Quartz, and
the third for
Activiti, if
needed. But I
have not done
this yet;
these are pure
speculations
of mine...)<br>
<br>
Best regards,<br>
</div>
<span>
<pre cols="72">Pavol Mederly
Software developer
<a moz-do-not-send="true" href="http://evolveum.com" target="_blank">evolveum.com</a>
</pre>
</span>
<div>
<div>
<div>On
24.08.2016
22:41, Florin.
Stingaciu
wrote:<br>
</div>
<blockquote
type="cite">
<div dir="ltr"><font
face="Source
Sans Pro,
Helvetica
Neue,
Helvetica,
Arial,
sans-serif"
color="#333333"><span
style="font-size:14px;line-height:20px">I just rebuilt the latest
snapshot
(git-v3.5devel-221-g9744fa0)
and am
experiencing
the same
problem. </span></font>
<div><font
face="Source
Sans Pro,
Helvetica
Neue,
Helvetica,
Arial,
sans-serif"
color="#333333"><span
style="font-size:14px;line-height:20px"><br>
</span></font></div>
<div><font
face="Source
Sans Pro,
Helvetica
Neue,
Helvetica,
Arial,
sans-serif"
color="#333333"><span
style="font-size:14px;line-height:20px">Thanks,</span></font></div>
<div><font
face="Source
Sans Pro,
Helvetica
Neue,
Helvetica,
Arial,
sans-serif"
color="#333333"><span
style="font-size:14px;line-height:20px">-F </span></font></div>
<div>
<div><br>
</div>
</div>
</div>
<div
class="gmail_extra"><br>
<div
class="gmail_quote">On
Wed, Aug 24,
2016 at 1:24
PM, Florin.
Stingaciu <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:fstingaciu@mirantis.com"
target="_blank">fstingaciu@mirantis.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Upon
further
investigation
I found that
this error
only occurs
when I try to
access the
shadow
repository via
Configuration->Repository Objects->Shadows
<div><br>
</div>
<div>It seems
that something
triggers the
creation of DB
connections
until it runs
out of
available
connections.
For this
particular
instance, I'm
at <span>git-v3.5devel-170-g26b0644.<wbr> </span></div>
<div><span><br>
</span></div>
<div><span>Thanks, </span></div>
<span><font
color="#888888">
<div><span>-F </span></div>
</font></span></div>
<div>
<div>
<div
class="gmail_extra"><br>
<div
class="gmail_quote">On
Wed, Aug 24,
2016 at 1:13
PM, Florin.
Stingaciu <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:fstingaciu@mirantis.com"
target="_blank">fstingaciu@mirantis.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hello
Pavol,
<div><br>
</div>
<div>After
running this
configuration
my dev
environment
with two
instances up
for a couple
of days, I
started to get
the following
error
messages: <a
moz-do-not-send="true"
href="http://pastebin.com/nkxE9LNA" target="_blank">http://pastebin.com/<wbr>nkxE9LNA</a></div>
<div><br>
</div>
<div>It seems
as though the
connection
pool is not
properly
cleaned up.
The database
logs indicate
no failure and
in fact my
production
environment is
hooked up to
the same DB
instance and
is working
fine. </div>
<div><br>
</div>
<div>Any
recommendations
would be very
appreciated. </div>
<div><br>
</div>
<div>Thanks, </div>
<span><font
color="#888888">
<div>-F</div>
</font></span></div>
<div>
<div>
<div
class="gmail_extra"><br>
<div
class="gmail_quote">On
Mon, Aug 15,
2016 at 3:29
PM, Florin.
Stingaciu <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:fstingaciu@mirantis.com"
target="_blank">fstingaciu@mirantis.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Awesome!
Thanks for all
your help
today Pavol. I
will try and
test in my
environment
and let you
know how that
goes.
<div><br>
</div>
<div>Thanks
again!</div>
<span><font
color="#888888">
<div>-F </div>
</font></span></div>
<div>
<div>
<div
class="gmail_extra"><br>
<div
class="gmail_quote">On
Mon, Aug 15,
2016 at 3:00
PM, Pavol
Mederly <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:mederly@evolveum.com"
target="_blank">mederly@evolveum.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div
bgcolor="#FFFFFF"
text="#000000">
<p>Florin,</p>
<p>I think
I've fixed it:
<a
moz-do-not-send="true"
href="https://github.com/Evolveum/midpoint/commit/c00f2979cea6f6ada0a1194b13bad51c0ecec2e3"
target="_blank">https://github.com/Evolveum/mi<wbr>dpoint/commit/c00f2979cea6f6ad<wbr>a0a1194b13bad51c0ecec2e3</a><br>
</p>
<p>Cannot
verify as the
issue isn't
visible on my
computer, but
I am quite
convinced it
should be
fixed now. The
suspected
cause is that
some
query-only
transactions
weren't closed
properly. C3P0
closes them
automatically
but <a
moz-do-not-send="true"
href="http://stackoverflow.com/questions/19396563/does-hibernates-session-close-automatically-rollback-uncommitted-transactions"
target="_blank">other connection providers probably not</a>.</p>
<p>You can
safely use the
master, or you
can apply the
above patch to
3.4 yourself.
The master is
something like
"3.4-support"
these days
(preparing for
3.4.1), so it
should be of
production-level
quality, or at
least near
it. <br>
(I'm going to
run full test
suite during
the night, but
I don't expect
any failures.
Or you can
check after
approx 3 hours
<a
moz-do-not-send="true"
href="https://travis-ci.org/Evolveum/midpoint" target="_blank">here</a>.)</p>
<span>
<p>Best
regards,<br>
</p>
<pre cols="72">Pavol Mederly
Software developer
<a moz-do-not-send="true" href="http://evolveum.com" target="_blank">evolveum.com</a>
</pre>
</span>
<div>
<div>
<div>On
15.08.2016
22:57, Florin.
Stingaciu
wrote:<br>
</div>
<blockquote
type="cite">
<div dir="ltr">Hey
Pavol,
<div><br>
</div>
<div>I
uploaded the
log file you
requested to
my gdrive (<a
moz-do-not-send="true"
href="https://drive.google.com/file/d/0B9f4-vWBsztoZy1zWThGYnpGeGs/view?usp=sharing"
target="_blank">https://drive.google.com/file<wbr>/d/0B9f4-vWBsztoZy1zWThGYnpGeG<wbr>s/view?usp=sharing</a>)
It's a trace
level debug so
the file is
quite large.
Let me know if
anything jumps
out at you. </div>
<div><br>
Thanks, </div>
<div>-F </div>
</div>
<div
class="gmail_extra"><br>
<div
class="gmail_quote">On
Mon, Aug 15,
2016 at 1:07
PM, Pavol
Mederly <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:mederly@evolveum.com"
target="_blank">mederly@evolveum.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div
bgcolor="#FFFFFF"
text="#000000">
<p>Strange
enough. I've
replicated
your setup on
my computer
(using
PostgreSQL, as
I have no
MySQL
currently
installed),
and it works:
midPoint
starts
cleanly. I am
able to
add/delete
repository
objects.<br>
</p>
<p>Attached
are my
config.xml and
server.xml;
they are
pretty much
the same as
yours.</p>
<p>So, I have
no idea. You
could try
adding
<database>mysql</database>
into
<repository>
element in
your
config.xml,
but it would
most probably
make no
difference.</p>
<p>As last
resort, you
could turn on
the most
detailed
debugging for
the repository
on startup by
adding a line
like this</p>
<p><logger
name="<a
moz-do-not-send="true"
href="http://com.evolveum.midpoint.re" target="_blank">com.evolveum.midpoint.re</a><wbr>po"
level="TRACE"
/></p>
<p>into
webapps\midpoint\WEB-INF\class<wbr>es\logback.xml
file and start
midPoint. In
idm.log there
should be
quite a
detailed
record of
what's going
on.<br>
</p>
<span>
<pre cols="72">Pavol Mederly
Software developer
<a moz-do-not-send="true" href="http://evolveum.com" target="_blank">evolveum.com</a>
</pre>
</span>
<div>
<div>
<div>On
15.08.2016
21:03, Florin.
Stingaciu
wrote:<br>
</div>
<blockquote
type="cite">
<div dir="ltr">Here's
my datasource
config:<br>
<br>
<div>
<Resource
name="jdbc/mysql"
auth="Container" type="javax.sql.DataSource"</div>
<div>
username="midpoint"
password="pass"</div>
<div>
url="jdbc:mysql://SERVER:3306/<wbr>midpoint_dev"</div>
<div>
driverClassName="com.mysql.jdb<wbr>c.Driver"</div>
<div>
accessToUnderlyingConnectionAl<wbr>lowed="true"</div>
<div>
initialSize="5"
maxWait="5000"</div>
<div>
maxActive="30"
maxIdle="5"</div>
<div>
validationQuery="select
1"</div>
<div>
poolPreparedStatements="true"/<wbr>></div>
<div><br>
</div>
<div>And
here's my
config.xml:<br>
<br>
<div>
<repository></div>
<div>
<embedded>false</embedded></div>
<div>
<repositoryServiceFactoryClass<wbr>>com.evolveum.midpoint.repo.sq<wbr>l.SqlRepositoryFactory</reposi<wbr>toryServiceFactoryClass></div>
<div>
<hibernateHbm2ddl>validate</hi<wbr>bernateHbm2ddl></div>
<div>
<hibernateDialect>com.evolveum<wbr>.midpoint.repo.sql.util.MidPoi<wbr>ntMySQLDialect</hibernateDiale<wbr>ct></div>
<div>
<dataSource>java:comp/env/jdbc<wbr>/mysql</dataSource></div>
<div>
</repository></div>
<div>
<taskManager></div>
<div>
<clustered>true</clustered></div>
<div>
<jmxUsername>midpoint</jmxUser<wbr>name></div>
<div>
<jmxPassword>password</jmxPass<wbr>word></div>
<div>
</taskManager></div>
</div>
</div>
<div
class="gmail_extra"><br>
<div
class="gmail_quote">On
Mon, Aug 15,
2016 at 11:44
AM, Pavol
Mederly <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:mederly@evolveum.com"
target="_blank">mederly@evolveum.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div
bgcolor="#FFFFFF"
text="#000000">
<p>Hello
Florin,</p>
<p>this is
really
interesting.
Please, could
you also share
your midPoint
config.xml, as
well as your
data source
configuration?
(except
credentials,
of course)</p>
<span>
<p>Best
regards,<br>
</p>
<pre cols="72">Pavol Mederly
Software developer
<a moz-do-not-send="true" href="http://evolveum.com" target="_blank">evolveum.com</a>
</pre>
</span>
<div>
<div>
<div>On
15.08.2016
19:39, Florin.
Stingaciu
wrote:<br>
</div>
<blockquote
type="cite">
<div dir="ltr">Hello
Pavol,
<div><br>
</div>
<div>Thanks
for your
detailed
response. I
tried setting
up the
datasource
with
validationQuery
set up
properly for
the mySQL
backed I have.
However, upon
service
restart I get
the following
errors: <a
moz-do-not-send="true"
href="http://pastebin.com/8dpGN0JC" target="_blank">http://pastebin.com/8d<wbr>pGN0JC</a><br>
<br>
To save you a
click, it
seems as
though the
connection is
set up in
readonly mode
or some other
strange things
happen. I've
tried setting <span>readonly</span><span>=</span><span>"false"</span><span> in
the resource
definition in
server.xml but
that didn't
help. I will
continue
researching
this, however
any guidance
would be quite
appreciated. </span></div>
<div><span><br>
</span></div>
<div><span>Thanks!</span></div>
<div><span>-F
</span></div>
</div>
<div
class="gmail_extra"><br>
<div
class="gmail_quote">On
Mon, Aug 15,
2016 at 2:29
AM, Pavol
Mederly <span
dir="ltr"><<a
moz-do-not-send="true" href="mailto:mederly@evolveum.com"
target="_blank">mederly@evolveum.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div
bgcolor="#FFFFFF"
text="#000000">
<p>Hello
Florin,</p>
<p>having
looked at your
logs, it seems
that maybe
explicit
setting of
validationQuery
in Quartz data
source setup
would help.
(See <a
moz-do-not-send="true"
href="http://www.quartz-scheduler.org/documentation/quartz-2.x/configuration/ConfigDataSources.html"
target="_blank">http://www.quartz-scheduler.or<wbr>g/documentation/quartz-2.x/con<wbr>figuration/ConfigDataSources.h<wbr>tml</a>.)</p>
<p>Unfortunately,
current
midPoint
implementation
does not allow
to configure
Quartz data
source
parameters.
So, there are
the following
three
possibilities:</p>
<ol>
<li>Take an
alternative
route, and use
application
server-defined
data source
(with
validationQuery
set up).<br>
</li>
<li>Patch Task
Manager
implementation
by adding
appropriate
lines to
Quartz
configuration
(see
LocalNodeManager.java:87-90).</li>
<li>Wait until
we implement
it - I've
created an
issue <a
moz-do-not-send="true"
href="https://jira.evolveum.com/browse/MID-3347" target="_blank">MID-3347</a>
for this.<br>
</li>
</ol>
<p>As for the
first one
(externally
defined data
source):
Please see <a
moz-do-not-send="true"
href="https://wiki.evolveum.com/display/midPoint/Repository+Configuration#RepositoryConfiguration-Datasourceconfiguration"
target="_blank">https://wiki.evolveum.com/disp<wbr>lay/midPoint/Repository+Config<wbr>uration#RepositoryConfiguratio<wbr>n-Datasourceconfiguration</a>
on how to
configure
midPoint
repository
with the data
source. This
data source
will be used
also by
Quartz, if not
overriden in
<taskManager>
section. It
should work
but I don't
remember if
someone
actually
tested this.</p>
<p>Concerning
autoReconnect:
I have no
experiences
with this
setting. I
agree with you
that about the
hesitation of
using it in
production
environment.
If really
needed, I'd
recommend to
separate
midPoint
repository
configuration
from Quartz
configuration
by using
different JDBC
URLs for the
two: the
standard one
for the
repository and
the one with
"autoReconnect=true"
for Quartz. In
this way,
potential
negative
effects should
be restricted
to task
management
functionality
only. But,
overall, I'd
suggest trying
to eliminate
the problem by
setting
validationQuery
first.<br>
</p>
<p>Best
regards,<br>
</p>
<pre cols="72">Pavol Mederly
Software developer
<a moz-do-not-send="true" href="http://evolveum.com" target="_blank">evolveum.com</a>
</pre>
<div>
<div>
<div>On
14.08.2016
22:40, Florin.
Stingaciu
wrote:<br>
</div>
</div>
</div>
<blockquote
type="cite">
<div>
<div>
<div dir="ltr">Hello,
<div><br>
</div>
<div>I'm
trying to
configure an
active/active
configuration,
and
experiencing
some issues
with the
Quartz
scheduler. The
SQL connection
seems to
timeout quite
often and
result in many
warning
messages. I'm
also
experiencing
some errors --
as the timeout
closes the
connection,
some processes
are still
trying to
commit using
that stale
handler. </div>
<div><br>
</div>
<div>Here are
some relevant
logs from both
midPoint
instances in
this cluster: <a
moz-do-not-send="true" href="http://pastebin.com/JFjqrLnT"
target="_blank">http://pastebin.com/J<wbr>FjqrLnT</a> <a
moz-do-not-send="true" href="http://pastebin.com/1vy4iYPY"
target="_blank">http://pastebin.com/1v<wbr>y4iYPY</a><br>
<br>
Also here's
the relevant
portion of my
config.xml
file: <a
moz-do-not-send="true"
href="http://pastebin.com/WbydcCC8" target="_blank">http://pastebin.com/Wbyd<wbr>cCC8</a></div>
<div><br>
Following the
suggestions in
the warnings,
I've started
to look at
autoReconnect
propriety of
the JDBC
connector and
applied it to
my configs
like so:</div>
<div><br>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span><jdbcUrl>jdbc:mysql://SERVER:3<wbr>306/midpoint_dev?autoReconnect<wbr>=true</jdbcUrl></span></blockquote>
<div><br>
</div>
<div>Since
implementing
this change,
the errors and
warnings seem
to have
disappeared. I
will continue
to monitor the
logs and
ensure this
actually the
case. </div>
<div><br>
</div>
<div>Reading
the mysql
docs, I found
that this is
not
recommended as
this may cause
data
inconsistency
issues and
that stale
connection
exceptions
should be
properly
caught within
the
application.
Namely:</div>
<div><br>
</div>
<blockquote
class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">The
use of this
feature is not
recommended,
because it has
side effects
related to
session state
and data
consistency
when
applications
don't handle
SQLExceptions
properly, and
is only
designed to be
used when you
are unable to
configure your
application to
handle
SQLExceptions
resulting from
dead and stale
connections
properly.</blockquote>
<div><br>
Do you have
any
recommended
configuration
for this
scenario? I'd
like to move
my current
production
environment in
an active
active
configuration,
however as of
right now I'm
hesitant to do
so in order to
avoid any data
corruption.
Especially
since it's
quite
difficult to
test for data
consistency
issues that
may arise from
using
autoReconnect
in my dev
environment. <br>
<br>
Thanks, </div>
<div>-F </div>
<div> </div>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
</div>
</div>
<pre>______________________________<wbr>_________________
midPoint mailing list
<a moz-do-not-send="true" href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>
<a moz-do-not-send="true" href="http://lists.evolveum.com/mailman/listinfo/midpoint" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a>
</pre>
</blockquote>
</div>
______________________________<wbr>_________________
midPoint mailing list
<a moz-do-not-send="true" href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>
<a moz-do-not-send="true" href="http://lists.evolveum.com/mailman/listinfo/midpoint" rel="noreferrer" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a>
</blockquote></div>
</div>
<fieldset></fieldset>
<pre>______________________________<wbr>_________________
midPoint mailing list
<a moz-do-not-send="true" href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>
<a moz-do-not-send="true" href="http://lists.evolveum.com/mailman/listinfo/midpoint" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a>
</pre>
</blockquote>
</div></div></div>
______________________________<wbr>_________________
midPoint mailing list
<a moz-do-not-send="true" href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>
<a moz-do-not-send="true" href="http://lists.evolveum.com/mailman/listinfo/midpoint" rel="noreferrer" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a>
</blockquote></div>
</div>
<fieldset></fieldset>
<pre>______________________________<wbr>_________________
midPoint mailing list
<a moz-do-not-send="true" href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>
<a moz-do-not-send="true" href="http://lists.evolveum.com/mailman/listinfo/midpoint" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a>
</pre>
</blockquote>
</div></div></div>
______________________________<wbr>_________________
midPoint mailing list
<a moz-do-not-send="true" href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>
<a moz-do-not-send="true" href="http://lists.evolveum.com/mailman/listinfo/midpoint" rel="noreferrer" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a>
</blockquote></div>
</div>
<fieldset></fieldset>
<pre>______________________________<wbr>_________________
midPoint mailing list
<a moz-do-not-send="true" href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>
<a moz-do-not-send="true" href="http://lists.evolveum.com/mailman/listinfo/midpoint" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a>
</pre>
</blockquote>
</div></div></div>
______________________________<wbr>_________________
midPoint mailing list
<a moz-do-not-send="true" href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>
<a moz-do-not-send="true" href="http://lists.evolveum.com/mailman/listinfo/midpoint" rel="noreferrer" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a>
</blockquote></div>
</div>
</div></div></blockquote></div>
</div>
</div></div></blockquote></div>
</div>
</div></div></blockquote></div>
</div>
<fieldset></fieldset>
<pre>______________________________<wbr>_________________
midPoint mailing list
<a moz-do-not-send="true" href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>
<a moz-do-not-send="true" href="http://lists.evolveum.com/mailman/listinfo/midpoint" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a>
</pre>
</blockquote>
</div></div></div>
______________________________<wbr>_________________
midPoint mailing list
<a moz-do-not-send="true" href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>
<a moz-do-not-send="true" href="http://lists.evolveum.com/mailman/listinfo/midpoint" rel="noreferrer" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a>
</blockquote></div>
</div>
</div></div></blockquote></div>
</div>
<fieldset></fieldset>
<pre>______________________________<wbr>_________________
midPoint mailing list
<a moz-do-not-send="true" href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>
<a moz-do-not-send="true" href="http://lists.evolveum.com/mailman/listinfo/midpoint" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a>
</pre>
</blockquote>
</div></div></div>
______________________________<wbr>_________________
midPoint mailing list
<a moz-do-not-send="true" href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>
<a moz-do-not-send="true" href="http://lists.evolveum.com/mailman/listinfo/midpoint" rel="noreferrer" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a>
</blockquote></div>
</div>
</div></div></blockquote></div>
</div>
<fieldset></fieldset>
<pre>______________________________<wbr>_________________
midPoint mailing list
<a moz-do-not-send="true" href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>
<a moz-do-not-send="true" href="http://lists.evolveum.com/mailman/listinfo/midpoint" target="_blank">http://lists.evolveum.com/<wbr>mailman/listinfo/midpoint</a>
</pre>
</blockquote>
</div></div></div>
______________________________<wbr>_________________
midPoint mailing list
<a moz-do-not-send="true" href="mailto:midPoint@lists.evolveum.com">midPoint@lists.evolveum.com</a>
<a moz-do-not-send="true" href="http://lists.evolveum.com/mailman/listinfo/midpoint" rel="noreferrer" target="_blank">http://lists.evolveum.com/<wbr>mailman/listinfo/midpoint</a>
</blockquote></div>
</div>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre wrap="">_______________________________________________
midPoint mailing list
<a class="moz-txt-link-abbreviated" href="mailto:midPoint@lists.evolveum.com">midPoint@lists.evolveum.com</a>
<a class="moz-txt-link-freetext" href="http://lists.evolveum.com/mailman/listinfo/midpoint">http://lists.evolveum.com/mailman/listinfo/midpoint</a>
</pre>
</blockquote>
</body></html>