<div dir="ltr">Hey Pavol, <div><br></div><div>Here's the relevant lines:</div><div><br></div><div><div>2016-08-30 18:18:44,443 [] [localhost-startStop-1] INFO (com.evolveum.midpoint.task.quartzimpl.Initializer): Task Manager initialization.</div><div>2016-08-30 18:18:44,452 [] [localhost-startStop-1] INFO (com.evolveum.midpoint.task.quartzimpl.Initializer): Task Manager: Quartz Job Store: JDBC, clustered. Threads: 10</div><div>2016-08-30 18:18:44,452 [] [localhost-startStop-1] INFO (com.evolveum.midpoint.task.quartzimpl.TaskManagerConfiguration): Quartz database is at java:comp/env/jdbc/mysql (a data source)</div><div>2016-08-30 18:18:44,461 [] [localhost-startStop-1] INFO (com.evolveum.midpoint.task.quartzimpl.cluster.NodeRegistrar): Registering this node in the repository as 001 at <a href="http://100.73.170.188:20001">100.73.170.188:20001</a></div><div>2016-08-30 18:18:44,945 [] [localhost-startStop-1] INFO (com.evolveum.midpoint.task.quartzimpl.execution.TaskSynchronizer): Synchronizing Quartz job store with midPoint repository.</div><div>2016-08-30 18:18:46,036 [] [localhost-startStop-1] INFO (com.evolveum.midpoint.task.quartzimpl.execution.TaskSynchronizer): Synchronization of midpoint and Quartz task store finished. 26 task(s) existing in midPoint repository successfully processed, resulting in 0 updated Quartz job(s). 0 task(s) removed from Quartz job store. Processing of 0 task(s) failed.</div><div>2016-08-30 18:18:46,037 [] [localhost-startStop-1] INFO (com.evolveum.midpoint.task.quartzimpl.Initializer): Task Manager initialized</div></div><div><br></div><div>Thanks, </div><div>-F </div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Aug 30, 2016 at 11:17 AM, Pavol Mederly <span dir="ltr"><<a href="mailto:mederly@evolveum.com" target="_blank">mederly@evolveum.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
  
    
  
  <div bgcolor="#FFFFFF" text="#000000">
    <p>Florin,</p>
    <p>this is really interesting. The exception of <br>
    </p>
    <p><tt>Caused by:
        com.mysql.jdbc.exceptions.<wbr>jdbc4.CommunicationsException: The
        last packet successfully received from the server was 64,841,997
        milliseconds ago.  The last packet sent successfully to the
        server was 64,841,998 milliseconds ago. is longer than the
        server configured value of 'wait_timeout'. You should consider
        either expiring and/or testing connection validity before use in
        your application, increasing the server configured values for
        client timeouts, or using the Connector/J connection property
        'autoReconnect=true' to avoid this problem.</tt></p>
    <p>occurs in the context of Quartz scheduler, where the data source
      that checks connection validity is used!</p>
    <p>When you start midPoint there should be messages like these:</p>
    <p><tt>2016-08-30 20:14:31,388 [] [RMI TCP Connection(5)-127.0.0.1]
        INFO (com.evolveum.midpoint.task.<wbr>quartzimpl.Initializer): Task
        Manager: Quartz Job Store: JDBC, NOT clustered. Threads: 10</tt><tt><br>
      </tt><tt>2016-08-30 20:14:31,388 [] [RMI TCP
        Connection(5)-127.0.0.1] INFO
        (com.evolveum.midpoint.task.<wbr>quartzimpl.<wbr>TaskManagerConfiguration):
        Quartz database is at jdbc:postgresql://localhost/<wbr>midpoint (a
        JDBC URL)</tt></p>
    <p>In your case there should be a reference to the data source.
      Could you please verify it is OK, and perhaps paste relevant lines
      here?<br>
    </p><span class="">
    <pre cols="72">Pavol Mederly
Software developer
<a href="http://evolveum.com" target="_blank">evolveum.com</a>
</pre>
    </span><div><div class="h5"><div>On 30.08.2016 19:21, Florin. Stingaciu
      wrote:<br>
    </div>
    <blockquote type="cite">
      <div dir="ltr">Hey Pavol, 
        <div><br>
        </div>
        <div>Over night, I now have the same error messages I initially
          had: <a href="http://pastebin.com/iWeSaswD" target="_blank">http://pastebin.com/<wbr>iWeSaswD</a></div>
        <div><br>
        </div>
        <div>Thank you for all your help so far. Any further
          recommendations would be really appreciated. </div>
        <div><br>
        </div>
        <div>Thanks, </div>
        <div>-F </div>
      </div>
      <div class="gmail_extra"><br>
        <div class="gmail_quote">On Mon, Aug 29, 2016 at 3:54 PM,
          Florin. Stingaciu <span dir="ltr"><<a href="mailto:fstingaciu@mirantis.com" target="_blank">fstingaciu@mirantis.com</a>></span>
          wrote:<br>
          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div dir="ltr">Hey Pavol, 
              <div><br>
              </div>
              <div>I switched the repository to the configuration you
                mentioned, and I could now access the shadow page.
                However (minus the task manager config), this is the
                same configuration as I previously had where I ran into
                errors and then made this thread because of it. For now
                I'm not seeing any errors in the log files. </div>
              <div><br>
              </div>
              <div>I'll update this thread once I run into issues. As of
                right now, I'm still definitely not comfortable moving
                to active active considering all these issues. Have you
                guys seen similar issues before? Or do you believe I've
                just misconfigured something?</div>
              <div><br>
              </div>
              <div>Thanks,</div>
              <div>-F </div>
              <div><br>
              </div>
              <div><br>
              </div>
            </div>
            <div>
              <div>
                <div class="gmail_extra"><br>
                  <div class="gmail_quote">On Mon, Aug 29, 2016 at 3:39
                    PM, Pavol Mederly <span dir="ltr"><<a href="mailto:mederly@evolveum.com" target="_blank">mederly@evolveum.com</a>></span>
                    wrote:<br>
                    <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                      <div bgcolor="#FFFFFF" text="#000000">
                        <p>Hello Florin,</p>
                        <p>I've tried to use your setup in my
                          environment (slightly modified), but I was not
                          able to replicate the exceptions. I've tried
                          in particular to play with the listing of
                          shadow objects.</p>
                        <p>So we perhaps have to continue diagnosing on
                          your side. <br>
                        </p>
                        <p>I suggest to try to isolate the problem. If
                          you could use the following config:</p>
                        <p><span><span> </span></span></p>
                        <div>        <repository></div>
                        <div>               
                          <embedded>false</embedded></div>
                        <div>               
                          <repositoryServiceFactoryClass<wbr>>com.evolveum.midpoint.repo.sq<wbr>l.SqlRepositoryFactory</reposi<wbr>toryServiceFactoryClass></div>
                        <div>               
                          <hibernateHbm2ddl>validate</hi<wbr>bernateHbm2ddl></div>
                        <div>               
                          <database>mysql</database><br>
                        </div>
                        <div>               
                          <jdbcUsername>...</jdbcUsernam<wbr>e><br>
                                         
                          <jdbcPassword>...</jdbcPasswor<wbr>d><br>
                                         
                          <jdbcUrl>jdbc:mysql://servder:<wbr>3306/midpoint_dev</jdbcUrl><br>
                                  </repository></div>
                        <span>
                          <div>        <taskManager></div>
                          <div>               
                            <clustered>true</clustered></div>
                          <div>               
                            <jmxUsername>midpoint</jmxUser<wbr>name></div>
                        </span>
                        <div>               
                          <jmxPassword>password</jmxPass<wbr>word><br>
                                          <dataSource><span>java:comp/env/jdbc<wbr>/mysql</span></dataSource><br>
                        </div>
                        <div>        </taskManager><br>
                                  <workflow><br>
                                         
                          <enabled>false</enabled><br>
                          <span></span>
                          <div>        </workflow><br>
                          </div>
                          <div><br>
                          </div>
                          ...and try the testing. The configuration
                          simply makes midPoint use its own connection
                          mechanism (c3p0) and Quartz use the tomcat's
                          connection pool.<br>
                        </div>
                        <span>
                          <div>Best regards,<br>
                          </div>
                          <pre cols="72">Pavol Mederly
Software developer
<a href="http://evolveum.com" target="_blank">evolveum.com</a>
</pre>
                        </span>
                        <div>
                          <div>
                            <div>On 29.08.2016 20:17, Florin. Stingaciu
                              wrote:<br>
                            </div>
                            <blockquote type="cite">
                              <div dir="ltr">As an update, after having
                                the instance up for half an hour or so
                                with the above config, I got the
                                following errors again: <a href="http://pastebin.com/tJJVFVsu" target="_blank">http://pastebin.com/tJJ<wbr>VFVsu</a>
                                <div><br>
                                </div>
                                <div>Also, in the deployment, there's no
                                  workflows at all and very little user
                                  load as it's just a dev env.</div>
                                <div><br>
                                </div>
                                <div>Thanks,</div>
                                <div>-F </div>
                              </div>
                              <div class="gmail_extra"><br>
                                <div class="gmail_quote">On Mon, Aug 29,
                                  2016 at 10:43 AM, Florin. Stingaciu <span dir="ltr"><<a href="mailto:fstingaciu@mirantis.com" target="_blank">fstingaciu@mirantis.com</a>></span>
                                  wrote:<br>
                                  <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                    <div dir="ltr">Hey Pavol, 
                                      <div><br>
                                      </div>
                                      <div>Currently, for Quartz and
                                        miPoint, I'm using a tomcat7
                                        defined datasource with the
                                        following config:<span><br>
                                          <br>
                                          <div>    <Resource
                                            name="jdbc/mysql"
                                            auth="Container"
                                            type="javax.sql.DataSource"</div>
                                        </span>
                                        <div>        username="midpoint"
                                          password="password"</div>
                                        <div>       
                                          url="jdbc:mysql://servder:3306<wbr>/midpoint_dev"</div>
                                        <span>
                                          <div>       
                                            driverClassName="com.mysql.jdb<wbr>c.Driver"</div>
                                          <div>       
                                            accessToUnderlyingConnectionAl<wbr>lowed="true"</div>
                                          <div>        initialSize="5"
                                            maxWait="5000"</div>
                                          <div>        maxActive="30"
                                            maxIdle="5"</div>
                                          <div>       
                                            validationQuery="select 1"</div>
                                        </span>
                                        <div>        readonly="false"  
                                               </div>
                                        <div>       
                                          poolPreparedStatements="true"/<wbr>><br>
                                          <br>
                                          And here's my relevant
                                          config.xml:<span><br>
                                            <br>
                                            <div>       
                                              <repository></div>
                                            <div>               
                                              <embedded>false</embedded></div>
                                            <div>               
                                              <repositoryServiceFactoryClass<wbr>>com.evolveum.midpoint.repo.sq<wbr>l.SqlRepositoryFactory</reposi<wbr>toryServiceFactoryClass></div>
                                            <div>               
                                              <hibernateHbm2ddl>validate</hi<wbr>bernateHbm2ddl></div>
                                            <div>               
                                              <hibernateDialect>com.evolveum<wbr>.midpoint.repo.sql.util.MidPoi<wbr>ntMySQLDialect</hibernateDiale<wbr>ct></div>
                                            <div>               
                                              <dataSource>java:comp/env/jdbc<wbr>/mysql</dataSource></div>
                                            <div>       
                                              </repository></div>
                                            <div>       
                                              <taskManager></div>
                                            <div>               
                                              <clustered>true</clustered></div>
                                            <div>               
                                              <jmxUsername>midpoint</jmxUser<wbr>name></div>
                                            <div>               
                                              <jmxPassword>password</jmxPass<wbr>word></div>
                                            <div>       
                                              </taskManager></div>
                                          </span></div>
                                        <div><br>
                                        </div>
                                        <div>With this configuration,
                                          when I was trying to access
                                          the Shadows page, I got the
                                          errors mentioned in my
                                          previous message. </div>
                                        <div><br>
                                        </div>
                                        <div>Following your last email,
                                          I added the following
                                          <dataSource>java:comp/env/jdbc<wbr>/mysql</dataSource>
                                          under <taskManager>. Now
                                          when I try to access the
                                          Shadows Page I get the
                                          following error: <a href="http://imgur.com/a/391CU" target="_blank">http://imgur.com/a/391C<wbr>U</a>.
                                          Please note there are four
                                          photos in that album. Also the
                                          reason, I chose an image is
                                          due to the fact that this
                                          error does not appear anywhere
                                          in the logs and it's in the
                                          GUI only. </div>
                                        <div><br>
                                        </div>
                                        <div>I would also like to add
                                          that even though this error
                                          occurs, midPoint still
                                          continues to function. Before
                                          I added the <dataSource>
                                          in the taskManager section,
                                          even though I was using a
                                          datasource for the Repository,
                                          when I tried to access the
                                          shadows page, midPoint would
                                          not work anymore (due to lack
                                          of mysql connections) until I
                                          restarted tomcat. </div>
                                        <div><br>
                                        </div>
                                        <div>The last point I want to
                                          make is that from what I've
                                          seen so far, this issue only
                                          happens when I try to access
                                          the shadow page. I don't see
                                          this issue while trying to
                                          access any other type of
                                          repository objects. </div>
                                        <div><br>
                                        </div>
                                        <div>Thanks, </div>
                                        <span><font color="#888888">
                                            <div>-F</div>
                                            <br>
                                          </font></span></div>
                                    </div>
                                    <div>
                                      <div>
                                        <div class="gmail_extra"><br>
                                          <div class="gmail_quote">On
                                            Fri, Aug 26, 2016 at 3:41
                                            PM, Pavol Mederly <span dir="ltr"><<a href="mailto:mederly@evolveum.com" target="_blank">mederly@evolveum.com</a>></span>
                                            wrote:<br>
                                            <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                              <div bgcolor="#FFFFFF" text="#000000">
                                                <p>Hello Florin,</p>
                                                <p>it seems that either
                                                  midPoint (i.e.
                                                  hibernate), Quartz
                                                  scheduler, or Activiti
                                                  BPMS use connections
                                                  without properly
                                                  closing them. <br>
                                                </p>
                                                <p>First step to
                                                  diagnose it would be
                                                  perhaps to setup
                                                  things like this:</p>
                                                <ol>
                                                  <li>Let midPoint and
                                                    Activiti use their
                                                    own data source (by
                                                    configuring them
                                                    with a JDBC URL):
                                                    see <a href="https://wiki.evolveum.com/display/midPoint/Repository+Configuration" target="_blank">Repository
                                                      configuration</a>
                                                    and <a href="https://wiki.evolveum.com/display/midPoint/Workflow+configuration" target="_blank">Workflow
                                                      configuration</a>
                                                    wiki pages. (If you
                                                    don't use approvals
                                                    you can disable
                                                    workflows altogether
                                                    by setting "enabled"
                                                    to "false".)</li>
                                                  <li>Keep Quartz
                                                    schedule using
                                                    tomcat data source:
                                                    something like</li>
                                                </ol>
                                                <div><code><</code><code>taskManager</code><code>></code></div>
                                                <div><code>  </code><code><</code><code>clustered</code><code>>true</</code><code>clustered</code><code>><br>
                                                     
                                                    <dataSource>...your
                                                    data source
                                                    name...</dataSource><br>
                                                  </code></div>
                                                    ...<br>
                                                <div><code></</code><code>taskManager</code><code>><br>
                                                  </code><br>
                                                  And you'll see if the
                                                  problem would appear
                                                  again. (Maybe you
                                                  could even configure
                                                  three Tomcat data
                                                  sources; one for
                                                  midPoint, second for
                                                  Quartz, and the third
                                                  for Activiti, if
                                                  needed. But I have not
                                                  done this yet; these
                                                  are pure speculations
                                                  of mine...)<br>
                                                  <br>
                                                  Best regards,<br>
                                                </div>
                                                <span>
                                                  <pre cols="72">Pavol Mederly
Software developer
<a href="http://evolveum.com" target="_blank">evolveum.com</a>
</pre>
                                                </span>
                                                <div>
                                                  <div>
                                                    <div>On 24.08.2016
                                                      22:41, Florin.
                                                      Stingaciu wrote:<br>
                                                    </div>
                                                    <blockquote type="cite">
                                                      <div dir="ltr"><font face="Source
                                                          Sans Pro,
                                                          Helvetica
                                                          Neue,
                                                          Helvetica,
                                                          Arial,
                                                          sans-serif" color="#333333"><span style="font-size:14px;line-height:20px">I just rebuilt the latest
                                                          snapshot
                                                          (git-v3.5devel-221-g9744fa0)
                                                          and am
                                                          experiencing
                                                          the same
                                                          problem. </span></font>
                                                        <div><font face="Source
                                                          Sans Pro,
                                                          Helvetica
                                                          Neue,
                                                          Helvetica,
                                                          Arial,
                                                          sans-serif" color="#333333"><span style="font-size:14px;line-height:20px"><br>
                                                          </span></font></div>
                                                        <div><font face="Source
                                                          Sans Pro,
                                                          Helvetica
                                                          Neue,
                                                          Helvetica,
                                                          Arial,
                                                          sans-serif" color="#333333"><span style="font-size:14px;line-height:20px">Thanks,</span></font></div>
                                                        <div><font face="Source
                                                          Sans Pro,
                                                          Helvetica
                                                          Neue,
                                                          Helvetica,
                                                          Arial,
                                                          sans-serif" color="#333333"><span style="font-size:14px;line-height:20px">-F </span></font></div>
                                                        <div>
                                                          <div><br>
                                                          </div>
                                                        </div>
                                                      </div>
                                                      <div class="gmail_extra"><br>
                                                        <div class="gmail_quote">On
                                                          Wed, Aug 24,
                                                          2016 at 1:24
                                                          PM, Florin.
                                                          Stingaciu <span dir="ltr"><<a href="mailto:fstingaciu@mirantis.com" target="_blank">fstingaciu@mirantis.com</a>></span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                                          <div dir="ltr">Upon
                                                          further
                                                          investigation
                                                          I found that
                                                          this error
                                                          only occurs
                                                          when I try to
                                                          access the
                                                          shadow
                                                          repository via
Configuration->Repository Objects->Shadows 
                                                          <div><br>
                                                          </div>
                                                          <div>It seems
                                                          that something
                                                          triggers the
                                                          creation of DB
                                                          connections
                                                          until it runs
                                                          out of
                                                          available
                                                          connections.
                                                          For this
                                                          particular
                                                          instance, I'm
                                                          at <span>git-v3.5devel-170-g26b0644.<wbr> </span></div>
                                                          <div><span><br>
                                                          </span></div>
                                                          <div><span>Thanks, </span></div>
                                                          <span><font color="#888888">
                                                          <div><span>-F </span></div>
                                                          </font></span></div>
                                                          <div>
                                                          <div>
                                                          <div class="gmail_extra"><br>
                                                          <div class="gmail_quote">On
                                                          Wed, Aug 24,
                                                          2016 at 1:13
                                                          PM, Florin.
                                                          Stingaciu <span dir="ltr"><<a href="mailto:fstingaciu@mirantis.com" target="_blank">fstingaciu@mirantis.com</a>></span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                                          <div dir="ltr">Hello
                                                          Pavol, 
                                                          <div><br>
                                                          </div>
                                                          <div>After
                                                          running this
                                                          configuration
                                                          my dev
                                                          environment
                                                          with two
                                                          instances up
                                                          for a couple
                                                          of days, I
                                                          started to get
                                                          the following
                                                          error
                                                          messages: <a href="http://pastebin.com/nkxE9LNA" target="_blank">http://pastebin.com/<wbr>nkxE9LNA</a></div>
                                                          <div><br>
                                                          </div>
                                                          <div>It seems
                                                          as though the
                                                          connection
                                                          pool is not
                                                          properly
                                                          cleaned up.
                                                          The database
                                                          logs indicate
                                                          no failure and
                                                          in fact my
                                                          production
                                                          environment is
                                                          hooked up to
                                                          the same DB
                                                          instance and
                                                          is working
                                                          fine. </div>
                                                          <div><br>
                                                          </div>
                                                          <div>Any
                                                          recommendations
                                                          would be very
                                                          appreciated. </div>
                                                          <div><br>
                                                          </div>
                                                          <div>Thanks, </div>
                                                          <span><font color="#888888">
                                                          <div>-F</div>
                                                          </font></span></div>
                                                          <div>
                                                          <div>
                                                          <div class="gmail_extra"><br>
                                                          <div class="gmail_quote">On
                                                          Mon, Aug 15,
                                                          2016 at 3:29
                                                          PM, Florin.
                                                          Stingaciu <span dir="ltr"><<a href="mailto:fstingaciu@mirantis.com" target="_blank">fstingaciu@mirantis.com</a>></span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                                          <div dir="ltr">Awesome!
                                                          Thanks for all
                                                          your help
                                                          today Pavol. I
                                                          will try and
                                                          test in my
                                                          environment
                                                          and let you
                                                          know how that
                                                          goes. 
                                                          <div><br>
                                                          </div>
                                                          <div>Thanks
                                                          again!</div>
                                                          <span><font color="#888888">
                                                          <div>-F </div>
                                                          </font></span></div>
                                                          <div>
                                                          <div>
                                                          <div class="gmail_extra"><br>
                                                          <div class="gmail_quote">On
                                                          Mon, Aug 15,
                                                          2016 at 3:00
                                                          PM, Pavol
                                                          Mederly <span dir="ltr"><<a href="mailto:mederly@evolveum.com" target="_blank">mederly@evolveum.com</a>></span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                                          <div bgcolor="#FFFFFF" text="#000000">
                                                          <p>Florin,</p>
                                                          <p>I think
                                                          I've fixed it:
                                                          <a href="https://github.com/Evolveum/midpoint/commit/c00f2979cea6f6ada0a1194b13bad51c0ecec2e3" target="_blank">https://github.com/Evolveum/mi<wbr>dpoint/commit/c00f2979cea6f6ad<wbr>a0a1194b13bad51c0ecec2e3</a><br>
                                                          </p>
                                                          <p>Cannot
                                                          verify as the
                                                          issue isn't
                                                          visible on my
                                                          computer, but
                                                          I am quite
                                                          convinced it
                                                          should be
                                                          fixed now. The
                                                          suspected
                                                          cause is that
                                                          some
                                                          query-only
                                                          transactions
                                                          weren't closed
                                                          properly. C3P0
                                                          closes them
                                                          automatically
                                                          but <a href="http://stackoverflow.com/questions/19396563/does-hibernates-session-close-automatically-rollback-uncommitted-transactions" target="_blank">other connection providers probably not</a>.</p>
                                                          <p>You can
                                                          safely use the
                                                          master, or you
                                                          can apply the
                                                          above patch to
                                                          3.4 yourself.
                                                          The master is
                                                          something like
                                                          "3.4-support"
                                                          these days
                                                          (preparing for
                                                          3.4.1), so it
                                                          should be of
                                                          production-level
                                                          quality, or at
                                                          least near
                                                          it.  <br>
                                                          (I'm going to
                                                          run full test
                                                          suite during
                                                          the night, but
                                                          I don't expect
                                                          any failures.
                                                          Or you can
                                                          check after
                                                          approx 3 hours
                                                          <a href="https://travis-ci.org/Evolveum/midpoint" target="_blank">here</a>.)</p>
                                                          <span>
                                                          <p>Best
                                                          regards,<br>
                                                          </p>
                                                          <pre cols="72">Pavol Mederly
Software developer
<a href="http://evolveum.com" target="_blank">evolveum.com</a>
</pre>
                                                          </span>
                                                          <div>
                                                          <div>
                                                          <div>On
                                                          15.08.2016
                                                          22:57, Florin.
                                                          Stingaciu
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote type="cite">
                                                          <div dir="ltr">Hey
                                                          Pavol, 
                                                          <div><br>
                                                          </div>
                                                          <div>I
                                                          uploaded the
                                                          log file you
                                                          requested to
                                                          my gdrive (<a href="https://drive.google.com/file/d/0B9f4-vWBsztoZy1zWThGYnpGeGs/view?usp=sharing" target="_blank">https://drive.google.com/file<wbr>/d/0B9f4-vWBsztoZy1zWThGYnpGeG<wbr>s/view?usp=sharing</a>)
                                                          It's a trace
                                                          level debug so
                                                          the file is
                                                          quite large.
                                                          Let me know if
                                                          anything jumps
                                                          out at you. </div>
                                                          <div><br>
                                                          Thanks, </div>
                                                          <div>-F </div>
                                                          </div>
                                                          <div class="gmail_extra"><br>
                                                          <div class="gmail_quote">On
                                                          Mon, Aug 15,
                                                          2016 at 1:07
                                                          PM, Pavol
                                                          Mederly <span dir="ltr"><<a href="mailto:mederly@evolveum.com" target="_blank">mederly@evolveum.com</a>></span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                                          <div bgcolor="#FFFFFF" text="#000000">
                                                          <p>Strange
                                                          enough. I've
                                                          replicated
                                                          your setup on
                                                          my computer
                                                          (using
                                                          PostgreSQL, as
                                                          I have no
                                                          MySQL
                                                          currently
                                                          installed),
                                                          and it works:
                                                          midPoint
                                                          starts
                                                          cleanly. I am
                                                          able to
                                                          add/delete
                                                          repository
                                                          objects.<br>
                                                          </p>
                                                          <p>Attached
                                                          are my
                                                          config.xml and
                                                          server.xml;
                                                          they are
                                                          pretty much
                                                          the same as
                                                          yours.</p>
                                                          <p>So, I have
                                                          no idea. You
                                                          could try
                                                          adding
                                                          <database>mysql</database>
                                                          into
                                                          <repository>
                                                          element in
                                                          your
                                                          config.xml,
                                                          but it would
                                                          most probably
                                                          make no
                                                          difference.</p>
                                                          <p>As last
                                                          resort, you
                                                          could turn on
                                                          the most
                                                          detailed
                                                          debugging for
                                                          the repository
                                                          on startup by
                                                          adding a line
                                                          like this</p>
                                                          <p><logger
                                                          name="<a href="http://com.evolveum.midpoint.re" target="_blank">com.evolveum.midpoint.re</a><wbr>po"
                                                          level="TRACE"
                                                          /></p>
                                                          <p>into
                                                          webapps\midpoint\WEB-INF\class<wbr>es\logback.xml
                                                          file and start
                                                          midPoint. In
                                                          idm.log there
                                                          should be
                                                          quite a
                                                          detailed
                                                          record of
                                                          what's going
                                                          on.<br>
                                                          </p>
                                                          <span>
                                                          <pre cols="72">Pavol Mederly
Software developer
<a href="http://evolveum.com" target="_blank">evolveum.com</a>
</pre>
                                                          </span>
                                                          <div>
                                                          <div>
                                                          <div>On
                                                          15.08.2016
                                                          21:03, Florin.
                                                          Stingaciu
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote type="cite">
                                                          <div dir="ltr">Here's
                                                          my datasource
                                                          config:<br>
                                                          <br>
                                                          <div>   
                                                          <Resource
                                                          name="jdbc/mysql"
auth="Container" type="javax.sql.DataSource"</div>
                                                          <div>       
                                                          username="midpoint"
password="pass"</div>
                                                          <div>       
                                                          url="jdbc:mysql://SERVER:3306/<wbr>midpoint_dev"</div>
                                                          <div>       
                                                          driverClassName="com.mysql.jdb<wbr>c.Driver"</div>
                                                          <div>       
                                                          accessToUnderlyingConnectionAl<wbr>lowed="true"</div>
                                                          <div>       
                                                          initialSize="5"
                                                          maxWait="5000"</div>
                                                          <div>       
                                                          maxActive="30"
                                                          maxIdle="5"</div>
                                                          <div>       
                                                          validationQuery="select
                                                          1"</div>
                                                          <div>       
                                                          poolPreparedStatements="true"/<wbr>></div>
                                                          <div><br>
                                                          </div>
                                                          <div>And
                                                          here's my
                                                          config.xml:<br>
                                                          <br>
                                                          <div>       
                                                          <repository></div>
                                                          <div>         
                                                               
                                                          <embedded>false</embedded></div>
                                                          <div>         
                                                               
                                                          <repositoryServiceFactoryClass<wbr>>com.evolveum.midpoint.repo.sq<wbr>l.SqlRepositoryFactory</reposi<wbr>toryServiceFactoryClass></div>
                                                          <div>         
                                                               
                                                          <hibernateHbm2ddl>validate</hi<wbr>bernateHbm2ddl></div>
                                                          <div>         
                                                               
                                                          <hibernateDialect>com.evolveum<wbr>.midpoint.repo.sql.util.MidPoi<wbr>ntMySQLDialect</hibernateDiale<wbr>ct></div>
                                                          <div>         
                                                               
                                                          <dataSource>java:comp/env/jdbc<wbr>/mysql</dataSource></div>
                                                          <div>       
                                                          </repository></div>
                                                          <div>       
                                                          <taskManager></div>
                                                          <div>         
                                                               
                                                          <clustered>true</clustered></div>
                                                          <div>         
                                                               
                                                          <jmxUsername>midpoint</jmxUser<wbr>name></div>
                                                          <div>         
                                                               
                                                          <jmxPassword>password</jmxPass<wbr>word></div>
                                                          <div>       
                                                          </taskManager></div>
                                                          </div>
                                                          </div>
                                                          <div class="gmail_extra"><br>
                                                          <div class="gmail_quote">On
                                                          Mon, Aug 15,
                                                          2016 at 11:44
                                                          AM, Pavol
                                                          Mederly <span dir="ltr"><<a href="mailto:mederly@evolveum.com" target="_blank">mederly@evolveum.com</a>></span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                                          <div bgcolor="#FFFFFF" text="#000000">
                                                          <p>Hello
                                                          Florin,</p>
                                                          <p>this is
                                                          really
                                                          interesting.
                                                          Please, could
                                                          you also share
                                                          your midPoint
                                                          config.xml, as
                                                          well as your
                                                          data source
                                                          configuration?
                                                          (except
                                                          credentials,
                                                          of course)</p>
                                                          <span>
                                                          <p>Best
                                                          regards,<br>
                                                          </p>
                                                          <pre cols="72">Pavol Mederly
Software developer
<a href="http://evolveum.com" target="_blank">evolveum.com</a>
</pre>
                                                          </span>
                                                          <div>
                                                          <div>
                                                          <div>On
                                                          15.08.2016
                                                          19:39, Florin.
                                                          Stingaciu
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote type="cite">
                                                          <div dir="ltr">Hello
                                                          Pavol, 
                                                          <div><br>
                                                          </div>
                                                          <div>Thanks
                                                          for your
                                                          detailed
                                                          response. I
                                                          tried setting
                                                          up the
                                                          datasource
                                                          with
                                                          validationQuery
                                                          set up
                                                          properly for
                                                          the mySQL
                                                          backed I have.
                                                          However, upon
                                                          service
                                                          restart I get
                                                          the following
                                                          errors: <a href="http://pastebin.com/8dpGN0JC" target="_blank">http://pastebin.com/8d<wbr>pGN0JC</a><br>
                                                          <br>
                                                          To save you a
                                                          click, it
                                                          seems as
                                                          though the
                                                          connection is
                                                          set up in
                                                          readonly mode
                                                          or some other
                                                          strange things
                                                          happen. I've
                                                          tried setting <span>readonly</span><span>=</span><span>"false"</span><span> in
                                                          the resource
                                                          definition in
                                                          server.xml but
                                                          that didn't
                                                          help. I will
                                                          continue
                                                          researching
                                                          this, however
                                                          any guidance
                                                          would be quite
                                                          appreciated. </span></div>
                                                          <div><span><br>
                                                          </span></div>
                                                          <div><span>Thanks!</span></div>
                                                          <div><span>-F
                                                                </span></div>
                                                          </div>
                                                          <div class="gmail_extra"><br>
                                                          <div class="gmail_quote">On
                                                          Mon, Aug 15,
                                                          2016 at 2:29
                                                          AM, Pavol
                                                          Mederly <span dir="ltr"><<a href="mailto:mederly@evolveum.com" target="_blank">mederly@evolveum.com</a>></span>
                                                          wrote:<br>
                                                          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                                          <div bgcolor="#FFFFFF" text="#000000">
                                                          <p>Hello
                                                          Florin,</p>
                                                          <p>having
                                                          looked at your
                                                          logs, it seems
                                                          that maybe
                                                          explicit
                                                          setting of
                                                          validationQuery
                                                          in Quartz data
                                                          source setup
                                                          would help.
                                                          (See <a href="http://www.quartz-scheduler.org/documentation/quartz-2.x/configuration/ConfigDataSources.html" target="_blank">http://www.quartz-scheduler.or<wbr>g/documentation/quartz-2.x/con<wbr>figuration/ConfigDataSources.h<wbr>tml</a>.)</p>
                                                          <p>Unfortunately,
                                                          current
                                                          midPoint
                                                          implementation
                                                          does not allow
                                                          to configure
                                                          Quartz data
                                                          source
                                                          parameters.
                                                          So, there are
                                                          the following
                                                          three
                                                          possibilities:</p>
                                                          <ol>
                                                          <li>Take an
                                                          alternative
                                                          route, and use
                                                          application
                                                          server-defined
                                                          data source
                                                          (with
                                                          validationQuery
                                                          set up).<br>
                                                          </li>
                                                          <li>Patch Task
                                                          Manager
                                                          implementation
                                                          by adding
                                                          appropriate
                                                          lines to
                                                          Quartz
                                                          configuration
                                                          (see
LocalNodeManager.java:87-90).</li>
                                                          <li>Wait until
                                                          we implement
                                                          it - I've
                                                          created an
                                                          issue <a href="https://jira.evolveum.com/browse/MID-3347" target="_blank">MID-3347</a>
                                                          for this.<br>
                                                          </li>
                                                          </ol>
                                                          <p>As for the
                                                          first one
                                                          (externally
                                                          defined data
                                                          source):
                                                          Please see <a href="https://wiki.evolveum.com/display/midPoint/Repository+Configuration#RepositoryConfiguration-Datasourceconfiguration" target="_blank">https://wiki.evolveum.com/disp<wbr>lay/midPoint/Repository+Config<wbr>uration#RepositoryConfiguratio<wbr>n-Datasourceconfiguration</a>
                                                          on how to
                                                          configure
                                                          midPoint
                                                          repository
                                                          with the data
                                                          source. This
                                                          data source
                                                          will be used
                                                          also by
                                                          Quartz, if not
                                                          overriden in
                                                          <taskManager>
                                                          section. It
                                                          should work
                                                          but I don't
                                                          remember if
                                                          someone
                                                          actually
                                                          tested this.</p>
                                                          <p>Concerning
                                                          autoReconnect:
                                                          I have no
                                                          experiences
                                                          with this
                                                          setting. I
                                                          agree with you
                                                          that about the
                                                          hesitation of
                                                          using it in
                                                          production
                                                          environment.
                                                          If really
                                                          needed, I'd
                                                          recommend to
                                                          separate
                                                          midPoint
                                                          repository
                                                          configuration
                                                          from Quartz
                                                          configuration
                                                          by using
                                                          different JDBC
                                                          URLs for the
                                                          two: the
                                                          standard one
                                                          for the
                                                          repository and
                                                          the one with
                                                          "autoReconnect=true"
                                                          for Quartz. In
                                                          this way,
                                                          potential
                                                          negative
                                                          effects should
                                                          be restricted
                                                          to task
                                                          management
                                                          functionality
                                                          only. But,
                                                          overall, I'd
                                                          suggest trying
                                                          to eliminate
                                                          the problem by
                                                          setting
                                                          validationQuery
                                                          first.<br>
                                                          </p>
                                                          <p>Best
                                                          regards,<br>
                                                          </p>
                                                          <pre cols="72">Pavol Mederly
Software developer
<a href="http://evolveum.com" target="_blank">evolveum.com</a>
</pre>
                                                          <div>
                                                          <div>
                                                          <div>On
                                                          14.08.2016
                                                          22:40, Florin.
                                                          Stingaciu
                                                          wrote:<br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          <blockquote type="cite">
                                                          <div>
                                                          <div>
                                                          <div dir="ltr">Hello, 
                                                          <div><br>
                                                          </div>
                                                          <div>I'm
                                                          trying to
                                                          configure an
                                                          active/active
                                                          configuration,
                                                          and
                                                          experiencing
                                                          some issues
                                                          with the
                                                          Quartz
                                                          scheduler. The
                                                          SQL connection
                                                          seems to
                                                          timeout quite
                                                          often and
                                                          result in many
                                                          warning
                                                          messages. I'm
                                                          also
                                                          experiencing
                                                          some errors --
                                                          as the timeout
                                                          closes the
                                                          connection,
                                                          some processes
                                                          are still
                                                          trying to
                                                          commit using
                                                          that stale
                                                          handler. </div>
                                                          <div><br>
                                                          </div>
                                                          <div>Here are
                                                          some relevant
                                                          logs from both
                                                          midPoint
                                                          instances in
                                                          this cluster: <a href="http://pastebin.com/JFjqrLnT" target="_blank">http://pastebin.com/J<wbr>FjqrLnT</a> <a href="http://pastebin.com/1vy4iYPY" target="_blank">http://pastebin.com/1v<wbr>y4iYPY</a><br>
                                                          <br>
                                                          Also here's
                                                          the relevant
                                                          portion of my
                                                          config.xml
                                                          file: <a href="http://pastebin.com/WbydcCC8" target="_blank">http://pastebin.com/Wbyd<wbr>cCC8</a></div>
                                                          <div><br>
                                                          Following the
                                                          suggestions in
                                                          the warnings,
                                                          I've started
                                                          to look at
                                                          autoReconnect
                                                          propriety of
                                                          the JDBC
                                                          connector and
                                                          applied it to
                                                          my configs
                                                          like so:</div>
                                                          <div><br>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span><jdbcUrl>jdbc:mysql://SERVER:3<wbr>306/midpoint_dev?autoReconnect<wbr>=true</jdbcUrl></span></blockquote>
                                                          <div><br>
                                                          </div>
                                                          <div>Since
                                                          implementing
                                                          this change,
                                                          the errors and
                                                          warnings seem
                                                          to have
                                                          disappeared. I
                                                          will continue
                                                          to monitor the
                                                          logs and
                                                          ensure this
                                                          actually the
                                                          case.  </div>
                                                          <div><br>
                                                          </div>
                                                          <div>Reading
                                                          the mysql
                                                          docs, I found
                                                          that this is
                                                          not
                                                          recommended as
                                                          this may cause
                                                          data
                                                          inconsistency
                                                          issues and
                                                          that stale
                                                          connection
                                                          exceptions
                                                          should be
                                                          properly
                                                          caught within
                                                          the
                                                          application.
                                                          Namely:</div>
                                                          <div><br>
                                                          </div>
                                                          <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">The
                                                          use of this
                                                          feature is not
                                                          recommended,
                                                          because it has
                                                          side effects
                                                          related to
                                                          session state
                                                          and data
                                                          consistency
                                                          when
                                                          applications
                                                          don't handle
                                                          SQLExceptions
                                                          properly, and
                                                          is only
                                                          designed to be
                                                          used when you
                                                          are unable to
                                                          configure your
                                                          application to
                                                          handle
                                                          SQLExceptions
                                                          resulting from
                                                          dead and stale
                                                          connections
                                                          properly.</blockquote>
                                                          <div><br>
                                                          Do you have
                                                          any
                                                          recommended
                                                          configuration
                                                          for this
                                                          scenario? I'd
                                                          like to move
                                                          my current
                                                          production
                                                          environment in
                                                          an active
                                                          active
                                                          configuration,
                                                          however as of
                                                          right now I'm
                                                          hesitant to do
                                                          so in order to
                                                          avoid any data
                                                          corruption.
                                                          Especially
                                                          since it's
                                                          quite
                                                          difficult to
                                                          test for data
                                                          consistency
                                                          issues that
                                                          may arise from
                                                          using
                                                          autoReconnect
                                                          in my dev
                                                          environment. <br>
                                                          <br>
                                                          Thanks, </div>
                                                          <div>-F </div>
                                                          <div> </div>
                                                          </div>
                                                          </div>
                                                          <br>
                                                          <fieldset></fieldset>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          <pre>______________________________<wbr>_________________
midPoint mailing list
<a href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>
<a href="http://lists.evolveum.com/mailman/listinfo/midpoint" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a>
</pre>
    </blockquote>
    

  </div>


______________________________<wbr>_________________

midPoint mailing list

<a href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>

<a href="http://lists.evolveum.com/mailman/listinfo/midpoint" rel="noreferrer" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a>


</blockquote></div>
</div>


<fieldset></fieldset>
<pre>______________________________<wbr>_________________
midPoint mailing list
<a href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>
<a href="http://lists.evolveum.com/mailman/listinfo/midpoint" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a>
</pre>

</blockquote>
</div></div></div>
______________________________<wbr>_________________

midPoint mailing list

<a href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>

<a href="http://lists.evolveum.com/mailman/listinfo/midpoint" rel="noreferrer" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a>


</blockquote></div>
</div>


<fieldset></fieldset>
<pre>______________________________<wbr>_________________
midPoint mailing list
<a href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>
<a href="http://lists.evolveum.com/mailman/listinfo/midpoint" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a>
</pre>

</blockquote>
</div></div></div>
______________________________<wbr>_________________

midPoint mailing list

<a href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>

<a href="http://lists.evolveum.com/mailman/listinfo/midpoint" rel="noreferrer" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a>


</blockquote></div>
</div>


<fieldset></fieldset>
<pre>______________________________<wbr>_________________
midPoint mailing list
<a href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>
<a href="http://lists.evolveum.com/mailman/listinfo/midpoint" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a>
</pre>

</blockquote>
</div></div></div>
______________________________<wbr>_________________

midPoint mailing list

<a href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>

<a href="http://lists.evolveum.com/mailman/listinfo/midpoint" rel="noreferrer" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a>


</blockquote></div>
</div>
</div></div></blockquote></div>
</div>
</div></div></blockquote></div>
</div>
</div></div></blockquote></div>
</div>


<fieldset></fieldset>
<pre>______________________________<wbr>_________________
midPoint mailing list
<a href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>
<a href="http://lists.evolveum.com/mailman/listinfo/midpoint" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a>
</pre>

</blockquote>
</div></div></div>
______________________________<wbr>_________________

midPoint mailing list

<a href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>

<a href="http://lists.evolveum.com/mailman/listinfo/midpoint" rel="noreferrer" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a>


</blockquote></div>
</div>
</div></div></blockquote></div>
</div>


<fieldset></fieldset>
<pre>______________________________<wbr>_________________
midPoint mailing list
<a href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>
<a href="http://lists.evolveum.com/mailman/listinfo/midpoint" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a>
</pre>

</blockquote>
</div></div></div>
______________________________<wbr>_________________

midPoint mailing list

<a href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>

<a href="http://lists.evolveum.com/mailman/listinfo/midpoint" rel="noreferrer" target="_blank">http://lists.evolveum.com/mail<wbr>man/listinfo/midpoint</a>


</blockquote></div>
</div>
</div></div></blockquote></div>
</div>


<fieldset></fieldset>
<pre>______________________________<wbr>_________________
midPoint mailing list
<a href="mailto:midPoint@lists.evolveum.com" target="_blank">midPoint@lists.evolveum.com</a>
<a href="http://lists.evolveum.com/mailman/listinfo/midpoint" target="_blank">http://lists.evolveum.com/<wbr>mailman/listinfo/midpoint</a>
</pre>

</blockquote>
</div></div></div><br>______________________________<wbr>_________________<br>
midPoint mailing list<br>
<a href="mailto:midPoint@lists.evolveum.com">midPoint@lists.evolveum.com</a><br>
<a href="http://lists.evolveum.com/mailman/listinfo/midpoint" rel="noreferrer" target="_blank">http://lists.evolveum.com/<wbr>mailman/listinfo/midpoint</a><br>
<br></blockquote></div><br></div>