[midPoint] [midpoint] Active/Active Configuration

Pavol Mederly mederly at evolveum.com
Sat Aug 27 00:41:00 CEST 2016


Hello Florin,

it seems that either midPoint (i.e. hibernate), Quartz scheduler, or 
Activiti BPMS use connections without properly closing them.

First step to diagnose it would be perhaps to setup things like this:

 1. Let midPoint and Activiti use their own data source (by configuring
    them with a JDBC URL): see Repository configuration
    <https://wiki.evolveum.com/display/midPoint/Repository+Configuration>
    and Workflow configuration
    <https://wiki.evolveum.com/display/midPoint/Workflow+configuration>
    wiki pages. (If you don't use approvals you can disable workflows
    altogether by setting "enabled" to "false".)
 2. Keep Quartz schedule using tomcat data source: something like

|<||taskManager||>|
|||<||clustered||>true</||clustered||>
   <dataSource>...your data source name...</dataSource>
|
     ...
|</||taskManager||>
|
And you'll see if the problem would appear again. (Maybe you could even 
configure three Tomcat data sources; one for midPoint, second for 
Quartz, and the third for Activiti, if needed. But I have not done this 
yet; these are pure speculations of mine...)

Best regards,

Pavol Mederly
Software developer
evolveum.com

On 24.08.2016 22:41, Florin. Stingaciu wrote:
> I just rebuilt the latest snapshot (git-v3.5devel-221-g9744fa0) and am 
> experiencing the same problem.
>
> Thanks,
> -F
>
>
> On Wed, Aug 24, 2016 at 1:24 PM, Florin. Stingaciu 
> <fstingaciu at mirantis.com <mailto:fstingaciu at mirantis.com>> wrote:
>
>     Upon further investigation I found that this error only occurs
>     when I try to access the shadow repository via
>     Configuration->Repository Objects->Shadows
>
>     It seems that something triggers the creation of DB connections
>     until it runs out of available connections. For this particular
>     instance, I'm at git-v3.5devel-170-g26b0644.
>
>     Thanks,
>     -F
>
>     On Wed, Aug 24, 2016 at 1:13 PM, Florin. Stingaciu
>     <fstingaciu at mirantis.com <mailto:fstingaciu at mirantis.com>> wrote:
>
>         Hello Pavol,
>
>         After running this configuration my dev environment with two
>         instances up for a couple of days, I started to get the
>         following error messages: http://pastebin.com/nkxE9LNA
>         <http://pastebin.com/nkxE9LNA>
>
>         It seems as though the connection pool is not properly cleaned
>         up. The database logs indicate no failure and in fact my
>         production environment is hooked up to the same DB instance
>         and is working fine.
>
>         Any recommendations would be very appreciated.
>
>         Thanks,
>         -F
>
>         On Mon, Aug 15, 2016 at 3:29 PM, Florin. Stingaciu
>         <fstingaciu at mirantis.com <mailto:fstingaciu at mirantis.com>> wrote:
>
>             Awesome! Thanks for all your help today Pavol. I will try
>             and test in my environment and let you know how that goes.
>
>             Thanks again!
>             -F
>
>             On Mon, Aug 15, 2016 at 3:00 PM, Pavol Mederly
>             <mederly at evolveum.com <mailto:mederly at evolveum.com>> wrote:
>
>                 Florin,
>
>                 I think I've fixed it:
>                 https://github.com/Evolveum/midpoint/commit/c00f2979cea6f6ada0a1194b13bad51c0ecec2e3
>                 <https://github.com/Evolveum/midpoint/commit/c00f2979cea6f6ada0a1194b13bad51c0ecec2e3>
>
>                 Cannot verify as the issue isn't visible on my
>                 computer, but I am quite convinced it should be fixed
>                 now. The suspected cause is that some query-only
>                 transactions weren't closed properly. C3P0 closes them
>                 automatically but other connection providers probably
>                 not
>                 <http://stackoverflow.com/questions/19396563/does-hibernates-session-close-automatically-rollback-uncommitted-transactions>.
>
>                 You can safely use the master, or you can apply the
>                 above patch to 3.4 yourself. The master is something
>                 like "3.4-support" these days (preparing for 3.4.1),
>                 so it should be of production-level quality, or at
>                 least near it.
>                 (I'm going to run full test suite during the night,
>                 but I don't expect any failures. Or you can check
>                 after approx 3 hours here
>                 <https://travis-ci.org/Evolveum/midpoint>.)
>
>                 Best regards,
>
>                 Pavol Mederly
>                 Software developer
>                 evolveum.com <http://evolveum.com>
>
>                 On 15.08.2016 22:57, Florin. Stingaciu wrote:
>>                 Hey Pavol,
>>
>>                 I uploaded the log file you requested to my gdrive
>>                 (https://drive.google.com/file/d/0B9f4-vWBsztoZy1zWThGYnpGeGs/view?usp=sharing
>>                 <https://drive.google.com/file/d/0B9f4-vWBsztoZy1zWThGYnpGeGs/view?usp=sharing>)
>>                 It's a trace level debug so the file is quite large.
>>                 Let me know if anything jumps out at you.
>>
>>                 Thanks,
>>                 -F
>>
>>                 On Mon, Aug 15, 2016 at 1:07 PM, Pavol Mederly
>>                 <mederly at evolveum.com <mailto:mederly at evolveum.com>>
>>                 wrote:
>>
>>                     Strange enough. I've replicated your setup on my
>>                     computer (using PostgreSQL, as I have no MySQL
>>                     currently installed), and it works: midPoint
>>                     starts cleanly. I am able to add/delete
>>                     repository objects.
>>
>>                     Attached are my config.xml and server.xml; they
>>                     are pretty much the same as yours.
>>
>>                     So, I have no idea. You could try adding
>>                     <database>mysql</database> into <repository>
>>                     element in your config.xml, but it would most
>>                     probably make no difference.
>>
>>                     As last resort, you could turn on the most
>>                     detailed debugging for the repository on startup
>>                     by adding a line like this
>>
>>                     <logger name="com.evolveum.midpoint.re
>>                     <http://com.evolveum.midpoint.re>po" level="TRACE" />
>>
>>                     into webapps\midpoint\WEB-INF\classes\logback.xml
>>                     file and start midPoint. In idm.log there should
>>                     be quite a detailed record of what's going on.
>>
>>                     Pavol Mederly
>>                     Software developer
>>                     evolveum.com <http://evolveum.com>
>>
>>                     On 15.08.2016 21:03, Florin. Stingaciu wrote:
>>>                     Here's my datasource config:
>>>
>>>                     <Resource name="jdbc/mysql" auth="Container"
>>>                     type="javax.sql.DataSource"
>>>                     username="midpoint" password="pass"
>>>                     url="jdbc:mysql://SERVER:3306/midpoint_dev"
>>>                     driverClassName="com.mysql.jdbc.Driver"
>>>                     accessToUnderlyingConnectionAllowed="true"
>>>                     initialSize="5" maxWait="5000"
>>>                     maxActive="30" maxIdle="5"
>>>                     validationQuery="select 1"
>>>                     poolPreparedStatements="true"/>
>>>
>>>                     And here's my config.xml:
>>>
>>>                     <repository>
>>>                     <embedded>false</embedded>
>>>                     <repositoryServiceFactoryClass>com.evolveum.midpoint.repo.sql.SqlRepositoryFactory</repositoryServiceFactoryClass>
>>>                     <hibernateHbm2ddl>validate</hibernateHbm2ddl>
>>>                     <hibernateDialect>com.evolveum.midpoint.repo.sql.util.MidPointMySQLDialect</hibernateDialect>
>>>                     <dataSource>java:comp/env/jdbc/mysql</dataSource>
>>>                     </repository>
>>>                     <taskManager>
>>>                     <clustered>true</clustered>
>>>                     <jmxUsername>midpoint</jmxUsername>
>>>                     <jmxPassword>password</jmxPassword>
>>>                     </taskManager>
>>>
>>>                     On Mon, Aug 15, 2016 at 11:44 AM, Pavol Mederly
>>>                     <mederly at evolveum.com
>>>                     <mailto:mederly at evolveum.com>> wrote:
>>>
>>>                         Hello Florin,
>>>
>>>                         this is really interesting. Please, could
>>>                         you also share your midPoint config.xml, as
>>>                         well as your data source configuration?
>>>                         (except credentials, of course)
>>>
>>>                         Best regards,
>>>
>>>                         Pavol Mederly
>>>                         Software developer
>>>                         evolveum.com <http://evolveum.com>
>>>
>>>                         On 15.08.2016 19:39, Florin. Stingaciu wrote:
>>>>                         Hello Pavol,
>>>>
>>>>                         Thanks for your detailed response. I tried
>>>>                         setting up the datasource with
>>>>                         validationQuery set up properly for the
>>>>                         mySQL backed I have. However, upon service
>>>>                         restart I get the following errors:
>>>>                         http://pastebin.com/8dpGN0JC
>>>>                         <http://pastebin.com/8dpGN0JC>
>>>>
>>>>                         To save you a click, it seems as though the
>>>>                         connection is set up in readonly mode or
>>>>                         some other strange things happen. I've
>>>>                         tried setting readonly="false" in the
>>>>                         resource definition in server.xml but that
>>>>                         didn't help. I will continue researching
>>>>                         this, however any guidance would be quite
>>>>                         appreciated.
>>>>
>>>>                         Thanks!
>>>>                         -F
>>>>
>>>>                         On Mon, Aug 15, 2016 at 2:29 AM, Pavol
>>>>                         Mederly <mederly at evolveum.com
>>>>                         <mailto:mederly at evolveum.com>> wrote:
>>>>
>>>>                             Hello Florin,
>>>>
>>>>                             having looked at your logs, it seems
>>>>                             that maybe explicit setting of
>>>>                             validationQuery in Quartz data source
>>>>                             setup would help. (See
>>>>                             http://www.quartz-scheduler.org/documentation/quartz-2.x/configuration/ConfigDataSources.html
>>>>                             <http://www.quartz-scheduler.org/documentation/quartz-2.x/configuration/ConfigDataSources.html>.)
>>>>
>>>>                             Unfortunately, current midPoint
>>>>                             implementation does not allow to
>>>>                             configure Quartz data source
>>>>                             parameters. So, there are the following
>>>>                             three possibilities:
>>>>
>>>>                              1. Take an alternative route, and use
>>>>                                 application server-defined data
>>>>                                 source (with validationQuery set up).
>>>>                              2. Patch Task Manager implementation
>>>>                                 by adding appropriate lines to
>>>>                                 Quartz configuration (see
>>>>                                 LocalNodeManager.java:87-90).
>>>>                              3. Wait until we implement it - I've
>>>>                                 created an issue MID-3347
>>>>                                 <https://jira.evolveum.com/browse/MID-3347>
>>>>                                 for this.
>>>>
>>>>                             As for the first one (externally
>>>>                             defined data source): Please see
>>>>                             https://wiki.evolveum.com/display/midPoint/Repository+Configuration#RepositoryConfiguration-Datasourceconfiguration
>>>>                             <https://wiki.evolveum.com/display/midPoint/Repository+Configuration#RepositoryConfiguration-Datasourceconfiguration>
>>>>                             on how to configure midPoint repository
>>>>                             with the data source. This data source
>>>>                             will be used also by Quartz, if not
>>>>                             overriden in <taskManager> section. It
>>>>                             should work but I don't remember if
>>>>                             someone actually tested this.
>>>>
>>>>                             Concerning autoReconnect: I have no
>>>>                             experiences with this setting. I agree
>>>>                             with you that about the hesitation of
>>>>                             using it in production environment. If
>>>>                             really needed, I'd recommend to
>>>>                             separate midPoint repository
>>>>                             configuration from Quartz configuration
>>>>                             by using different JDBC URLs for the
>>>>                             two: the standard one for the
>>>>                             repository and the one with
>>>>                             "autoReconnect=true" for Quartz. In
>>>>                             this way, potential negative effects
>>>>                             should be restricted to task management
>>>>                             functionality only. But, overall, I'd
>>>>                             suggest trying to eliminate the problem
>>>>                             by setting validationQuery first.
>>>>
>>>>                             Best regards,
>>>>
>>>>                             Pavol Mederly
>>>>                             Software developer
>>>>                             evolveum.com <http://evolveum.com>
>>>>
>>>>                             On 14.08.2016 22:40, Florin. Stingaciu
>>>>                             wrote:
>>>>>                             Hello,
>>>>>
>>>>>                             I'm trying to configure an
>>>>>                             active/active configuration, and
>>>>>                             experiencing some issues with the
>>>>>                             Quartz scheduler. The SQL connection
>>>>>                             seems to timeout quite often and
>>>>>                             result in many warning messages. I'm
>>>>>                             also experiencing some errors -- as
>>>>>                             the timeout closes the connection,
>>>>>                             some processes are still trying to
>>>>>                             commit using that stale handler.
>>>>>
>>>>>                             Here are some relevant logs from both
>>>>>                             midPoint instances in this cluster:
>>>>>                             http://pastebin.com/JFjqrLnT
>>>>>                             <http://pastebin.com/JFjqrLnT>
>>>>>                             http://pastebin.com/1vy4iYPY
>>>>>                             <http://pastebin.com/1vy4iYPY>
>>>>>
>>>>>                             Also here's the relevant portion of my
>>>>>                             config.xml file:
>>>>>                             http://pastebin.com/WbydcCC8
>>>>>                             <http://pastebin.com/WbydcCC8>
>>>>>
>>>>>                             Following the suggestions in the
>>>>>                             warnings, I've started to look at
>>>>>                             autoReconnect propriety of the JDBC
>>>>>                             connector and applied it to my configs
>>>>>                             like so:
>>>>>
>>>>>                                 <jdbcUrl>jdbc:mysql://SERVER:3306/midpoint_dev?autoReconnect=true</jdbcUrl>
>>>>>
>>>>>
>>>>>                             Since implementing this change, the
>>>>>                             errors and warnings seem to have
>>>>>                             disappeared. I will continue to
>>>>>                             monitor the logs and ensure this
>>>>>                             actually the case.
>>>>>
>>>>>                             Reading the mysql docs, I found that
>>>>>                             this is not recommended as this may
>>>>>                             cause data inconsistency issues and
>>>>>                             that stale connection exceptions
>>>>>                             should be properly caught within the
>>>>>                             application. Namely:
>>>>>
>>>>>                                 The use of this feature is not
>>>>>                                 recommended, because it has side
>>>>>                                 effects related to session state
>>>>>                                 and data consistency when
>>>>>                                 applications don't handle
>>>>>                                 SQLExceptions properly, and is
>>>>>                                 only designed to be used when you
>>>>>                                 are unable to configure your
>>>>>                                 application to handle
>>>>>                                 SQLExceptions resulting from dead
>>>>>                                 and stale connections properly.
>>>>>
>>>>>
>>>>>                             Do you have any recommended
>>>>>                             configuration for this scenario? I'd
>>>>>                             like to move my current production
>>>>>                             environment in an active active
>>>>>                             configuration, however as of right now
>>>>>                             I'm hesitant to do so in order to
>>>>>                             avoid any data corruption. Especially
>>>>>                             since it's quite difficult to test for
>>>>>                             data consistency issues that may arise
>>>>>                             from using autoReconnect in my dev
>>>>>                             environment.
>>>>>
>>>>>                             Thanks,
>>>>>                             -F
>>>>>
>>>>>
>>>>>                             _______________________________________________
>>>>>                             midPoint mailing list
>>>>>                             midPoint at lists.evolveum.com
>>>>>                             <mailto:midPoint at lists.evolveum.com>
>>>>>                             http://lists.evolveum.com/mailman/listinfo/midpoint
>>>>>                             <http://lists.evolveum.com/mailman/listinfo/midpoint>
>>>>                             _______________________________________________
>>>>                             midPoint mailing list
>>>>                             midPoint at lists.evolveum.com
>>>>                             <mailto:midPoint at lists.evolveum.com>
>>>>                             http://lists.evolveum.com/mailman/listinfo/midpoint
>>>>                             <http://lists.evolveum.com/mailman/listinfo/midpoint>
>>>>
>>>>
>>>>                         _______________________________________________
>>>>                         midPoint mailing list
>>>>                         midPoint at lists.evolveum.com
>>>>                         <mailto:midPoint at lists.evolveum.com>
>>>>                         http://lists.evolveum.com/mailman/listinfo/midpoint
>>>>                         <http://lists.evolveum.com/mailman/listinfo/midpoint>
>>>                         _______________________________________________
>>>                         midPoint mailing list
>>>                         midPoint at lists.evolveum.com
>>>                         <mailto:midPoint at lists.evolveum.com>
>>>                         http://lists.evolveum.com/mailman/listinfo/midpoint
>>>                         <http://lists.evolveum.com/mailman/listinfo/midpoint>
>>>
>>>
>>>                     _______________________________________________
>>>                     midPoint mailing list
>>>                     midPoint at lists.evolveum.com
>>>                     <mailto:midPoint at lists.evolveum.com>
>>>                     http://lists.evolveum.com/mailman/listinfo/midpoint
>>>                     <http://lists.evolveum.com/mailman/listinfo/midpoint>
>>                     _______________________________________________
>>                     midPoint mailing list midPoint at lists.evolveum.com
>>                     <mailto:midPoint at lists.evolveum.com>
>>                     http://lists.evolveum.com/mailman/listinfo/midpoint
>>                     <http://lists.evolveum.com/mailman/listinfo/midpoint>
>>
>>
>>                 _______________________________________________
>>                 midPoint mailing list
>>                 midPoint at lists.evolveum.com
>>                 <mailto:midPoint at lists.evolveum.com>
>>                 http://lists.evolveum.com/mailman/listinfo/midpoint
>>                 <http://lists.evolveum.com/mailman/listinfo/midpoint>
>                 _______________________________________________
>                 midPoint mailing list midPoint at lists.evolveum.com
>                 <mailto:midPoint at lists.evolveum.com>
>                 http://lists.evolveum.com/mailman/listinfo/midpoint
>                 <http://lists.evolveum.com/mailman/listinfo/midpoint> 
>
> _______________________________________________
> midPoint mailing list
> midPoint at lists.evolveum.com
> http://lists.evolveum.com/mailman/listinfo/midpoint
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20160827/584c19ce/attachment.htm>


More information about the midPoint mailing list