[midPoint] [midpoint] Active/Active Configuration
Florin. Stingaciu
fstingaciu at mirantis.com
Mon Aug 29 19:43:40 CEST 2016
Hey Pavol,
Currently, for Quartz and miPoint, I'm using a tomcat7 defined datasource
with the following config:
<Resource name="jdbc/mysql" auth="Container" type="javax.sql.DataSource"
username="midpoint" password="password"
url="jdbc:mysql://servder:3306/midpoint_dev"
driverClassName="com.mysql.jdbc.Driver"
accessToUnderlyingConnectionAllowed="true"
initialSize="5" maxWait="5000"
maxActive="30" maxIdle="5"
validationQuery="select 1"
readonly="false"
poolPreparedStatements="true"/>
And here's my relevant config.xml:
<repository>
<embedded>false</embedded>
<repositoryServiceFactoryClass>com.evolveum.midpoint.repo.
sql.SqlRepositoryFactory</repositoryServiceFactoryClass>
<hibernateHbm2ddl>validate</hibernateHbm2ddl>
<hibernateDialect>com.evolveum.midpoint.repo.sql.
util.MidPointMySQLDialect</hibernateDialect>
<dataSource>java:comp/env/jdbc/mysql</dataSource>
</repository>
<taskManager>
<clustered>true</clustered>
<jmxUsername>midpoint</jmxUsername>
<jmxPassword>password</jmxPassword>
</taskManager>
With this configuration, when I was trying to access the Shadows page, I
got the errors mentioned in my previous message.
Following your last email, I added the following
<dataSource>java:comp/env/jdbc/mysql</dataSource> under <taskManager>. Now
when I try to access the Shadows Page I get the following error:
http://imgur.com/a/391CU. Please note there are four photos in that album.
Also the reason, I chose an image is due to the fact that this error does
not appear anywhere in the logs and it's in the GUI only.
I would also like to add that even though this error occurs, midPoint still
continues to function. Before I added the <dataSource> in the taskManager
section, even though I was using a datasource for the Repository, when I
tried to access the shadows page, midPoint would not work anymore (due to
lack of mysql connections) until I restarted tomcat.
The last point I want to make is that from what I've seen so far, this
issue only happens when I try to access the shadow page. I don't see this
issue while trying to access any other type of repository objects.
Thanks,
-F
On Fri, Aug 26, 2016 at 3:41 PM, Pavol Mederly <mederly at evolveum.com> wrote:
> Hello Florin,
>
> it seems that either midPoint (i.e. hibernate), Quartz scheduler, or
> Activiti BPMS use connections without properly closing them.
>
> First step to diagnose it would be perhaps to setup things like this:
>
> 1. Let midPoint and Activiti use their own data source (by configuring
> them with a JDBC URL): see Repository configuration
> <https://wiki.evolveum.com/display/midPoint/Repository+Configuration>
> and Workflow configuration
> <https://wiki.evolveum.com/display/midPoint/Workflow+configuration>
> wiki pages. (If you don't use approvals you can disable workflows
> altogether by setting "enabled" to "false".)
> 2. Keep Quartz schedule using tomcat data source: something like
>
> <taskManager>
> <clustered>true</clustered>
> <dataSource>...your data source name...</dataSource>
> ...
> </taskManager>
>
> And you'll see if the problem would appear again. (Maybe you could even
> configure three Tomcat data sources; one for midPoint, second for Quartz,
> and the third for Activiti, if needed. But I have not done this yet; these
> are pure speculations of mine...)
>
> Best regards,
>
> Pavol Mederly
> Software developerevolveum.com
>
> On 24.08.2016 22:41, Florin. Stingaciu wrote:
>
> I just rebuilt the latest snapshot (git-v3.5devel-221-g9744fa0) and am
> experiencing the same problem.
>
> Thanks,
> -F
>
>
> On Wed, Aug 24, 2016 at 1:24 PM, Florin. Stingaciu <
> fstingaciu at mirantis.com> wrote:
>
>> Upon further investigation I found that this error only occurs when I try
>> to access the shadow repository via Configuration->Repository
>> Objects->Shadows
>>
>> It seems that something triggers the creation of DB connections until it
>> runs out of available connections. For this particular instance, I'm at
>> git-v3.5devel-170-g26b0644.
>>
>> Thanks,
>> -F
>>
>> On Wed, Aug 24, 2016 at 1:13 PM, Florin. Stingaciu <
>> fstingaciu at mirantis.com> wrote:
>>
>>> Hello Pavol,
>>>
>>> After running this configuration my dev environment with two instances
>>> up for a couple of days, I started to get the following error messages:
>>> http://pastebin.com/nkxE9LNA
>>>
>>> It seems as though the connection pool is not properly cleaned up. The
>>> database logs indicate no failure and in fact my production environment is
>>> hooked up to the same DB instance and is working fine.
>>>
>>> Any recommendations would be very appreciated.
>>>
>>> Thanks,
>>> -F
>>>
>>> On Mon, Aug 15, 2016 at 3:29 PM, Florin. Stingaciu <
>>> fstingaciu at mirantis.com> wrote:
>>>
>>>> Awesome! Thanks for all your help today Pavol. I will try and test in
>>>> my environment and let you know how that goes.
>>>>
>>>> Thanks again!
>>>> -F
>>>>
>>>> On Mon, Aug 15, 2016 at 3:00 PM, Pavol Mederly <mederly at evolveum.com>
>>>> wrote:
>>>>
>>>>> Florin,
>>>>>
>>>>> I think I've fixed it: https://github.com/Evolveum/mi
>>>>> dpoint/commit/c00f2979cea6f6ada0a1194b13bad51c0ecec2e3
>>>>>
>>>>> Cannot verify as the issue isn't visible on my computer, but I am
>>>>> quite convinced it should be fixed now. The suspected cause is that some
>>>>> query-only transactions weren't closed properly. C3P0 closes them
>>>>> automatically but other connection providers probably not
>>>>> <http://stackoverflow.com/questions/19396563/does-hibernates-session-close-automatically-rollback-uncommitted-transactions>
>>>>> .
>>>>>
>>>>> You can safely use the master, or you can apply the above patch to 3.4
>>>>> yourself. The master is something like "3.4-support" these days (preparing
>>>>> for 3.4.1), so it should be of production-level quality, or at least near
>>>>> it.
>>>>> (I'm going to run full test suite during the night, but I don't expect
>>>>> any failures. Or you can check after approx 3 hours here
>>>>> <https://travis-ci.org/Evolveum/midpoint>.)
>>>>>
>>>>> Best regards,
>>>>>
>>>>> Pavol Mederly
>>>>> Software developerevolveum.com
>>>>>
>>>>> On 15.08.2016 22:57, Florin. Stingaciu wrote:
>>>>>
>>>>> Hey Pavol,
>>>>>
>>>>> I uploaded the log file you requested to my gdrive (
>>>>> https://drive.google.com/file/d/0B9f4-vWBsztoZy1zWThGYnpGeG
>>>>> s/view?usp=sharing) It's a trace level debug so the file is quite
>>>>> large. Let me know if anything jumps out at you.
>>>>>
>>>>> Thanks,
>>>>> -F
>>>>>
>>>>> On Mon, Aug 15, 2016 at 1:07 PM, Pavol Mederly <mederly at evolveum.com>
>>>>> wrote:
>>>>>
>>>>>> Strange enough. I've replicated your setup on my computer (using
>>>>>> PostgreSQL, as I have no MySQL currently installed), and it works: midPoint
>>>>>> starts cleanly. I am able to add/delete repository objects.
>>>>>>
>>>>>> Attached are my config.xml and server.xml; they are pretty much the
>>>>>> same as yours.
>>>>>>
>>>>>> So, I have no idea. You could try adding <database>mysql</database>
>>>>>> into <repository> element in your config.xml, but it would most probably
>>>>>> make no difference.
>>>>>>
>>>>>> As last resort, you could turn on the most detailed debugging for the
>>>>>> repository on startup by adding a line like this
>>>>>>
>>>>>> <logger name="com.evolveum.midpoint.repo" level="TRACE" />
>>>>>>
>>>>>> into webapps\midpoint\WEB-INF\classes\logback.xml file and start
>>>>>> midPoint. In idm.log there should be quite a detailed record of what's
>>>>>> going on.
>>>>>>
>>>>>> Pavol Mederly
>>>>>> Software developerevolveum.com
>>>>>>
>>>>>> On 15.08.2016 21:03, Florin. Stingaciu wrote:
>>>>>>
>>>>>> Here's my datasource config:
>>>>>>
>>>>>> <Resource name="jdbc/mysql" auth="Container"
>>>>>> type="javax.sql.DataSource"
>>>>>> username="midpoint" password="pass"
>>>>>> url="jdbc:mysql://SERVER:3306/midpoint_dev"
>>>>>> driverClassName="com.mysql.jdbc.Driver"
>>>>>> accessToUnderlyingConnectionAllowed="true"
>>>>>> initialSize="5" maxWait="5000"
>>>>>> maxActive="30" maxIdle="5"
>>>>>> validationQuery="select 1"
>>>>>> poolPreparedStatements="true"/>
>>>>>>
>>>>>> And here's my config.xml:
>>>>>>
>>>>>> <repository>
>>>>>> <embedded>false</embedded>
>>>>>> <repositoryServiceFactoryClass
>>>>>> >com.evolveum.midpoint.repo.sql.SqlRepositoryFactory</reposi
>>>>>> toryServiceFactoryClass>
>>>>>> <hibernateHbm2ddl>validate</hibernateHbm2ddl>
>>>>>> <hibernateDialect>com.evolveum
>>>>>> .midpoint.repo.sql.util.MidPointMySQLDialect</hibernateDialect>
>>>>>> <dataSource>java:comp/env/jdbc/mysql</dataSource>
>>>>>> </repository>
>>>>>> <taskManager>
>>>>>> <clustered>true</clustered>
>>>>>> <jmxUsername>midpoint</jmxUsername>
>>>>>> <jmxPassword>password</jmxPassword>
>>>>>> </taskManager>
>>>>>>
>>>>>> On Mon, Aug 15, 2016 at 11:44 AM, Pavol Mederly <mederly at evolveum.com
>>>>>> > wrote:
>>>>>>
>>>>>>> Hello Florin,
>>>>>>>
>>>>>>> this is really interesting. Please, could you also share your
>>>>>>> midPoint config.xml, as well as your data source configuration? (except
>>>>>>> credentials, of course)
>>>>>>>
>>>>>>> Best regards,
>>>>>>>
>>>>>>> Pavol Mederly
>>>>>>> Software developerevolveum.com
>>>>>>>
>>>>>>> On 15.08.2016 19:39, Florin. Stingaciu wrote:
>>>>>>>
>>>>>>> Hello Pavol,
>>>>>>>
>>>>>>> Thanks for your detailed response. I tried setting up the datasource
>>>>>>> with validationQuery set up properly for the mySQL backed I have. However,
>>>>>>> upon service restart I get the following errors:
>>>>>>> http://pastebin.com/8dpGN0JC
>>>>>>>
>>>>>>> To save you a click, it seems as though the connection is set up in
>>>>>>> readonly mode or some other strange things happen. I've tried setting
>>>>>>> readonly="false" in the resource definition in server.xml but that
>>>>>>> didn't help. I will continue researching this, however any guidance would
>>>>>>> be quite appreciated.
>>>>>>>
>>>>>>> Thanks!
>>>>>>> -F
>>>>>>>
>>>>>>> On Mon, Aug 15, 2016 at 2:29 AM, Pavol Mederly <mederly at evolveum.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> Hello Florin,
>>>>>>>>
>>>>>>>> having looked at your logs, it seems that maybe explicit setting of
>>>>>>>> validationQuery in Quartz data source setup would help. (See
>>>>>>>> http://www.quartz-scheduler.org/documentation/quartz-2.x/con
>>>>>>>> figuration/ConfigDataSources.html.)
>>>>>>>>
>>>>>>>> Unfortunately, current midPoint implementation does not allow to
>>>>>>>> configure Quartz data source parameters. So, there are the following three
>>>>>>>> possibilities:
>>>>>>>>
>>>>>>>> 1. Take an alternative route, and use application
>>>>>>>> server-defined data source (with validationQuery set up).
>>>>>>>> 2. Patch Task Manager implementation by adding appropriate
>>>>>>>> lines to Quartz configuration (see LocalNodeManager.java:87-90).
>>>>>>>> 3. Wait until we implement it - I've created an issue MID-3347
>>>>>>>> <https://jira.evolveum.com/browse/MID-3347> for this.
>>>>>>>>
>>>>>>>> As for the first one (externally defined data source): Please see
>>>>>>>> https://wiki.evolveum.com/display/midPoint/Repository+Config
>>>>>>>> uration#RepositoryConfiguration-Datasourceconfiguration on how to
>>>>>>>> configure midPoint repository with the data source. This data source will
>>>>>>>> be used also by Quartz, if not overriden in <taskManager> section. It
>>>>>>>> should work but I don't remember if someone actually tested this.
>>>>>>>>
>>>>>>>> Concerning autoReconnect: I have no experiences with this setting.
>>>>>>>> I agree with you that about the hesitation of using it in production
>>>>>>>> environment. If really needed, I'd recommend to separate midPoint
>>>>>>>> repository configuration from Quartz configuration by using different JDBC
>>>>>>>> URLs for the two: the standard one for the repository and the one with
>>>>>>>> "autoReconnect=true" for Quartz. In this way, potential negative effects
>>>>>>>> should be restricted to task management functionality only. But, overall,
>>>>>>>> I'd suggest trying to eliminate the problem by setting validationQuery
>>>>>>>> first.
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>>
>>>>>>>> Pavol Mederly
>>>>>>>> Software developerevolveum.com
>>>>>>>>
>>>>>>>> On 14.08.2016 22:40, Florin. Stingaciu wrote:
>>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> I'm trying to configure an active/active configuration, and
>>>>>>>> experiencing some issues with the Quartz scheduler. The SQL connection
>>>>>>>> seems to timeout quite often and result in many warning messages. I'm also
>>>>>>>> experiencing some errors -- as the timeout closes the connection, some
>>>>>>>> processes are still trying to commit using that stale handler.
>>>>>>>>
>>>>>>>> Here are some relevant logs from both midPoint instances in this
>>>>>>>> cluster: http://pastebin.com/JFjqrLnT http://pastebin.com/1vy4iYPY
>>>>>>>>
>>>>>>>> Also here's the relevant portion of my config.xml file:
>>>>>>>> http://pastebin.com/WbydcCC8
>>>>>>>>
>>>>>>>> Following the suggestions in the warnings, I've started to look at
>>>>>>>> autoReconnect propriety of the JDBC connector and applied it to my configs
>>>>>>>> like so:
>>>>>>>>
>>>>>>>> <jdbcUrl>jdbc:mysql://SERVER:3306/midpoint_dev?autoReconnect
>>>>>>>>> =true</jdbcUrl>
>>>>>>>>
>>>>>>>>
>>>>>>>> Since implementing this change, the errors and warnings seem to
>>>>>>>> have disappeared. I will continue to monitor the logs and ensure this
>>>>>>>> actually the case.
>>>>>>>>
>>>>>>>> Reading the mysql docs, I found that this is not recommended as
>>>>>>>> this may cause data inconsistency issues and that stale connection
>>>>>>>> exceptions should be properly caught within the application. Namely:
>>>>>>>>
>>>>>>>> The use of this feature is not recommended, because it has side
>>>>>>>>> effects related to session state and data consistency when applications
>>>>>>>>> don't handle SQLExceptions properly, and is only designed to be used when
>>>>>>>>> you are unable to configure your application to handle SQLExceptions
>>>>>>>>> resulting from dead and stale connections properly.
>>>>>>>>
>>>>>>>>
>>>>>>>> Do you have any recommended configuration for this scenario? I'd
>>>>>>>> like to move my current production environment in an active active
>>>>>>>> configuration, however as of right now I'm hesitant to do so in order to
>>>>>>>> avoid any data corruption. Especially since it's quite difficult to test
>>>>>>>> for data consistency issues that may arise from using autoReconnect in my
>>>>>>>> dev environment.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> -F
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> midPoint mailing listmidPoint at lists.evolveum.comhttp://lists.evolveum.com/mailman/listinfo/midpoint
>>>>>>>>
>>>>>>>> _______________________________________________ midPoint mailing
>>>>>>>> list midPoint at lists.evolveum.com http://lists.evolveum.com/mail
>>>>>>>> man/listinfo/midpoint
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> midPoint mailing listmidPoint at lists.evolveum.comhttp://lists.evolveum.com/mailman/listinfo/midpoint
>>>>>>>
>>>>>>> _______________________________________________ midPoint mailing
>>>>>>> list midPoint at lists.evolveum.com http://lists.evolveum.com/mail
>>>>>>> man/listinfo/midpoint
>>>>>>
>>>>>> _______________________________________________
>>>>>> midPoint mailing listmidPoint at lists.evolveum.comhttp://lists.evolveum.com/mailman/listinfo/midpoint
>>>>>>
>>>>>> _______________________________________________ midPoint mailing
>>>>>> list midPoint at lists.evolveum.com http://lists.evolveum.com/mail
>>>>>> man/listinfo/midpoint
>>>>>
>>>>> _______________________________________________
>>>>> midPoint mailing listmidPoint at lists.evolveum.comhttp://lists.evolveum.com/mailman/listinfo/midpoint
>>>>>
>>>>> _______________________________________________ midPoint mailing list
>>>>> midPoint at lists.evolveum.com http://lists.evolveum.com/mail
>>>>> man/listinfo/midpoint
>>>>
>>>> _______________________________________________
> midPoint mailing listmidPoint at lists.evolveum.comhttp://lists.evolveum.com/mailman/listinfo/midpoint
>
>
> _______________________________________________
> midPoint mailing list
> midPoint at lists.evolveum.com
> http://lists.evolveum.com/mailman/listinfo/midpoint
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20160829/885a67d9/attachment.htm>
More information about the midPoint
mailing list