[midPoint] [midpoint] Active/Active Configuration
Florin. Stingaciu
fstingaciu at mirantis.com
Mon Aug 29 20:17:23 CEST 2016
As an update, after having the instance up for half an hour or so with the
above config, I got the following errors again: http://pastebin.com/tJJVFVsu
Also, in the deployment, there's no workflows at all and very little user
load as it's just a dev env.
Thanks,
-F
On Mon, Aug 29, 2016 at 10:43 AM, Florin. Stingaciu <fstingaciu at mirantis.com
> wrote:
> Hey Pavol,
>
> Currently, for Quartz and miPoint, I'm using a tomcat7 defined datasource
> with the following config:
>
> <Resource name="jdbc/mysql" auth="Container"
> type="javax.sql.DataSource"
> username="midpoint" password="password"
> url="jdbc:mysql://servder:3306/midpoint_dev"
> driverClassName="com.mysql.jdbc.Driver"
> accessToUnderlyingConnectionAllowed="true"
> initialSize="5" maxWait="5000"
> maxActive="30" maxIdle="5"
> validationQuery="select 1"
> readonly="false"
> poolPreparedStatements="true"/>
>
> And here's my relevant config.xml:
>
> <repository>
> <embedded>false</embedded>
> <repositoryServiceFactoryClass
> >com.evolveum.midpoint.repo.sql.SqlRepositoryFactory</reposi
> toryServiceFactoryClass>
> <hibernateHbm2ddl>validate</hibernateHbm2ddl>
> <hibernateDialect>com.evolveum.midpoint.repo.sql.util.
> MidPointMySQLDialect</hibernateDialect>
> <dataSource>java:comp/env/jdbc/mysql</dataSource>
> </repository>
> <taskManager>
> <clustered>true</clustered>
> <jmxUsername>midpoint</jmxUsername>
> <jmxPassword>password</jmxPassword>
> </taskManager>
>
> With this configuration, when I was trying to access the Shadows page, I
> got the errors mentioned in my previous message.
>
> Following your last email, I added the following <dataSource>java:comp/env/jdbc/mysql</dataSource>
> under <taskManager>. Now when I try to access the Shadows Page I get the
> following error: http://imgur.com/a/391CU. Please note there are four
> photos in that album. Also the reason, I chose an image is due to the fact
> that this error does not appear anywhere in the logs and it's in the GUI
> only.
>
> I would also like to add that even though this error occurs, midPoint
> still continues to function. Before I added the <dataSource> in the
> taskManager section, even though I was using a datasource for the
> Repository, when I tried to access the shadows page, midPoint would not
> work anymore (due to lack of mysql connections) until I restarted tomcat.
>
> The last point I want to make is that from what I've seen so far, this
> issue only happens when I try to access the shadow page. I don't see this
> issue while trying to access any other type of repository objects.
>
> Thanks,
> -F
>
>
> On Fri, Aug 26, 2016 at 3:41 PM, Pavol Mederly <mederly at evolveum.com>
> wrote:
>
>> Hello Florin,
>>
>> it seems that either midPoint (i.e. hibernate), Quartz scheduler, or
>> Activiti BPMS use connections without properly closing them.
>>
>> First step to diagnose it would be perhaps to setup things like this:
>>
>> 1. Let midPoint and Activiti use their own data source (by
>> configuring them with a JDBC URL): see Repository configuration
>> <https://wiki.evolveum.com/display/midPoint/Repository+Configuration>
>> and Workflow configuration
>> <https://wiki.evolveum.com/display/midPoint/Workflow+configuration>
>> wiki pages. (If you don't use approvals you can disable workflows
>> altogether by setting "enabled" to "false".)
>> 2. Keep Quartz schedule using tomcat data source: something like
>>
>> <taskManager>
>> <clustered>true</clustered>
>> <dataSource>...your data source name...</dataSource>
>> ...
>> </taskManager>
>>
>> And you'll see if the problem would appear again. (Maybe you could even
>> configure three Tomcat data sources; one for midPoint, second for Quartz,
>> and the third for Activiti, if needed. But I have not done this yet; these
>> are pure speculations of mine...)
>>
>> Best regards,
>>
>> Pavol Mederly
>> Software developerevolveum.com
>>
>> On 24.08.2016 22:41, Florin. Stingaciu wrote:
>>
>> I just rebuilt the latest snapshot (git-v3.5devel-221-g9744fa0) and am
>> experiencing the same problem.
>>
>> Thanks,
>> -F
>>
>>
>> On Wed, Aug 24, 2016 at 1:24 PM, Florin. Stingaciu <
>> fstingaciu at mirantis.com> wrote:
>>
>>> Upon further investigation I found that this error only occurs when I
>>> try to access the shadow repository via Configuration->Repository
>>> Objects->Shadows
>>>
>>> It seems that something triggers the creation of DB connections until it
>>> runs out of available connections. For this particular instance, I'm at
>>> git-v3.5devel-170-g26b0644.
>>>
>>> Thanks,
>>> -F
>>>
>>> On Wed, Aug 24, 2016 at 1:13 PM, Florin. Stingaciu <
>>> fstingaciu at mirantis.com> wrote:
>>>
>>>> Hello Pavol,
>>>>
>>>> After running this configuration my dev environment with two instances
>>>> up for a couple of days, I started to get the following error messages:
>>>> http://pastebin.com/nkxE9LNA
>>>>
>>>> It seems as though the connection pool is not properly cleaned up. The
>>>> database logs indicate no failure and in fact my production environment is
>>>> hooked up to the same DB instance and is working fine.
>>>>
>>>> Any recommendations would be very appreciated.
>>>>
>>>> Thanks,
>>>> -F
>>>>
>>>> On Mon, Aug 15, 2016 at 3:29 PM, Florin. Stingaciu <
>>>> fstingaciu at mirantis.com> wrote:
>>>>
>>>>> Awesome! Thanks for all your help today Pavol. I will try and test in
>>>>> my environment and let you know how that goes.
>>>>>
>>>>> Thanks again!
>>>>> -F
>>>>>
>>>>> On Mon, Aug 15, 2016 at 3:00 PM, Pavol Mederly <mederly at evolveum.com>
>>>>> wrote:
>>>>>
>>>>>> Florin,
>>>>>>
>>>>>> I think I've fixed it: https://github.com/Evolveum/mi
>>>>>> dpoint/commit/c00f2979cea6f6ada0a1194b13bad51c0ecec2e3
>>>>>>
>>>>>> Cannot verify as the issue isn't visible on my computer, but I am
>>>>>> quite convinced it should be fixed now. The suspected cause is that some
>>>>>> query-only transactions weren't closed properly. C3P0 closes them
>>>>>> automatically but other connection providers probably not
>>>>>> <http://stackoverflow.com/questions/19396563/does-hibernates-session-close-automatically-rollback-uncommitted-transactions>
>>>>>> .
>>>>>>
>>>>>> You can safely use the master, or you can apply the above patch to
>>>>>> 3.4 yourself. The master is something like "3.4-support" these days
>>>>>> (preparing for 3.4.1), so it should be of production-level quality, or at
>>>>>> least near it.
>>>>>> (I'm going to run full test suite during the night, but I don't
>>>>>> expect any failures. Or you can check after approx 3 hours here
>>>>>> <https://travis-ci.org/Evolveum/midpoint>.)
>>>>>>
>>>>>> Best regards,
>>>>>>
>>>>>> Pavol Mederly
>>>>>> Software developerevolveum.com
>>>>>>
>>>>>> On 15.08.2016 22:57, Florin. Stingaciu wrote:
>>>>>>
>>>>>> Hey Pavol,
>>>>>>
>>>>>> I uploaded the log file you requested to my gdrive (
>>>>>> https://drive.google.com/file/d/0B9f4-vWBsztoZy1zWThGYnpGeG
>>>>>> s/view?usp=sharing) It's a trace level debug so the file is quite
>>>>>> large. Let me know if anything jumps out at you.
>>>>>>
>>>>>> Thanks,
>>>>>> -F
>>>>>>
>>>>>> On Mon, Aug 15, 2016 at 1:07 PM, Pavol Mederly <mederly at evolveum.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Strange enough. I've replicated your setup on my computer (using
>>>>>>> PostgreSQL, as I have no MySQL currently installed), and it works: midPoint
>>>>>>> starts cleanly. I am able to add/delete repository objects.
>>>>>>>
>>>>>>> Attached are my config.xml and server.xml; they are pretty much the
>>>>>>> same as yours.
>>>>>>>
>>>>>>> So, I have no idea. You could try adding <database>mysql</database>
>>>>>>> into <repository> element in your config.xml, but it would most probably
>>>>>>> make no difference.
>>>>>>>
>>>>>>> As last resort, you could turn on the most detailed debugging for
>>>>>>> the repository on startup by adding a line like this
>>>>>>>
>>>>>>> <logger name="com.evolveum.midpoint.repo" level="TRACE" />
>>>>>>>
>>>>>>> into webapps\midpoint\WEB-INF\classes\logback.xml file and start
>>>>>>> midPoint. In idm.log there should be quite a detailed record of what's
>>>>>>> going on.
>>>>>>>
>>>>>>> Pavol Mederly
>>>>>>> Software developerevolveum.com
>>>>>>>
>>>>>>> On 15.08.2016 21:03, Florin. Stingaciu wrote:
>>>>>>>
>>>>>>> Here's my datasource config:
>>>>>>>
>>>>>>> <Resource name="jdbc/mysql" auth="Container"
>>>>>>> type="javax.sql.DataSource"
>>>>>>> username="midpoint" password="pass"
>>>>>>> url="jdbc:mysql://SERVER:3306/midpoint_dev"
>>>>>>> driverClassName="com.mysql.jdbc.Driver"
>>>>>>> accessToUnderlyingConnectionAllowed="true"
>>>>>>> initialSize="5" maxWait="5000"
>>>>>>> maxActive="30" maxIdle="5"
>>>>>>> validationQuery="select 1"
>>>>>>> poolPreparedStatements="true"/>
>>>>>>>
>>>>>>> And here's my config.xml:
>>>>>>>
>>>>>>> <repository>
>>>>>>> <embedded>false</embedded>
>>>>>>> <repositoryServiceFactoryClass
>>>>>>> >com.evolveum.midpoint.repo.sql.SqlRepositoryFactory</reposi
>>>>>>> toryServiceFactoryClass>
>>>>>>> <hibernateHbm2ddl>validate</hibernateHbm2ddl>
>>>>>>> <hibernateDialect>com.evolveum
>>>>>>> .midpoint.repo.sql.util.MidPointMySQLDialect</hibernateDialect>
>>>>>>> <dataSource>java:comp/env/jdbc/mysql</dataSource>
>>>>>>> </repository>
>>>>>>> <taskManager>
>>>>>>> <clustered>true</clustered>
>>>>>>> <jmxUsername>midpoint</jmxUsername>
>>>>>>> <jmxPassword>password</jmxPassword>
>>>>>>> </taskManager>
>>>>>>>
>>>>>>> On Mon, Aug 15, 2016 at 11:44 AM, Pavol Mederly <
>>>>>>> mederly at evolveum.com> wrote:
>>>>>>>
>>>>>>>> Hello Florin,
>>>>>>>>
>>>>>>>> this is really interesting. Please, could you also share your
>>>>>>>> midPoint config.xml, as well as your data source configuration? (except
>>>>>>>> credentials, of course)
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>>
>>>>>>>> Pavol Mederly
>>>>>>>> Software developerevolveum.com
>>>>>>>>
>>>>>>>> On 15.08.2016 19:39, Florin. Stingaciu wrote:
>>>>>>>>
>>>>>>>> Hello Pavol,
>>>>>>>>
>>>>>>>> Thanks for your detailed response. I tried setting up the
>>>>>>>> datasource with validationQuery set up properly for the mySQL backed I
>>>>>>>> have. However, upon service restart I get the following errors:
>>>>>>>> http://pastebin.com/8dpGN0JC
>>>>>>>>
>>>>>>>> To save you a click, it seems as though the connection is set up in
>>>>>>>> readonly mode or some other strange things happen. I've tried setting
>>>>>>>> readonly="false" in the resource definition in server.xml but that
>>>>>>>> didn't help. I will continue researching this, however any guidance would
>>>>>>>> be quite appreciated.
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>> -F
>>>>>>>>
>>>>>>>> On Mon, Aug 15, 2016 at 2:29 AM, Pavol Mederly <
>>>>>>>> mederly at evolveum.com> wrote:
>>>>>>>>
>>>>>>>>> Hello Florin,
>>>>>>>>>
>>>>>>>>> having looked at your logs, it seems that maybe explicit setting
>>>>>>>>> of validationQuery in Quartz data source setup would help. (See
>>>>>>>>> http://www.quartz-scheduler.org/documentation/quartz-2.x/con
>>>>>>>>> figuration/ConfigDataSources.html.)
>>>>>>>>>
>>>>>>>>> Unfortunately, current midPoint implementation does not allow to
>>>>>>>>> configure Quartz data source parameters. So, there are the following three
>>>>>>>>> possibilities:
>>>>>>>>>
>>>>>>>>> 1. Take an alternative route, and use application
>>>>>>>>> server-defined data source (with validationQuery set up).
>>>>>>>>> 2. Patch Task Manager implementation by adding appropriate
>>>>>>>>> lines to Quartz configuration (see LocalNodeManager.java:87-90).
>>>>>>>>> 3. Wait until we implement it - I've created an issue MID-3347
>>>>>>>>> <https://jira.evolveum.com/browse/MID-3347> for this.
>>>>>>>>>
>>>>>>>>> As for the first one (externally defined data source): Please see
>>>>>>>>> https://wiki.evolveum.com/display/midPoint/Repository+Config
>>>>>>>>> uration#RepositoryConfiguration-Datasourceconfiguration on how to
>>>>>>>>> configure midPoint repository with the data source. This data source will
>>>>>>>>> be used also by Quartz, if not overriden in <taskManager> section. It
>>>>>>>>> should work but I don't remember if someone actually tested this.
>>>>>>>>>
>>>>>>>>> Concerning autoReconnect: I have no experiences with this setting.
>>>>>>>>> I agree with you that about the hesitation of using it in production
>>>>>>>>> environment. If really needed, I'd recommend to separate midPoint
>>>>>>>>> repository configuration from Quartz configuration by using different JDBC
>>>>>>>>> URLs for the two: the standard one for the repository and the one with
>>>>>>>>> "autoReconnect=true" for Quartz. In this way, potential negative effects
>>>>>>>>> should be restricted to task management functionality only. But, overall,
>>>>>>>>> I'd suggest trying to eliminate the problem by setting validationQuery
>>>>>>>>> first.
>>>>>>>>>
>>>>>>>>> Best regards,
>>>>>>>>>
>>>>>>>>> Pavol Mederly
>>>>>>>>> Software developerevolveum.com
>>>>>>>>>
>>>>>>>>> On 14.08.2016 22:40, Florin. Stingaciu wrote:
>>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> I'm trying to configure an active/active configuration, and
>>>>>>>>> experiencing some issues with the Quartz scheduler. The SQL connection
>>>>>>>>> seems to timeout quite often and result in many warning messages. I'm also
>>>>>>>>> experiencing some errors -- as the timeout closes the connection, some
>>>>>>>>> processes are still trying to commit using that stale handler.
>>>>>>>>>
>>>>>>>>> Here are some relevant logs from both midPoint instances in this
>>>>>>>>> cluster: http://pastebin.com/JFjqrLnT http://pastebin.com/1vy4iYPY
>>>>>>>>>
>>>>>>>>> Also here's the relevant portion of my config.xml file:
>>>>>>>>> http://pastebin.com/WbydcCC8
>>>>>>>>>
>>>>>>>>> Following the suggestions in the warnings, I've started to look at
>>>>>>>>> autoReconnect propriety of the JDBC connector and applied it to my configs
>>>>>>>>> like so:
>>>>>>>>>
>>>>>>>>> <jdbcUrl>jdbc:mysql://SERVER:3306/midpoint_dev?autoReconnect
>>>>>>>>>> =true</jdbcUrl>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Since implementing this change, the errors and warnings seem to
>>>>>>>>> have disappeared. I will continue to monitor the logs and ensure this
>>>>>>>>> actually the case.
>>>>>>>>>
>>>>>>>>> Reading the mysql docs, I found that this is not recommended as
>>>>>>>>> this may cause data inconsistency issues and that stale connection
>>>>>>>>> exceptions should be properly caught within the application. Namely:
>>>>>>>>>
>>>>>>>>> The use of this feature is not recommended, because it has side
>>>>>>>>>> effects related to session state and data consistency when applications
>>>>>>>>>> don't handle SQLExceptions properly, and is only designed to be used when
>>>>>>>>>> you are unable to configure your application to handle SQLExceptions
>>>>>>>>>> resulting from dead and stale connections properly.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Do you have any recommended configuration for this scenario? I'd
>>>>>>>>> like to move my current production environment in an active active
>>>>>>>>> configuration, however as of right now I'm hesitant to do so in order to
>>>>>>>>> avoid any data corruption. Especially since it's quite difficult to test
>>>>>>>>> for data consistency issues that may arise from using autoReconnect in my
>>>>>>>>> dev environment.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> -F
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> midPoint mailing listmidPoint at lists.evolveum.comhttp://lists.evolveum.com/mailman/listinfo/midpoint
>>>>>>>>>
>>>>>>>>> _______________________________________________ midPoint mailing
>>>>>>>>> list midPoint at lists.evolveum.com http://lists.evolveum.com/mail
>>>>>>>>> man/listinfo/midpoint
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> midPoint mailing listmidPoint at lists.evolveum.comhttp://lists.evolveum.com/mailman/listinfo/midpoint
>>>>>>>>
>>>>>>>> _______________________________________________ midPoint mailing
>>>>>>>> list midPoint at lists.evolveum.com http://lists.evolveum.com/mail
>>>>>>>> man/listinfo/midpoint
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> midPoint mailing listmidPoint at lists.evolveum.comhttp://lists.evolveum.com/mailman/listinfo/midpoint
>>>>>>>
>>>>>>> _______________________________________________ midPoint mailing
>>>>>>> list midPoint at lists.evolveum.com http://lists.evolveum.com/mail
>>>>>>> man/listinfo/midpoint
>>>>>>
>>>>>> _______________________________________________
>>>>>> midPoint mailing listmidPoint at lists.evolveum.comhttp://lists.evolveum.com/mailman/listinfo/midpoint
>>>>>>
>>>>>> _______________________________________________ midPoint mailing
>>>>>> list midPoint at lists.evolveum.com http://lists.evolveum.com/mail
>>>>>> man/listinfo/midpoint
>>>>>
>>>>> _______________________________________________
>> midPoint mailing listmidPoint at lists.evolveum.comhttp://lists.evolveum.com/mailman/listinfo/midpoint
>>
>>
>> _______________________________________________
>> midPoint mailing list
>> midPoint at lists.evolveum.com
>> http://lists.evolveum.com/mailman/listinfo/midpoint
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20160829/dadc784e/attachment.htm>
More information about the midPoint
mailing list