[midPoint] [midpoint] Active/Active Configuration
Florin. Stingaciu
fstingaciu at mirantis.com
Tue Aug 30 00:54:05 CEST 2016
Hey Pavol,
I switched the repository to the configuration you mentioned, and I could
now access the shadow page. However (minus the task manager config), this
is the same configuration as I previously had where I ran into errors and
then made this thread because of it. For now I'm not seeing any errors in
the log files.
I'll update this thread once I run into issues. As of right now, I'm still
definitely not comfortable moving to active active considering all these
issues. Have you guys seen similar issues before? Or do you believe I've
just misconfigured something?
Thanks,
-F
On Mon, Aug 29, 2016 at 3:39 PM, Pavol Mederly <mederly at evolveum.com> wrote:
> Hello Florin,
>
> I've tried to use your setup in my environment (slightly modified), but I
> was not able to replicate the exceptions. I've tried in particular to play
> with the listing of shadow objects.
>
> So we perhaps have to continue diagnosing on your side.
>
> I suggest to try to isolate the problem. If you could use the following
> config:
>
> <repository>
> <embedded>false</embedded>
> <repositoryServiceFactoryClass
> >com.evolveum.midpoint.repo.sql.SqlRepositoryFactory</reposi
> toryServiceFactoryClass>
> <hibernateHbm2ddl>validate</hibernateHbm2ddl>
> <database>mysql</database>
> <jdbcUsername>...</jdbcUsername>
> <jdbcPassword>...</jdbcPassword>
> <jdbcUrl>jdbc:mysql://servder:3306/midpoint_dev</jdbcUrl>
> </repository>
> <taskManager>
> <clustered>true</clustered>
> <jmxUsername>midpoint</jmxUsername>
> <jmxPassword>password</jmxPassword>
> <dataSource>java:comp/env/jdbc/mysql</dataSource>
> </taskManager>
> <workflow>
> <enabled>false</enabled>
> </workflow>
>
> ...and try the testing. The configuration simply makes midPoint use its
> own connection mechanism (c3p0) and Quartz use the tomcat's connection pool.
>
> Best regards,
>
> Pavol Mederly
> Software developerevolveum.com
>
> On 29.08.2016 20:17, Florin. Stingaciu wrote:
>
> As an update, after having the instance up for half an hour or so with the
> above config, I got the following errors again: http://pastebin.com/
> tJJVFVsu
>
> Also, in the deployment, there's no workflows at all and very little user
> load as it's just a dev env.
>
> Thanks,
> -F
>
> On Mon, Aug 29, 2016 at 10:43 AM, Florin. Stingaciu <
> fstingaciu at mirantis.com> wrote:
>
>> Hey Pavol,
>>
>> Currently, for Quartz and miPoint, I'm using a tomcat7 defined datasource
>> with the following config:
>>
>> <Resource name="jdbc/mysql" auth="Container"
>> type="javax.sql.DataSource"
>> username="midpoint" password="password"
>> url="jdbc:mysql://servder:3306/midpoint_dev"
>> driverClassName="com.mysql.jdbc.Driver"
>> accessToUnderlyingConnectionAllowed="true"
>> initialSize="5" maxWait="5000"
>> maxActive="30" maxIdle="5"
>> validationQuery="select 1"
>> readonly="false"
>> poolPreparedStatements="true"/>
>>
>> And here's my relevant config.xml:
>>
>> <repository>
>> <embedded>false</embedded>
>> <repositoryServiceFactoryClass
>> >com.evolveum.midpoint.repo.sql.SqlRepositoryFactory</reposi
>> toryServiceFactoryClass>
>> <hibernateHbm2ddl>validate</hibernateHbm2ddl>
>> <hibernateDialect>com.evolveum
>> .midpoint.repo.sql.util.MidPointMySQLDialect</hibernateDialect>
>> <dataSource>java:comp/env/jdbc/mysql</dataSource>
>> </repository>
>> <taskManager>
>> <clustered>true</clustered>
>> <jmxUsername>midpoint</jmxUsername>
>> <jmxPassword>password</jmxPassword>
>> </taskManager>
>>
>> With this configuration, when I was trying to access the Shadows page, I
>> got the errors mentioned in my previous message.
>>
>> Following your last email, I added the following
>> <dataSource>java:comp/env/jdbc/mysql</dataSource> under <taskManager>.
>> Now when I try to access the Shadows Page I get the following error:
>> http://imgur.com/a/391CU. Please note there are four photos in that
>> album. Also the reason, I chose an image is due to the fact that this error
>> does not appear anywhere in the logs and it's in the GUI only.
>>
>> I would also like to add that even though this error occurs, midPoint
>> still continues to function. Before I added the <dataSource> in the
>> taskManager section, even though I was using a datasource for the
>> Repository, when I tried to access the shadows page, midPoint would not
>> work anymore (due to lack of mysql connections) until I restarted tomcat.
>>
>> The last point I want to make is that from what I've seen so far, this
>> issue only happens when I try to access the shadow page. I don't see this
>> issue while trying to access any other type of repository objects.
>>
>> Thanks,
>> -F
>>
>>
>> On Fri, Aug 26, 2016 at 3:41 PM, Pavol Mederly <mederly at evolveum.com>
>> wrote:
>>
>>> Hello Florin,
>>>
>>> it seems that either midPoint (i.e. hibernate), Quartz scheduler, or
>>> Activiti BPMS use connections without properly closing them.
>>>
>>> First step to diagnose it would be perhaps to setup things like this:
>>>
>>> 1. Let midPoint and Activiti use their own data source (by
>>> configuring them with a JDBC URL): see Repository configuration
>>> <https://wiki.evolveum.com/display/midPoint/Repository+Configuration>
>>> and Workflow configuration
>>> <https://wiki.evolveum.com/display/midPoint/Workflow+configuration>
>>> wiki pages. (If you don't use approvals you can disable workflows
>>> altogether by setting "enabled" to "false".)
>>> 2. Keep Quartz schedule using tomcat data source: something like
>>>
>>> <taskManager>
>>> <clustered>true</clustered>
>>> <dataSource>...your data source name...</dataSource>
>>> ...
>>> </taskManager>
>>>
>>> And you'll see if the problem would appear again. (Maybe you could even
>>> configure three Tomcat data sources; one for midPoint, second for Quartz,
>>> and the third for Activiti, if needed. But I have not done this yet; these
>>> are pure speculations of mine...)
>>>
>>> Best regards,
>>>
>>> Pavol Mederly
>>> Software developerevolveum.com
>>>
>>> On 24.08.2016 22:41, Florin. Stingaciu wrote:
>>>
>>> I just rebuilt the latest snapshot (git-v3.5devel-221-g9744fa0) and am
>>> experiencing the same problem.
>>>
>>> Thanks,
>>> -F
>>>
>>>
>>> On Wed, Aug 24, 2016 at 1:24 PM, Florin. Stingaciu <
>>> fstingaciu at mirantis.com> wrote:
>>>
>>>> Upon further investigation I found that this error only occurs when I
>>>> try to access the shadow repository via Configuration->Repository
>>>> Objects->Shadows
>>>>
>>>> It seems that something triggers the creation of DB connections until
>>>> it runs out of available connections. For this particular instance, I'm at
>>>> git-v3.5devel-170-g26b0644.
>>>>
>>>> Thanks,
>>>> -F
>>>>
>>>> On Wed, Aug 24, 2016 at 1:13 PM, Florin. Stingaciu <
>>>> fstingaciu at mirantis.com> wrote:
>>>>
>>>>> Hello Pavol,
>>>>>
>>>>> After running this configuration my dev environment with two instances
>>>>> up for a couple of days, I started to get the following error messages:
>>>>> http://pastebin.com/nkxE9LNA
>>>>>
>>>>> It seems as though the connection pool is not properly cleaned up. The
>>>>> database logs indicate no failure and in fact my production environment is
>>>>> hooked up to the same DB instance and is working fine.
>>>>>
>>>>> Any recommendations would be very appreciated.
>>>>>
>>>>> Thanks,
>>>>> -F
>>>>>
>>>>> On Mon, Aug 15, 2016 at 3:29 PM, Florin. Stingaciu <
>>>>> fstingaciu at mirantis.com> wrote:
>>>>>
>>>>>> Awesome! Thanks for all your help today Pavol. I will try and test in
>>>>>> my environment and let you know how that goes.
>>>>>>
>>>>>> Thanks again!
>>>>>> -F
>>>>>>
>>>>>> On Mon, Aug 15, 2016 at 3:00 PM, Pavol Mederly <mederly at evolveum.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Florin,
>>>>>>>
>>>>>>> I think I've fixed it: https://github.com/Evolveum/mi
>>>>>>> dpoint/commit/c00f2979cea6f6ada0a1194b13bad51c0ecec2e3
>>>>>>>
>>>>>>> Cannot verify as the issue isn't visible on my computer, but I am
>>>>>>> quite convinced it should be fixed now. The suspected cause is that some
>>>>>>> query-only transactions weren't closed properly. C3P0 closes them
>>>>>>> automatically but other connection providers probably not
>>>>>>> <http://stackoverflow.com/questions/19396563/does-hibernates-session-close-automatically-rollback-uncommitted-transactions>
>>>>>>> .
>>>>>>>
>>>>>>> You can safely use the master, or you can apply the above patch to
>>>>>>> 3.4 yourself. The master is something like "3.4-support" these days
>>>>>>> (preparing for 3.4.1), so it should be of production-level quality, or at
>>>>>>> least near it.
>>>>>>> (I'm going to run full test suite during the night, but I don't
>>>>>>> expect any failures. Or you can check after approx 3 hours here
>>>>>>> <https://travis-ci.org/Evolveum/midpoint>.)
>>>>>>>
>>>>>>> Best regards,
>>>>>>>
>>>>>>> Pavol Mederly
>>>>>>> Software developerevolveum.com
>>>>>>>
>>>>>>> On 15.08.2016 22:57, Florin. Stingaciu wrote:
>>>>>>>
>>>>>>> Hey Pavol,
>>>>>>>
>>>>>>> I uploaded the log file you requested to my gdrive (
>>>>>>> https://drive.google.com/file/d/0B9f4-vWBsztoZy1zWThGYnpGeG
>>>>>>> s/view?usp=sharing) It's a trace level debug so the file is quite
>>>>>>> large. Let me know if anything jumps out at you.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> -F
>>>>>>>
>>>>>>> On Mon, Aug 15, 2016 at 1:07 PM, Pavol Mederly <mederly at evolveum.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> Strange enough. I've replicated your setup on my computer (using
>>>>>>>> PostgreSQL, as I have no MySQL currently installed), and it works: midPoint
>>>>>>>> starts cleanly. I am able to add/delete repository objects.
>>>>>>>>
>>>>>>>> Attached are my config.xml and server.xml; they are pretty much the
>>>>>>>> same as yours.
>>>>>>>>
>>>>>>>> So, I have no idea. You could try adding <database>mysql</database>
>>>>>>>> into <repository> element in your config.xml, but it would most probably
>>>>>>>> make no difference.
>>>>>>>>
>>>>>>>> As last resort, you could turn on the most detailed debugging for
>>>>>>>> the repository on startup by adding a line like this
>>>>>>>>
>>>>>>>> <logger name="com.evolveum.midpoint.repo" level="TRACE" />
>>>>>>>>
>>>>>>>> into webapps\midpoint\WEB-INF\classes\logback.xml file and start
>>>>>>>> midPoint. In idm.log there should be quite a detailed record of what's
>>>>>>>> going on.
>>>>>>>>
>>>>>>>> Pavol Mederly
>>>>>>>> Software developerevolveum.com
>>>>>>>>
>>>>>>>> On 15.08.2016 21:03, Florin. Stingaciu wrote:
>>>>>>>>
>>>>>>>> Here's my datasource config:
>>>>>>>>
>>>>>>>> <Resource name="jdbc/mysql" auth="Container"
>>>>>>>> type="javax.sql.DataSource"
>>>>>>>> username="midpoint" password="pass"
>>>>>>>> url="jdbc:mysql://SERVER:3306/midpoint_dev"
>>>>>>>> driverClassName="com.mysql.jdbc.Driver"
>>>>>>>> accessToUnderlyingConnectionAllowed="true"
>>>>>>>> initialSize="5" maxWait="5000"
>>>>>>>> maxActive="30" maxIdle="5"
>>>>>>>> validationQuery="select 1"
>>>>>>>> poolPreparedStatements="true"/>
>>>>>>>>
>>>>>>>> And here's my config.xml:
>>>>>>>>
>>>>>>>> <repository>
>>>>>>>> <embedded>false</embedded>
>>>>>>>> <repositoryServiceFactoryClass
>>>>>>>> >com.evolveum.midpoint.repo.sql.SqlRepositoryFactory</reposi
>>>>>>>> toryServiceFactoryClass>
>>>>>>>> <hibernateHbm2ddl>validate</hibernateHbm2ddl>
>>>>>>>> <hibernateDialect>com.evolveum
>>>>>>>> .midpoint.repo.sql.util.MidPointMySQLDialect</hibernateDialect>
>>>>>>>> <dataSource>java:comp/env/jdbc/mysql</dataSource>
>>>>>>>> </repository>
>>>>>>>> <taskManager>
>>>>>>>> <clustered>true</clustered>
>>>>>>>> <jmxUsername>midpoint</jmxUsername>
>>>>>>>> <jmxPassword>password</jmxPassword>
>>>>>>>> </taskManager>
>>>>>>>>
>>>>>>>> On Mon, Aug 15, 2016 at 11:44 AM, Pavol Mederly <
>>>>>>>> mederly at evolveum.com> wrote:
>>>>>>>>
>>>>>>>>> Hello Florin,
>>>>>>>>>
>>>>>>>>> this is really interesting. Please, could you also share your
>>>>>>>>> midPoint config.xml, as well as your data source configuration? (except
>>>>>>>>> credentials, of course)
>>>>>>>>>
>>>>>>>>> Best regards,
>>>>>>>>>
>>>>>>>>> Pavol Mederly
>>>>>>>>> Software developerevolveum.com
>>>>>>>>>
>>>>>>>>> On 15.08.2016 19:39, Florin. Stingaciu wrote:
>>>>>>>>>
>>>>>>>>> Hello Pavol,
>>>>>>>>>
>>>>>>>>> Thanks for your detailed response. I tried setting up the
>>>>>>>>> datasource with validationQuery set up properly for the mySQL backed I
>>>>>>>>> have. However, upon service restart I get the following errors:
>>>>>>>>> http://pastebin.com/8dpGN0JC
>>>>>>>>>
>>>>>>>>> To save you a click, it seems as though the connection is set up
>>>>>>>>> in readonly mode or some other strange things happen. I've tried setting
>>>>>>>>> readonly="false" in the resource definition in server.xml but
>>>>>>>>> that didn't help. I will continue researching this, however any guidance
>>>>>>>>> would be quite appreciated.
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>> -F
>>>>>>>>>
>>>>>>>>> On Mon, Aug 15, 2016 at 2:29 AM, Pavol Mederly <
>>>>>>>>> mederly at evolveum.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hello Florin,
>>>>>>>>>>
>>>>>>>>>> having looked at your logs, it seems that maybe explicit setting
>>>>>>>>>> of validationQuery in Quartz data source setup would help. (See
>>>>>>>>>> http://www.quartz-scheduler.org/documentation/quartz-2.x/con
>>>>>>>>>> figuration/ConfigDataSources.html.)
>>>>>>>>>>
>>>>>>>>>> Unfortunately, current midPoint implementation does not allow to
>>>>>>>>>> configure Quartz data source parameters. So, there are the following three
>>>>>>>>>> possibilities:
>>>>>>>>>>
>>>>>>>>>> 1. Take an alternative route, and use application
>>>>>>>>>> server-defined data source (with validationQuery set up).
>>>>>>>>>> 2. Patch Task Manager implementation by adding appropriate
>>>>>>>>>> lines to Quartz configuration (see LocalNodeManager.java:87-90).
>>>>>>>>>> 3. Wait until we implement it - I've created an issue MID-3347
>>>>>>>>>> <https://jira.evolveum.com/browse/MID-3347> for this.
>>>>>>>>>>
>>>>>>>>>> As for the first one (externally defined data source): Please see
>>>>>>>>>> https://wiki.evolveum.com/display/midPoint/Repository+Config
>>>>>>>>>> uration#RepositoryConfiguration-Datasourceconfiguration on how
>>>>>>>>>> to configure midPoint repository with the data source. This data source
>>>>>>>>>> will be used also by Quartz, if not overriden in <taskManager> section. It
>>>>>>>>>> should work but I don't remember if someone actually tested this.
>>>>>>>>>>
>>>>>>>>>> Concerning autoReconnect: I have no experiences with this
>>>>>>>>>> setting. I agree with you that about the hesitation of using it in
>>>>>>>>>> production environment. If really needed, I'd recommend to separate
>>>>>>>>>> midPoint repository configuration from Quartz configuration by using
>>>>>>>>>> different JDBC URLs for the two: the standard one for the repository and
>>>>>>>>>> the one with "autoReconnect=true" for Quartz. In this way, potential
>>>>>>>>>> negative effects should be restricted to task management functionality
>>>>>>>>>> only. But, overall, I'd suggest trying to eliminate the problem by setting
>>>>>>>>>> validationQuery first.
>>>>>>>>>>
>>>>>>>>>> Best regards,
>>>>>>>>>>
>>>>>>>>>> Pavol Mederly
>>>>>>>>>> Software developerevolveum.com
>>>>>>>>>>
>>>>>>>>>> On 14.08.2016 22:40, Florin. Stingaciu wrote:
>>>>>>>>>>
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> I'm trying to configure an active/active configuration, and
>>>>>>>>>> experiencing some issues with the Quartz scheduler. The SQL connection
>>>>>>>>>> seems to timeout quite often and result in many warning messages. I'm also
>>>>>>>>>> experiencing some errors -- as the timeout closes the connection, some
>>>>>>>>>> processes are still trying to commit using that stale handler.
>>>>>>>>>>
>>>>>>>>>> Here are some relevant logs from both midPoint instances in this
>>>>>>>>>> cluster: http://pastebin.com/JFjqrLnT http://pastebin.com/1v
>>>>>>>>>> y4iYPY
>>>>>>>>>>
>>>>>>>>>> Also here's the relevant portion of my config.xml file:
>>>>>>>>>> http://pastebin.com/WbydcCC8
>>>>>>>>>>
>>>>>>>>>> Following the suggestions in the warnings, I've started to look
>>>>>>>>>> at autoReconnect propriety of the JDBC connector and applied it to my
>>>>>>>>>> configs like so:
>>>>>>>>>>
>>>>>>>>>> <jdbcUrl>jdbc:mysql://SERVER:3306/midpoint_dev?autoReconnect
>>>>>>>>>>> =true</jdbcUrl>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Since implementing this change, the errors and warnings seem to
>>>>>>>>>> have disappeared. I will continue to monitor the logs and ensure this
>>>>>>>>>> actually the case.
>>>>>>>>>>
>>>>>>>>>> Reading the mysql docs, I found that this is not recommended as
>>>>>>>>>> this may cause data inconsistency issues and that stale connection
>>>>>>>>>> exceptions should be properly caught within the application. Namely:
>>>>>>>>>>
>>>>>>>>>> The use of this feature is not recommended, because it has side
>>>>>>>>>>> effects related to session state and data consistency when applications
>>>>>>>>>>> don't handle SQLExceptions properly, and is only designed to be used when
>>>>>>>>>>> you are unable to configure your application to handle SQLExceptions
>>>>>>>>>>> resulting from dead and stale connections properly.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Do you have any recommended configuration for this scenario? I'd
>>>>>>>>>> like to move my current production environment in an active active
>>>>>>>>>> configuration, however as of right now I'm hesitant to do so in order to
>>>>>>>>>> avoid any data corruption. Especially since it's quite difficult to test
>>>>>>>>>> for data consistency issues that may arise from using autoReconnect in my
>>>>>>>>>> dev environment.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> -F
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> midPoint mailing listmidPoint at lists.evolveum.comhttp://lists.evolveum.com/mailman/listinfo/midpoint
>>>>>>>>>>
>>>>>>>>>> _______________________________________________ midPoint mailing
>>>>>>>>>> list midPoint at lists.evolveum.com http://lists.evolveum.com/mail
>>>>>>>>>> man/listinfo/midpoint
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> midPoint mailing listmidPoint at lists.evolveum.comhttp://lists.evolveum.com/mailman/listinfo/midpoint
>>>>>>>>>
>>>>>>>>> _______________________________________________ midPoint mailing
>>>>>>>>> list midPoint at lists.evolveum.com http://lists.evolveum.com/mail
>>>>>>>>> man/listinfo/midpoint
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> midPoint mailing listmidPoint at lists.evolveum.comhttp://lists.evolveum.com/mailman/listinfo/midpoint
>>>>>>>>
>>>>>>>> _______________________________________________ midPoint mailing
>>>>>>>> list midPoint at lists.evolveum.com http://lists.evolveum.com/mail
>>>>>>>> man/listinfo/midpoint
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> midPoint mailing listmidPoint at lists.evolveum.comhttp://lists.evolveum.com/mailman/listinfo/midpoint
>>>>>>>
>>>>>>> _______________________________________________ midPoint mailing
>>>>>>> list midPoint at lists.evolveum.com http://lists.evolveum.com/mail
>>>>>>> man/listinfo/midpoint
>>>>>>
>>>>>> _______________________________________________
>>> midPoint mailing listmidPoint at lists.evolveum.comhttp://lists.evolveum.com/mailman/listinfo/midpoint
>>>
>>> _______________________________________________ midPoint mailing list
>>> midPoint at lists.evolveum.com http://lists.evolveum.com/mail
>>> man/listinfo/midpoint
>>
>> _______________________________________________
> midPoint mailing listmidPoint at lists.evolveum.comhttp://lists.evolveum.com/mailman/listinfo/midpoint
>
>
> _______________________________________________
> midPoint mailing list
> midPoint at lists.evolveum.com
> http://lists.evolveum.com/mailman/listinfo/midpoint
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20160829/9b738d90/attachment.htm>
More information about the midPoint
mailing list