[midPoint] [midpoint] Active/Active Configuration
Florin. Stingaciu
fstingaciu at mirantis.com
Tue Aug 30 19:21:38 CEST 2016
Hey Pavol,
Over night, I now have the same error messages I initially had:
http://pastebin.com/iWeSaswD
Thank you for all your help so far. Any further recommendations would be
really appreciated.
Thanks,
-F
On Mon, Aug 29, 2016 at 3:54 PM, Florin. Stingaciu <fstingaciu at mirantis.com>
wrote:
> Hey Pavol,
>
> I switched the repository to the configuration you mentioned, and I could
> now access the shadow page. However (minus the task manager config), this
> is the same configuration as I previously had where I ran into errors and
> then made this thread because of it. For now I'm not seeing any errors in
> the log files.
>
> I'll update this thread once I run into issues. As of right now, I'm still
> definitely not comfortable moving to active active considering all these
> issues. Have you guys seen similar issues before? Or do you believe I've
> just misconfigured something?
>
> Thanks,
> -F
>
>
>
> On Mon, Aug 29, 2016 at 3:39 PM, Pavol Mederly <mederly at evolveum.com>
> wrote:
>
>> Hello Florin,
>>
>> I've tried to use your setup in my environment (slightly modified), but I
>> was not able to replicate the exceptions. I've tried in particular to play
>> with the listing of shadow objects.
>>
>> So we perhaps have to continue diagnosing on your side.
>>
>> I suggest to try to isolate the problem. If you could use the following
>> config:
>>
>> <repository>
>> <embedded>false</embedded>
>> <repositoryServiceFactoryClass
>> >com.evolveum.midpoint.repo.sql.SqlRepositoryFactory</reposi
>> toryServiceFactoryClass>
>> <hibernateHbm2ddl>validate</hibernateHbm2ddl>
>> <database>mysql</database>
>> <jdbcUsername>...</jdbcUsername>
>> <jdbcPassword>...</jdbcPassword>
>> <jdbcUrl>jdbc:mysql://servder:3306/midpoint_dev</jdbcUrl>
>> </repository>
>> <taskManager>
>> <clustered>true</clustered>
>> <jmxUsername>midpoint</jmxUsername>
>> <jmxPassword>password</jmxPassword>
>> <dataSource>java:comp/env/jdbc/mysql</dataSource>
>> </taskManager>
>> <workflow>
>> <enabled>false</enabled>
>> </workflow>
>>
>> ...and try the testing. The configuration simply makes midPoint use its
>> own connection mechanism (c3p0) and Quartz use the tomcat's connection pool.
>>
>> Best regards,
>>
>> Pavol Mederly
>> Software developerevolveum.com
>>
>> On 29.08.2016 20:17, Florin. Stingaciu wrote:
>>
>> As an update, after having the instance up for half an hour or so with
>> the above config, I got the following errors again:
>> http://pastebin.com/tJJVFVsu
>>
>> Also, in the deployment, there's no workflows at all and very little user
>> load as it's just a dev env.
>>
>> Thanks,
>> -F
>>
>> On Mon, Aug 29, 2016 at 10:43 AM, Florin. Stingaciu <
>> fstingaciu at mirantis.com> wrote:
>>
>>> Hey Pavol,
>>>
>>> Currently, for Quartz and miPoint, I'm using a tomcat7 defined
>>> datasource with the following config:
>>>
>>> <Resource name="jdbc/mysql" auth="Container"
>>> type="javax.sql.DataSource"
>>> username="midpoint" password="password"
>>> url="jdbc:mysql://servder:3306/midpoint_dev"
>>> driverClassName="com.mysql.jdbc.Driver"
>>> accessToUnderlyingConnectionAllowed="true"
>>> initialSize="5" maxWait="5000"
>>> maxActive="30" maxIdle="5"
>>> validationQuery="select 1"
>>> readonly="false"
>>> poolPreparedStatements="true"/>
>>>
>>> And here's my relevant config.xml:
>>>
>>> <repository>
>>> <embedded>false</embedded>
>>> <repositoryServiceFactoryClass
>>> >com.evolveum.midpoint.repo.sql.SqlRepositoryFactory</reposi
>>> toryServiceFactoryClass>
>>> <hibernateHbm2ddl>validate</hibernateHbm2ddl>
>>> <hibernateDialect>com.evolveum
>>> .midpoint.repo.sql.util.MidPointMySQLDialect</hibernateDialect>
>>> <dataSource>java:comp/env/jdbc/mysql</dataSource>
>>> </repository>
>>> <taskManager>
>>> <clustered>true</clustered>
>>> <jmxUsername>midpoint</jmxUsername>
>>> <jmxPassword>password</jmxPassword>
>>> </taskManager>
>>>
>>> With this configuration, when I was trying to access the Shadows page, I
>>> got the errors mentioned in my previous message.
>>>
>>> Following your last email, I added the following
>>> <dataSource>java:comp/env/jdbc/mysql</dataSource> under <taskManager>.
>>> Now when I try to access the Shadows Page I get the following error:
>>> http://imgur.com/a/391CU. Please note there are four photos in that
>>> album. Also the reason, I chose an image is due to the fact that this error
>>> does not appear anywhere in the logs and it's in the GUI only.
>>>
>>> I would also like to add that even though this error occurs, midPoint
>>> still continues to function. Before I added the <dataSource> in the
>>> taskManager section, even though I was using a datasource for the
>>> Repository, when I tried to access the shadows page, midPoint would not
>>> work anymore (due to lack of mysql connections) until I restarted tomcat.
>>>
>>> The last point I want to make is that from what I've seen so far, this
>>> issue only happens when I try to access the shadow page. I don't see this
>>> issue while trying to access any other type of repository objects.
>>>
>>> Thanks,
>>> -F
>>>
>>>
>>> On Fri, Aug 26, 2016 at 3:41 PM, Pavol Mederly <mederly at evolveum.com>
>>> wrote:
>>>
>>>> Hello Florin,
>>>>
>>>> it seems that either midPoint (i.e. hibernate), Quartz scheduler, or
>>>> Activiti BPMS use connections without properly closing them.
>>>>
>>>> First step to diagnose it would be perhaps to setup things like this:
>>>>
>>>> 1. Let midPoint and Activiti use their own data source (by
>>>> configuring them with a JDBC URL): see Repository configuration
>>>> <https://wiki.evolveum.com/display/midPoint/Repository+Configuration>
>>>> and Workflow configuration
>>>> <https://wiki.evolveum.com/display/midPoint/Workflow+configuration>
>>>> wiki pages. (If you don't use approvals you can disable workflows
>>>> altogether by setting "enabled" to "false".)
>>>> 2. Keep Quartz schedule using tomcat data source: something like
>>>>
>>>> <taskManager>
>>>> <clustered>true</clustered>
>>>> <dataSource>...your data source name...</dataSource>
>>>> ...
>>>> </taskManager>
>>>>
>>>> And you'll see if the problem would appear again. (Maybe you could even
>>>> configure three Tomcat data sources; one for midPoint, second for Quartz,
>>>> and the third for Activiti, if needed. But I have not done this yet; these
>>>> are pure speculations of mine...)
>>>>
>>>> Best regards,
>>>>
>>>> Pavol Mederly
>>>> Software developerevolveum.com
>>>>
>>>> On 24.08.2016 22:41, Florin. Stingaciu wrote:
>>>>
>>>> I just rebuilt the latest snapshot (git-v3.5devel-221-g9744fa0) and am
>>>> experiencing the same problem.
>>>>
>>>> Thanks,
>>>> -F
>>>>
>>>>
>>>> On Wed, Aug 24, 2016 at 1:24 PM, Florin. Stingaciu <
>>>> fstingaciu at mirantis.com> wrote:
>>>>
>>>>> Upon further investigation I found that this error only occurs when I
>>>>> try to access the shadow repository via Configuration->Repository
>>>>> Objects->Shadows
>>>>>
>>>>> It seems that something triggers the creation of DB connections until
>>>>> it runs out of available connections. For this particular instance, I'm at
>>>>> git-v3.5devel-170-g26b0644.
>>>>>
>>>>> Thanks,
>>>>> -F
>>>>>
>>>>> On Wed, Aug 24, 2016 at 1:13 PM, Florin. Stingaciu <
>>>>> fstingaciu at mirantis.com> wrote:
>>>>>
>>>>>> Hello Pavol,
>>>>>>
>>>>>> After running this configuration my dev environment with two
>>>>>> instances up for a couple of days, I started to get the following error
>>>>>> messages: http://pastebin.com/nkxE9LNA
>>>>>>
>>>>>> It seems as though the connection pool is not properly cleaned up.
>>>>>> The database logs indicate no failure and in fact my production environment
>>>>>> is hooked up to the same DB instance and is working fine.
>>>>>>
>>>>>> Any recommendations would be very appreciated.
>>>>>>
>>>>>> Thanks,
>>>>>> -F
>>>>>>
>>>>>> On Mon, Aug 15, 2016 at 3:29 PM, Florin. Stingaciu <
>>>>>> fstingaciu at mirantis.com> wrote:
>>>>>>
>>>>>>> Awesome! Thanks for all your help today Pavol. I will try and test
>>>>>>> in my environment and let you know how that goes.
>>>>>>>
>>>>>>> Thanks again!
>>>>>>> -F
>>>>>>>
>>>>>>> On Mon, Aug 15, 2016 at 3:00 PM, Pavol Mederly <mederly at evolveum.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> Florin,
>>>>>>>>
>>>>>>>> I think I've fixed it: https://github.com/Evolveum/mi
>>>>>>>> dpoint/commit/c00f2979cea6f6ada0a1194b13bad51c0ecec2e3
>>>>>>>>
>>>>>>>> Cannot verify as the issue isn't visible on my computer, but I am
>>>>>>>> quite convinced it should be fixed now. The suspected cause is that some
>>>>>>>> query-only transactions weren't closed properly. C3P0 closes them
>>>>>>>> automatically but other connection providers probably not
>>>>>>>> <http://stackoverflow.com/questions/19396563/does-hibernates-session-close-automatically-rollback-uncommitted-transactions>
>>>>>>>> .
>>>>>>>>
>>>>>>>> You can safely use the master, or you can apply the above patch to
>>>>>>>> 3.4 yourself. The master is something like "3.4-support" these days
>>>>>>>> (preparing for 3.4.1), so it should be of production-level quality, or at
>>>>>>>> least near it.
>>>>>>>> (I'm going to run full test suite during the night, but I don't
>>>>>>>> expect any failures. Or you can check after approx 3 hours here
>>>>>>>> <https://travis-ci.org/Evolveum/midpoint>.)
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>>
>>>>>>>> Pavol Mederly
>>>>>>>> Software developerevolveum.com
>>>>>>>>
>>>>>>>> On 15.08.2016 22:57, Florin. Stingaciu wrote:
>>>>>>>>
>>>>>>>> Hey Pavol,
>>>>>>>>
>>>>>>>> I uploaded the log file you requested to my gdrive (
>>>>>>>> https://drive.google.com/file/d/0B9f4-vWBsztoZy1zWThGYnpGeG
>>>>>>>> s/view?usp=sharing) It's a trace level debug so the file is quite
>>>>>>>> large. Let me know if anything jumps out at you.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> -F
>>>>>>>>
>>>>>>>> On Mon, Aug 15, 2016 at 1:07 PM, Pavol Mederly <
>>>>>>>> mederly at evolveum.com> wrote:
>>>>>>>>
>>>>>>>>> Strange enough. I've replicated your setup on my computer (using
>>>>>>>>> PostgreSQL, as I have no MySQL currently installed), and it works: midPoint
>>>>>>>>> starts cleanly. I am able to add/delete repository objects.
>>>>>>>>>
>>>>>>>>> Attached are my config.xml and server.xml; they are pretty much
>>>>>>>>> the same as yours.
>>>>>>>>>
>>>>>>>>> So, I have no idea. You could try adding
>>>>>>>>> <database>mysql</database> into <repository> element in your config.xml,
>>>>>>>>> but it would most probably make no difference.
>>>>>>>>>
>>>>>>>>> As last resort, you could turn on the most detailed debugging for
>>>>>>>>> the repository on startup by adding a line like this
>>>>>>>>>
>>>>>>>>> <logger name="com.evolveum.midpoint.repo" level="TRACE" />
>>>>>>>>>
>>>>>>>>> into webapps\midpoint\WEB-INF\classes\logback.xml file and start
>>>>>>>>> midPoint. In idm.log there should be quite a detailed record of what's
>>>>>>>>> going on.
>>>>>>>>>
>>>>>>>>> Pavol Mederly
>>>>>>>>> Software developerevolveum.com
>>>>>>>>>
>>>>>>>>> On 15.08.2016 21:03, Florin. Stingaciu wrote:
>>>>>>>>>
>>>>>>>>> Here's my datasource config:
>>>>>>>>>
>>>>>>>>> <Resource name="jdbc/mysql" auth="Container"
>>>>>>>>> type="javax.sql.DataSource"
>>>>>>>>> username="midpoint" password="pass"
>>>>>>>>> url="jdbc:mysql://SERVER:3306/midpoint_dev"
>>>>>>>>> driverClassName="com.mysql.jdbc.Driver"
>>>>>>>>> accessToUnderlyingConnectionAllowed="true"
>>>>>>>>> initialSize="5" maxWait="5000"
>>>>>>>>> maxActive="30" maxIdle="5"
>>>>>>>>> validationQuery="select 1"
>>>>>>>>> poolPreparedStatements="true"/>
>>>>>>>>>
>>>>>>>>> And here's my config.xml:
>>>>>>>>>
>>>>>>>>> <repository>
>>>>>>>>> <embedded>false</embedded>
>>>>>>>>> <repositoryServiceFactoryClass
>>>>>>>>> >com.evolveum.midpoint.repo.sql.SqlRepositoryFactory</reposi
>>>>>>>>> toryServiceFactoryClass>
>>>>>>>>> <hibernateHbm2ddl>validate</hibernateHbm2ddl>
>>>>>>>>> <hibernateDialect>com.evolveum
>>>>>>>>> .midpoint.repo.sql.util.MidPointMySQLDialect</hibernateDialect>
>>>>>>>>> <dataSource>java:comp/env/jdbc/mysql</dataSource>
>>>>>>>>> </repository>
>>>>>>>>> <taskManager>
>>>>>>>>> <clustered>true</clustered>
>>>>>>>>> <jmxUsername>midpoint</jmxUsername>
>>>>>>>>> <jmxPassword>password</jmxPassword>
>>>>>>>>> </taskManager>
>>>>>>>>>
>>>>>>>>> On Mon, Aug 15, 2016 at 11:44 AM, Pavol Mederly <
>>>>>>>>> mederly at evolveum.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hello Florin,
>>>>>>>>>>
>>>>>>>>>> this is really interesting. Please, could you also share your
>>>>>>>>>> midPoint config.xml, as well as your data source configuration? (except
>>>>>>>>>> credentials, of course)
>>>>>>>>>>
>>>>>>>>>> Best regards,
>>>>>>>>>>
>>>>>>>>>> Pavol Mederly
>>>>>>>>>> Software developerevolveum.com
>>>>>>>>>>
>>>>>>>>>> On 15.08.2016 19:39, Florin. Stingaciu wrote:
>>>>>>>>>>
>>>>>>>>>> Hello Pavol,
>>>>>>>>>>
>>>>>>>>>> Thanks for your detailed response. I tried setting up the
>>>>>>>>>> datasource with validationQuery set up properly for the mySQL backed I
>>>>>>>>>> have. However, upon service restart I get the following errors:
>>>>>>>>>> http://pastebin.com/8dpGN0JC
>>>>>>>>>>
>>>>>>>>>> To save you a click, it seems as though the connection is set up
>>>>>>>>>> in readonly mode or some other strange things happen. I've tried setting
>>>>>>>>>> readonly="false" in the resource definition in server.xml but
>>>>>>>>>> that didn't help. I will continue researching this, however any guidance
>>>>>>>>>> would be quite appreciated.
>>>>>>>>>>
>>>>>>>>>> Thanks!
>>>>>>>>>> -F
>>>>>>>>>>
>>>>>>>>>> On Mon, Aug 15, 2016 at 2:29 AM, Pavol Mederly <
>>>>>>>>>> mederly at evolveum.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hello Florin,
>>>>>>>>>>>
>>>>>>>>>>> having looked at your logs, it seems that maybe explicit setting
>>>>>>>>>>> of validationQuery in Quartz data source setup would help. (See
>>>>>>>>>>> http://www.quartz-scheduler.org/documentation/quartz-2.x/con
>>>>>>>>>>> figuration/ConfigDataSources.html.)
>>>>>>>>>>>
>>>>>>>>>>> Unfortunately, current midPoint implementation does not allow to
>>>>>>>>>>> configure Quartz data source parameters. So, there are the following three
>>>>>>>>>>> possibilities:
>>>>>>>>>>>
>>>>>>>>>>> 1. Take an alternative route, and use application
>>>>>>>>>>> server-defined data source (with validationQuery set up).
>>>>>>>>>>> 2. Patch Task Manager implementation by adding appropriate
>>>>>>>>>>> lines to Quartz configuration (see LocalNodeManager.java:87-90).
>>>>>>>>>>> 3. Wait until we implement it - I've created an issue
>>>>>>>>>>> MID-3347 <https://jira.evolveum.com/browse/MID-3347> for
>>>>>>>>>>> this.
>>>>>>>>>>>
>>>>>>>>>>> As for the first one (externally defined data source): Please
>>>>>>>>>>> see https://wiki.evolveum.com/display/midPoint/Repository+Config
>>>>>>>>>>> uration#RepositoryConfiguration-Datasourceconfiguration on how
>>>>>>>>>>> to configure midPoint repository with the data source. This data source
>>>>>>>>>>> will be used also by Quartz, if not overriden in <taskManager> section. It
>>>>>>>>>>> should work but I don't remember if someone actually tested this.
>>>>>>>>>>>
>>>>>>>>>>> Concerning autoReconnect: I have no experiences with this
>>>>>>>>>>> setting. I agree with you that about the hesitation of using it in
>>>>>>>>>>> production environment. If really needed, I'd recommend to separate
>>>>>>>>>>> midPoint repository configuration from Quartz configuration by using
>>>>>>>>>>> different JDBC URLs for the two: the standard one for the repository and
>>>>>>>>>>> the one with "autoReconnect=true" for Quartz. In this way, potential
>>>>>>>>>>> negative effects should be restricted to task management functionality
>>>>>>>>>>> only. But, overall, I'd suggest trying to eliminate the problem by setting
>>>>>>>>>>> validationQuery first.
>>>>>>>>>>>
>>>>>>>>>>> Best regards,
>>>>>>>>>>>
>>>>>>>>>>> Pavol Mederly
>>>>>>>>>>> Software developerevolveum.com
>>>>>>>>>>>
>>>>>>>>>>> On 14.08.2016 22:40, Florin. Stingaciu wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hello,
>>>>>>>>>>>
>>>>>>>>>>> I'm trying to configure an active/active configuration, and
>>>>>>>>>>> experiencing some issues with the Quartz scheduler. The SQL connection
>>>>>>>>>>> seems to timeout quite often and result in many warning messages. I'm also
>>>>>>>>>>> experiencing some errors -- as the timeout closes the connection, some
>>>>>>>>>>> processes are still trying to commit using that stale handler.
>>>>>>>>>>>
>>>>>>>>>>> Here are some relevant logs from both midPoint instances in this
>>>>>>>>>>> cluster: http://pastebin.com/JFjqrLnT http://pastebin.com/1v
>>>>>>>>>>> y4iYPY
>>>>>>>>>>>
>>>>>>>>>>> Also here's the relevant portion of my config.xml file:
>>>>>>>>>>> http://pastebin.com/WbydcCC8
>>>>>>>>>>>
>>>>>>>>>>> Following the suggestions in the warnings, I've started to look
>>>>>>>>>>> at autoReconnect propriety of the JDBC connector and applied it to my
>>>>>>>>>>> configs like so:
>>>>>>>>>>>
>>>>>>>>>>> <jdbcUrl>jdbc:mysql://SERVER:3306/midpoint_dev?autoReconnect
>>>>>>>>>>>> =true</jdbcUrl>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Since implementing this change, the errors and warnings seem to
>>>>>>>>>>> have disappeared. I will continue to monitor the logs and ensure this
>>>>>>>>>>> actually the case.
>>>>>>>>>>>
>>>>>>>>>>> Reading the mysql docs, I found that this is not recommended as
>>>>>>>>>>> this may cause data inconsistency issues and that stale connection
>>>>>>>>>>> exceptions should be properly caught within the application. Namely:
>>>>>>>>>>>
>>>>>>>>>>> The use of this feature is not recommended, because it has side
>>>>>>>>>>>> effects related to session state and data consistency when applications
>>>>>>>>>>>> don't handle SQLExceptions properly, and is only designed to be used when
>>>>>>>>>>>> you are unable to configure your application to handle SQLExceptions
>>>>>>>>>>>> resulting from dead and stale connections properly.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Do you have any recommended configuration for this scenario? I'd
>>>>>>>>>>> like to move my current production environment in an active active
>>>>>>>>>>> configuration, however as of right now I'm hesitant to do so in order to
>>>>>>>>>>> avoid any data corruption. Especially since it's quite difficult to test
>>>>>>>>>>> for data consistency issues that may arise from using autoReconnect in my
>>>>>>>>>>> dev environment.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> -F
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> midPoint mailing listmidPoint at lists.evolveum.comhttp://lists.evolveum.com/mailman/listinfo/midpoint
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________ midPoint
>>>>>>>>>>> mailing list midPoint at lists.evolveum.com
>>>>>>>>>>> http://lists.evolveum.com/mailman/listinfo/midpoint
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> midPoint mailing listmidPoint at lists.evolveum.comhttp://lists.evolveum.com/mailman/listinfo/midpoint
>>>>>>>>>>
>>>>>>>>>> _______________________________________________ midPoint mailing
>>>>>>>>>> list midPoint at lists.evolveum.com http://lists.evolveum.com/mail
>>>>>>>>>> man/listinfo/midpoint
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> midPoint mailing listmidPoint at lists.evolveum.comhttp://lists.evolveum.com/mailman/listinfo/midpoint
>>>>>>>>>
>>>>>>>>> _______________________________________________ midPoint mailing
>>>>>>>>> list midPoint at lists.evolveum.com http://lists.evolveum.com/mail
>>>>>>>>> man/listinfo/midpoint
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> midPoint mailing listmidPoint at lists.evolveum.comhttp://lists.evolveum.com/mailman/listinfo/midpoint
>>>>>>>>
>>>>>>>> _______________________________________________ midPoint mailing
>>>>>>>> list midPoint at lists.evolveum.com http://lists.evolveum.com/mail
>>>>>>>> man/listinfo/midpoint
>>>>>>>
>>>>>>> _______________________________________________
>>>> midPoint mailing listmidPoint at lists.evolveum.comhttp://lists.evolveum.com/mailman/listinfo/midpoint
>>>>
>>>> _______________________________________________ midPoint mailing list
>>>> midPoint at lists.evolveum.com http://lists.evolveum.com/mail
>>>> man/listinfo/midpoint
>>>
>>> _______________________________________________
>> midPoint mailing listmidPoint at lists.evolveum.comhttp://lists.evolveum.com/mailman/listinfo/midpoint
>>
>>
>> _______________________________________________
>> midPoint mailing list
>> midPoint at lists.evolveum.com
>> http://lists.evolveum.com/mailman/listinfo/midpoint
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20160830/5c1e5684/attachment.htm>
More information about the midPoint
mailing list