[midPoint] [midpoint] Active/Active Configuration
Florin. Stingaciu
fstingaciu at mirantis.com
Wed Aug 24 22:24:41 CEST 2016
Upon further investigation I found that this error only occurs when I try
to access the shadow repository via Configuration->Repository
Objects->Shadows
It seems that something triggers the creation of DB connections until it
runs out of available connections. For this particular instance, I'm at
git-v3.5devel-170-g26b0644.
Thanks,
-F
On Wed, Aug 24, 2016 at 1:13 PM, Florin. Stingaciu <fstingaciu at mirantis.com>
wrote:
> Hello Pavol,
>
> After running this configuration my dev environment with two instances up
> for a couple of days, I started to get the following error messages:
> http://pastebin.com/nkxE9LNA
>
> It seems as though the connection pool is not properly cleaned up. The
> database logs indicate no failure and in fact my production environment is
> hooked up to the same DB instance and is working fine.
>
> Any recommendations would be very appreciated.
>
> Thanks,
> -F
>
> On Mon, Aug 15, 2016 at 3:29 PM, Florin. Stingaciu <
> fstingaciu at mirantis.com> wrote:
>
>> Awesome! Thanks for all your help today Pavol. I will try and test in my
>> environment and let you know how that goes.
>>
>> Thanks again!
>> -F
>>
>> On Mon, Aug 15, 2016 at 3:00 PM, Pavol Mederly <mederly at evolveum.com>
>> wrote:
>>
>>> Florin,
>>>
>>> I think I've fixed it: https://github.com/Evolveum/mi
>>> dpoint/commit/c00f2979cea6f6ada0a1194b13bad51c0ecec2e3
>>>
>>> Cannot verify as the issue isn't visible on my computer, but I am quite
>>> convinced it should be fixed now. The suspected cause is that some
>>> query-only transactions weren't closed properly. C3P0 closes them
>>> automatically but other connection providers probably not
>>> <http://stackoverflow.com/questions/19396563/does-hibernates-session-close-automatically-rollback-uncommitted-transactions>
>>> .
>>>
>>> You can safely use the master, or you can apply the above patch to 3.4
>>> yourself. The master is something like "3.4-support" these days (preparing
>>> for 3.4.1), so it should be of production-level quality, or at least near
>>> it.
>>> (I'm going to run full test suite during the night, but I don't expect
>>> any failures. Or you can check after approx 3 hours here
>>> <https://travis-ci.org/Evolveum/midpoint>.)
>>>
>>> Best regards,
>>>
>>> Pavol Mederly
>>> Software developerevolveum.com
>>>
>>> On 15.08.2016 22:57, Florin. Stingaciu wrote:
>>>
>>> Hey Pavol,
>>>
>>> I uploaded the log file you requested to my gdrive (
>>> https://drive.google.com/file/d/0B9f4-vWBsztoZy1zWThGYnpGeG
>>> s/view?usp=sharing) It's a trace level debug so the file is quite
>>> large. Let me know if anything jumps out at you.
>>>
>>> Thanks,
>>> -F
>>>
>>> On Mon, Aug 15, 2016 at 1:07 PM, Pavol Mederly <mederly at evolveum.com>
>>> wrote:
>>>
>>>> Strange enough. I've replicated your setup on my computer (using
>>>> PostgreSQL, as I have no MySQL currently installed), and it works: midPoint
>>>> starts cleanly. I am able to add/delete repository objects.
>>>>
>>>> Attached are my config.xml and server.xml; they are pretty much the
>>>> same as yours.
>>>>
>>>> So, I have no idea. You could try adding <database>mysql</database>
>>>> into <repository> element in your config.xml, but it would most probably
>>>> make no difference.
>>>>
>>>> As last resort, you could turn on the most detailed debugging for the
>>>> repository on startup by adding a line like this
>>>>
>>>> <logger name="com.evolveum.midpoint.repo" level="TRACE" />
>>>>
>>>> into webapps\midpoint\WEB-INF\classes\logback.xml file and start
>>>> midPoint. In idm.log there should be quite a detailed record of what's
>>>> going on.
>>>>
>>>> Pavol Mederly
>>>> Software developerevolveum.com
>>>>
>>>> On 15.08.2016 21:03, Florin. Stingaciu wrote:
>>>>
>>>> Here's my datasource config:
>>>>
>>>> <Resource name="jdbc/mysql" auth="Container"
>>>> type="javax.sql.DataSource"
>>>> username="midpoint" password="pass"
>>>> url="jdbc:mysql://SERVER:3306/midpoint_dev"
>>>> driverClassName="com.mysql.jdbc.Driver"
>>>> accessToUnderlyingConnectionAllowed="true"
>>>> initialSize="5" maxWait="5000"
>>>> maxActive="30" maxIdle="5"
>>>> validationQuery="select 1"
>>>> poolPreparedStatements="true"/>
>>>>
>>>> And here's my config.xml:
>>>>
>>>> <repository>
>>>> <embedded>false</embedded>
>>>> <repositoryServiceFactoryClass
>>>> >com.evolveum.midpoint.repo.sql.SqlRepositoryFactory</reposi
>>>> toryServiceFactoryClass>
>>>> <hibernateHbm2ddl>validate</hibernateHbm2ddl>
>>>> <hibernateDialect>com.evolveum
>>>> .midpoint.repo.sql.util.MidPointMySQLDialect</hibernateDialect>
>>>> <dataSource>java:comp/env/jdbc/mysql</dataSource>
>>>> </repository>
>>>> <taskManager>
>>>> <clustered>true</clustered>
>>>> <jmxUsername>midpoint</jmxUsername>
>>>> <jmxPassword>password</jmxPassword>
>>>> </taskManager>
>>>>
>>>> On Mon, Aug 15, 2016 at 11:44 AM, Pavol Mederly <mederly at evolveum.com>
>>>> wrote:
>>>>
>>>>> Hello Florin,
>>>>>
>>>>> this is really interesting. Please, could you also share your midPoint
>>>>> config.xml, as well as your data source configuration? (except credentials,
>>>>> of course)
>>>>>
>>>>> Best regards,
>>>>>
>>>>> Pavol Mederly
>>>>> Software developerevolveum.com
>>>>>
>>>>> On 15.08.2016 19:39, Florin. Stingaciu wrote:
>>>>>
>>>>> Hello Pavol,
>>>>>
>>>>> Thanks for your detailed response. I tried setting up the datasource
>>>>> with validationQuery set up properly for the mySQL backed I have. However,
>>>>> upon service restart I get the following errors:
>>>>> http://pastebin.com/8dpGN0JC
>>>>>
>>>>> To save you a click, it seems as though the connection is set up in
>>>>> readonly mode or some other strange things happen. I've tried setting
>>>>> readonly="false" in the resource definition in server.xml but that
>>>>> didn't help. I will continue researching this, however any guidance would
>>>>> be quite appreciated.
>>>>>
>>>>> Thanks!
>>>>> -F
>>>>>
>>>>> On Mon, Aug 15, 2016 at 2:29 AM, Pavol Mederly <mederly at evolveum.com>
>>>>> wrote:
>>>>>
>>>>>> Hello Florin,
>>>>>>
>>>>>> having looked at your logs, it seems that maybe explicit setting of
>>>>>> validationQuery in Quartz data source setup would help. (See
>>>>>> http://www.quartz-scheduler.org/documentation/quartz-2.x/con
>>>>>> figuration/ConfigDataSources.html.)
>>>>>>
>>>>>> Unfortunately, current midPoint implementation does not allow to
>>>>>> configure Quartz data source parameters. So, there are the following three
>>>>>> possibilities:
>>>>>>
>>>>>> 1. Take an alternative route, and use application server-defined
>>>>>> data source (with validationQuery set up).
>>>>>> 2. Patch Task Manager implementation by adding appropriate lines
>>>>>> to Quartz configuration (see LocalNodeManager.java:87-90).
>>>>>> 3. Wait until we implement it - I've created an issue MID-3347
>>>>>> <https://jira.evolveum.com/browse/MID-3347> for this.
>>>>>>
>>>>>> As for the first one (externally defined data source): Please see
>>>>>> https://wiki.evolveum.com/display/midPoint/Repository+Config
>>>>>> uration#RepositoryConfiguration-Datasourceconfiguration on how to
>>>>>> configure midPoint repository with the data source. This data source will
>>>>>> be used also by Quartz, if not overriden in <taskManager> section. It
>>>>>> should work but I don't remember if someone actually tested this.
>>>>>>
>>>>>> Concerning autoReconnect: I have no experiences with this setting. I
>>>>>> agree with you that about the hesitation of using it in production
>>>>>> environment. If really needed, I'd recommend to separate midPoint
>>>>>> repository configuration from Quartz configuration by using different JDBC
>>>>>> URLs for the two: the standard one for the repository and the one with
>>>>>> "autoReconnect=true" for Quartz. In this way, potential negative effects
>>>>>> should be restricted to task management functionality only. But, overall,
>>>>>> I'd suggest trying to eliminate the problem by setting validationQuery
>>>>>> first.
>>>>>>
>>>>>> Best regards,
>>>>>>
>>>>>> Pavol Mederly
>>>>>> Software developerevolveum.com
>>>>>>
>>>>>> On 14.08.2016 22:40, Florin. Stingaciu wrote:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I'm trying to configure an active/active configuration, and
>>>>>> experiencing some issues with the Quartz scheduler. The SQL connection
>>>>>> seems to timeout quite often and result in many warning messages. I'm also
>>>>>> experiencing some errors -- as the timeout closes the connection, some
>>>>>> processes are still trying to commit using that stale handler.
>>>>>>
>>>>>> Here are some relevant logs from both midPoint instances in this
>>>>>> cluster: http://pastebin.com/JFjqrLnT http://pastebin.com/1vy4iYPY
>>>>>>
>>>>>> Also here's the relevant portion of my config.xml file:
>>>>>> http://pastebin.com/WbydcCC8
>>>>>>
>>>>>> Following the suggestions in the warnings, I've started to look at
>>>>>> autoReconnect propriety of the JDBC connector and applied it to my configs
>>>>>> like so:
>>>>>>
>>>>>> <jdbcUrl>jdbc:mysql://SERVER:3306/midpoint_dev?autoReconnect
>>>>>>> =true</jdbcUrl>
>>>>>>
>>>>>>
>>>>>> Since implementing this change, the errors and warnings seem to have
>>>>>> disappeared. I will continue to monitor the logs and ensure this actually
>>>>>> the case.
>>>>>>
>>>>>> Reading the mysql docs, I found that this is not recommended as this
>>>>>> may cause data inconsistency issues and that stale connection exceptions
>>>>>> should be properly caught within the application. Namely:
>>>>>>
>>>>>> The use of this feature is not recommended, because it has side
>>>>>>> effects related to session state and data consistency when applications
>>>>>>> don't handle SQLExceptions properly, and is only designed to be used when
>>>>>>> you are unable to configure your application to handle SQLExceptions
>>>>>>> resulting from dead and stale connections properly.
>>>>>>
>>>>>>
>>>>>> Do you have any recommended configuration for this scenario? I'd like
>>>>>> to move my current production environment in an active active
>>>>>> configuration, however as of right now I'm hesitant to do so in order to
>>>>>> avoid any data corruption. Especially since it's quite difficult to test
>>>>>> for data consistency issues that may arise from using autoReconnect in my
>>>>>> dev environment.
>>>>>>
>>>>>> Thanks,
>>>>>> -F
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> midPoint mailing listmidPoint at lists.evolveum.comhttp://lists.evolveum.com/mailman/listinfo/midpoint
>>>>>>
>>>>>> _______________________________________________ midPoint mailing
>>>>>> list midPoint at lists.evolveum.com http://lists.evolveum.com/mail
>>>>>> man/listinfo/midpoint
>>>>>
>>>>> _______________________________________________
>>>>> midPoint mailing listmidPoint at lists.evolveum.comhttp://lists.evolveum.com/mailman/listinfo/midpoint
>>>>>
>>>>> _______________________________________________ midPoint mailing list
>>>>> midPoint at lists.evolveum.com http://lists.evolveum.com/mail
>>>>> man/listinfo/midpoint
>>>>
>>>> _______________________________________________
>>>> midPoint mailing listmidPoint at lists.evolveum.comhttp://lists.evolveum.com/mailman/listinfo/midpoint
>>>>
>>>> _______________________________________________ midPoint mailing list
>>>> midPoint at lists.evolveum.com http://lists.evolveum.com/mail
>>>> man/listinfo/midpoint
>>>
>>> _______________________________________________
>>> midPoint mailing listmidPoint at lists.evolveum.comhttp://lists.evolveum.com/mailman/listinfo/midpoint
>>>
>>>
>>> _______________________________________________
>>> midPoint mailing list
>>> midPoint at lists.evolveum.com
>>> http://lists.evolveum.com/mailman/listinfo/midpoint
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20160824/158d55ca/attachment.htm>
More information about the midPoint
mailing list