[midPoint] [midpoint] Active/Active Configuration

Florin. Stingaciu fstingaciu at mirantis.com
Wed Aug 24 22:41:33 CEST 2016


I just rebuilt the latest snapshot (git-v3.5devel-221-g9744fa0) and am
experiencing the same problem.

Thanks,
-F


On Wed, Aug 24, 2016 at 1:24 PM, Florin. Stingaciu <fstingaciu at mirantis.com>
wrote:

> Upon further investigation I found that this error only occurs when I try
> to access the shadow repository via Configuration->Repository
> Objects->Shadows
>
> It seems that something triggers the creation of DB connections until it
> runs out of available connections. For this particular instance, I'm at
> git-v3.5devel-170-g26b0644.
>
> Thanks,
> -F
>
> On Wed, Aug 24, 2016 at 1:13 PM, Florin. Stingaciu <
> fstingaciu at mirantis.com> wrote:
>
>> Hello Pavol,
>>
>> After running this configuration my dev environment with two instances up
>> for a couple of days, I started to get the following error messages:
>> http://pastebin.com/nkxE9LNA
>>
>> It seems as though the connection pool is not properly cleaned up. The
>> database logs indicate no failure and in fact my production environment is
>> hooked up to the same DB instance and is working fine.
>>
>> Any recommendations would be very appreciated.
>>
>> Thanks,
>> -F
>>
>> On Mon, Aug 15, 2016 at 3:29 PM, Florin. Stingaciu <
>> fstingaciu at mirantis.com> wrote:
>>
>>> Awesome! Thanks for all your help today Pavol. I will try and test in my
>>> environment and let you know how that goes.
>>>
>>> Thanks again!
>>> -F
>>>
>>> On Mon, Aug 15, 2016 at 3:00 PM, Pavol Mederly <mederly at evolveum.com>
>>> wrote:
>>>
>>>> Florin,
>>>>
>>>> I think I've fixed it: https://github.com/Evolveum/mi
>>>> dpoint/commit/c00f2979cea6f6ada0a1194b13bad51c0ecec2e3
>>>>
>>>> Cannot verify as the issue isn't visible on my computer, but I am quite
>>>> convinced it should be fixed now. The suspected cause is that some
>>>> query-only transactions weren't closed properly. C3P0 closes them
>>>> automatically but other connection providers probably not
>>>> <http://stackoverflow.com/questions/19396563/does-hibernates-session-close-automatically-rollback-uncommitted-transactions>
>>>> .
>>>>
>>>> You can safely use the master, or you can apply the above patch to 3.4
>>>> yourself. The master is something like "3.4-support" these days (preparing
>>>> for 3.4.1), so it should be of production-level quality, or at least near
>>>> it.
>>>> (I'm going to run full test suite during the night, but I don't expect
>>>> any failures. Or you can check after approx 3 hours here
>>>> <https://travis-ci.org/Evolveum/midpoint>.)
>>>>
>>>> Best regards,
>>>>
>>>> Pavol Mederly
>>>> Software developerevolveum.com
>>>>
>>>> On 15.08.2016 22:57, Florin. Stingaciu wrote:
>>>>
>>>> Hey Pavol,
>>>>
>>>> I uploaded the log file you requested to my gdrive (
>>>> https://drive.google.com/file/d/0B9f4-vWBsztoZy1zWThGYnpGeG
>>>> s/view?usp=sharing) It's a trace level debug so the file is quite
>>>> large. Let me know if anything jumps out at you.
>>>>
>>>> Thanks,
>>>> -F
>>>>
>>>> On Mon, Aug 15, 2016 at 1:07 PM, Pavol Mederly <mederly at evolveum.com>
>>>> wrote:
>>>>
>>>>> Strange enough. I've replicated your setup on my computer (using
>>>>> PostgreSQL, as I have no MySQL currently installed), and it works: midPoint
>>>>> starts cleanly. I am able to add/delete repository objects.
>>>>>
>>>>> Attached are my config.xml and server.xml; they are pretty much the
>>>>> same as yours.
>>>>>
>>>>> So, I have no idea. You could try adding <database>mysql</database>
>>>>> into <repository> element in your config.xml, but it would most probably
>>>>> make no difference.
>>>>>
>>>>> As last resort, you could turn on the most detailed debugging for the
>>>>> repository on startup by adding a line like this
>>>>>
>>>>> <logger name="com.evolveum.midpoint.repo" level="TRACE" />
>>>>>
>>>>> into webapps\midpoint\WEB-INF\classes\logback.xml file and start
>>>>> midPoint. In idm.log there should be quite a detailed record of what's
>>>>> going on.
>>>>>
>>>>> Pavol Mederly
>>>>> Software developerevolveum.com
>>>>>
>>>>> On 15.08.2016 21:03, Florin. Stingaciu wrote:
>>>>>
>>>>> Here's my datasource config:
>>>>>
>>>>>     <Resource name="jdbc/mysql" auth="Container"
>>>>> type="javax.sql.DataSource"
>>>>>         username="midpoint" password="pass"
>>>>>         url="jdbc:mysql://SERVER:3306/midpoint_dev"
>>>>>         driverClassName="com.mysql.jdbc.Driver"
>>>>>         accessToUnderlyingConnectionAllowed="true"
>>>>>         initialSize="5" maxWait="5000"
>>>>>         maxActive="30" maxIdle="5"
>>>>>         validationQuery="select 1"
>>>>>         poolPreparedStatements="true"/>
>>>>>
>>>>> And here's my config.xml:
>>>>>
>>>>>         <repository>
>>>>>                 <embedded>false</embedded>
>>>>>                 <repositoryServiceFactoryClass
>>>>> >com.evolveum.midpoint.repo.sql.SqlRepositoryFactory</reposi
>>>>> toryServiceFactoryClass>
>>>>>                 <hibernateHbm2ddl>validate</hibernateHbm2ddl>
>>>>>                 <hibernateDialect>com.evolveum
>>>>> .midpoint.repo.sql.util.MidPointMySQLDialect</hibernateDialect>
>>>>>                 <dataSource>java:comp/env/jdbc/mysql</dataSource>
>>>>>         </repository>
>>>>>         <taskManager>
>>>>>                 <clustered>true</clustered>
>>>>>                 <jmxUsername>midpoint</jmxUsername>
>>>>>                 <jmxPassword>password</jmxPassword>
>>>>>         </taskManager>
>>>>>
>>>>> On Mon, Aug 15, 2016 at 11:44 AM, Pavol Mederly <mederly at evolveum.com>
>>>>> wrote:
>>>>>
>>>>>> Hello Florin,
>>>>>>
>>>>>> this is really interesting. Please, could you also share your
>>>>>> midPoint config.xml, as well as your data source configuration? (except
>>>>>> credentials, of course)
>>>>>>
>>>>>> Best regards,
>>>>>>
>>>>>> Pavol Mederly
>>>>>> Software developerevolveum.com
>>>>>>
>>>>>> On 15.08.2016 19:39, Florin. Stingaciu wrote:
>>>>>>
>>>>>> Hello Pavol,
>>>>>>
>>>>>> Thanks for your detailed response. I tried setting up the datasource
>>>>>> with validationQuery set up properly for the mySQL backed I have. However,
>>>>>> upon service restart I get the following errors:
>>>>>> http://pastebin.com/8dpGN0JC
>>>>>>
>>>>>> To save you a click, it seems as though the connection is set up in
>>>>>> readonly mode or some other strange things happen. I've tried setting
>>>>>> readonly="false" in the resource definition in server.xml but that
>>>>>> didn't help. I will continue researching this, however any guidance would
>>>>>> be quite appreciated.
>>>>>>
>>>>>> Thanks!
>>>>>> -F
>>>>>>
>>>>>> On Mon, Aug 15, 2016 at 2:29 AM, Pavol Mederly <mederly at evolveum.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hello Florin,
>>>>>>>
>>>>>>> having looked at your logs, it seems that maybe explicit setting of
>>>>>>> validationQuery in Quartz data source setup would help. (See
>>>>>>> http://www.quartz-scheduler.org/documentation/quartz-2.x/con
>>>>>>> figuration/ConfigDataSources.html.)
>>>>>>>
>>>>>>> Unfortunately, current midPoint implementation does not allow to
>>>>>>> configure Quartz data source parameters. So, there are the following three
>>>>>>> possibilities:
>>>>>>>
>>>>>>>    1. Take an alternative route, and use application server-defined
>>>>>>>    data source (with validationQuery set up).
>>>>>>>    2. Patch Task Manager implementation by adding appropriate lines
>>>>>>>    to Quartz configuration (see LocalNodeManager.java:87-90).
>>>>>>>    3. Wait until we implement it - I've created an issue MID-3347
>>>>>>>    <https://jira.evolveum.com/browse/MID-3347> for this.
>>>>>>>
>>>>>>> As for the first one (externally defined data source): Please see
>>>>>>> https://wiki.evolveum.com/display/midPoint/Repository+Config
>>>>>>> uration#RepositoryConfiguration-Datasourceconfiguration on how to
>>>>>>> configure midPoint repository with the data source. This data source will
>>>>>>> be used also by Quartz, if not overriden in <taskManager> section. It
>>>>>>> should work but I don't remember if someone actually tested this.
>>>>>>>
>>>>>>> Concerning autoReconnect: I have no experiences with this setting. I
>>>>>>> agree with you that about the hesitation of using it in production
>>>>>>> environment. If really needed, I'd recommend to separate midPoint
>>>>>>> repository configuration from Quartz configuration by using different JDBC
>>>>>>> URLs for the two: the standard one for the repository and the one with
>>>>>>> "autoReconnect=true" for Quartz. In this way, potential negative effects
>>>>>>> should be restricted to task management functionality only. But, overall,
>>>>>>> I'd suggest trying to eliminate the problem by setting validationQuery
>>>>>>> first.
>>>>>>>
>>>>>>> Best regards,
>>>>>>>
>>>>>>> Pavol Mederly
>>>>>>> Software developerevolveum.com
>>>>>>>
>>>>>>> On 14.08.2016 22:40, Florin. Stingaciu wrote:
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> I'm trying to configure an active/active configuration, and
>>>>>>> experiencing some issues with the Quartz scheduler. The SQL connection
>>>>>>> seems to timeout quite often and result in many warning messages. I'm also
>>>>>>> experiencing some errors -- as the timeout closes the connection, some
>>>>>>> processes are still trying to commit using that stale handler.
>>>>>>>
>>>>>>> Here are some relevant logs from both midPoint instances in this
>>>>>>> cluster: http://pastebin.com/JFjqrLnT http://pastebin.com/1vy4iYPY
>>>>>>>
>>>>>>> Also here's the relevant portion of my config.xml file:
>>>>>>> http://pastebin.com/WbydcCC8
>>>>>>>
>>>>>>> Following the suggestions in the warnings, I've started to look at
>>>>>>> autoReconnect propriety of the JDBC connector and applied it to my configs
>>>>>>> like so:
>>>>>>>
>>>>>>> <jdbcUrl>jdbc:mysql://SERVER:3306/midpoint_dev?autoReconnect
>>>>>>>> =true</jdbcUrl>
>>>>>>>
>>>>>>>
>>>>>>> Since implementing this change, the errors and warnings seem to have
>>>>>>> disappeared. I will continue to monitor the logs and ensure this actually
>>>>>>> the case.
>>>>>>>
>>>>>>> Reading the mysql docs, I found that this is not recommended as this
>>>>>>> may cause data inconsistency issues and that stale connection exceptions
>>>>>>> should be properly caught within the application. Namely:
>>>>>>>
>>>>>>> The use of this feature is not recommended, because it has side
>>>>>>>> effects related to session state and data consistency when applications
>>>>>>>> don't handle SQLExceptions properly, and is only designed to be used when
>>>>>>>> you are unable to configure your application to handle SQLExceptions
>>>>>>>> resulting from dead and stale connections properly.
>>>>>>>
>>>>>>>
>>>>>>> Do you have any recommended configuration for this scenario? I'd
>>>>>>> like to move my current production environment in an active active
>>>>>>> configuration, however as of right now I'm hesitant to do so in order to
>>>>>>> avoid any data corruption. Especially since it's quite difficult to test
>>>>>>> for data consistency issues that may arise from using autoReconnect in my
>>>>>>> dev environment.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> -F
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> midPoint mailing listmidPoint at lists.evolveum.comhttp://lists.evolveum.com/mailman/listinfo/midpoint
>>>>>>>
>>>>>>> _______________________________________________ midPoint mailing
>>>>>>> list midPoint at lists.evolveum.com http://lists.evolveum.com/mail
>>>>>>> man/listinfo/midpoint
>>>>>>
>>>>>> _______________________________________________
>>>>>> midPoint mailing listmidPoint at lists.evolveum.comhttp://lists.evolveum.com/mailman/listinfo/midpoint
>>>>>>
>>>>>> _______________________________________________ midPoint mailing
>>>>>> list midPoint at lists.evolveum.com http://lists.evolveum.com/mail
>>>>>> man/listinfo/midpoint
>>>>>
>>>>> _______________________________________________
>>>>> midPoint mailing listmidPoint at lists.evolveum.comhttp://lists.evolveum.com/mailman/listinfo/midpoint
>>>>>
>>>>> _______________________________________________ midPoint mailing list
>>>>> midPoint at lists.evolveum.com http://lists.evolveum.com/mail
>>>>> man/listinfo/midpoint
>>>>
>>>> _______________________________________________
>>>> midPoint mailing listmidPoint at lists.evolveum.comhttp://lists.evolveum.com/mailman/listinfo/midpoint
>>>>
>>>>
>>>> _______________________________________________
>>>> midPoint mailing list
>>>> midPoint at lists.evolveum.com
>>>> http://lists.evolveum.com/mailman/listinfo/midpoint
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20160824/d95ab15d/attachment.htm>


More information about the midPoint mailing list