[midPoint] [midpoint] Active/Active Configuration
Florin. Stingaciu
fstingaciu at mirantis.com
Wed Aug 24 22:13:24 CEST 2016
Hello Pavol,
After running this configuration my dev environment with two instances up
for a couple of days, I started to get the following error messages:
http://pastebin.com/nkxE9LNA
It seems as though the connection pool is not properly cleaned up. The
database logs indicate no failure and in fact my production environment is
hooked up to the same DB instance and is working fine.
Any recommendations would be very appreciated.
Thanks,
-F
On Mon, Aug 15, 2016 at 3:29 PM, Florin. Stingaciu <fstingaciu at mirantis.com>
wrote:
> Awesome! Thanks for all your help today Pavol. I will try and test in my
> environment and let you know how that goes.
>
> Thanks again!
> -F
>
> On Mon, Aug 15, 2016 at 3:00 PM, Pavol Mederly <mederly at evolveum.com>
> wrote:
>
>> Florin,
>>
>> I think I've fixed it: https://github.com/Evolveum/mi
>> dpoint/commit/c00f2979cea6f6ada0a1194b13bad51c0ecec2e3
>>
>> Cannot verify as the issue isn't visible on my computer, but I am quite
>> convinced it should be fixed now. The suspected cause is that some
>> query-only transactions weren't closed properly. C3P0 closes them
>> automatically but other connection providers probably not
>> <http://stackoverflow.com/questions/19396563/does-hibernates-session-close-automatically-rollback-uncommitted-transactions>
>> .
>>
>> You can safely use the master, or you can apply the above patch to 3.4
>> yourself. The master is something like "3.4-support" these days (preparing
>> for 3.4.1), so it should be of production-level quality, or at least near
>> it.
>> (I'm going to run full test suite during the night, but I don't expect
>> any failures. Or you can check after approx 3 hours here
>> <https://travis-ci.org/Evolveum/midpoint>.)
>>
>> Best regards,
>>
>> Pavol Mederly
>> Software developerevolveum.com
>>
>> On 15.08.2016 22:57, Florin. Stingaciu wrote:
>>
>> Hey Pavol,
>>
>> I uploaded the log file you requested to my gdrive (
>> https://drive.google.com/file/d/0B9f4-vWBsztoZy1zWThGYnpGeG
>> s/view?usp=sharing) It's a trace level debug so the file is quite large.
>> Let me know if anything jumps out at you.
>>
>> Thanks,
>> -F
>>
>> On Mon, Aug 15, 2016 at 1:07 PM, Pavol Mederly <mederly at evolveum.com>
>> wrote:
>>
>>> Strange enough. I've replicated your setup on my computer (using
>>> PostgreSQL, as I have no MySQL currently installed), and it works: midPoint
>>> starts cleanly. I am able to add/delete repository objects.
>>>
>>> Attached are my config.xml and server.xml; they are pretty much the same
>>> as yours.
>>>
>>> So, I have no idea. You could try adding <database>mysql</database> into
>>> <repository> element in your config.xml, but it would most probably make no
>>> difference.
>>>
>>> As last resort, you could turn on the most detailed debugging for the
>>> repository on startup by adding a line like this
>>>
>>> <logger name="com.evolveum.midpoint.repo" level="TRACE" />
>>>
>>> into webapps\midpoint\WEB-INF\classes\logback.xml file and start
>>> midPoint. In idm.log there should be quite a detailed record of what's
>>> going on.
>>>
>>> Pavol Mederly
>>> Software developerevolveum.com
>>>
>>> On 15.08.2016 21:03, Florin. Stingaciu wrote:
>>>
>>> Here's my datasource config:
>>>
>>> <Resource name="jdbc/mysql" auth="Container"
>>> type="javax.sql.DataSource"
>>> username="midpoint" password="pass"
>>> url="jdbc:mysql://SERVER:3306/midpoint_dev"
>>> driverClassName="com.mysql.jdbc.Driver"
>>> accessToUnderlyingConnectionAllowed="true"
>>> initialSize="5" maxWait="5000"
>>> maxActive="30" maxIdle="5"
>>> validationQuery="select 1"
>>> poolPreparedStatements="true"/>
>>>
>>> And here's my config.xml:
>>>
>>> <repository>
>>> <embedded>false</embedded>
>>> <repositoryServiceFactoryClass
>>> >com.evolveum.midpoint.repo.sql.SqlRepositoryFactory</reposi
>>> toryServiceFactoryClass>
>>> <hibernateHbm2ddl>validate</hibernateHbm2ddl>
>>> <hibernateDialect>com.evolveum
>>> .midpoint.repo.sql.util.MidPointMySQLDialect</hibernateDialect>
>>> <dataSource>java:comp/env/jdbc/mysql</dataSource>
>>> </repository>
>>> <taskManager>
>>> <clustered>true</clustered>
>>> <jmxUsername>midpoint</jmxUsername>
>>> <jmxPassword>password</jmxPassword>
>>> </taskManager>
>>>
>>> On Mon, Aug 15, 2016 at 11:44 AM, Pavol Mederly <mederly at evolveum.com>
>>> wrote:
>>>
>>>> Hello Florin,
>>>>
>>>> this is really interesting. Please, could you also share your midPoint
>>>> config.xml, as well as your data source configuration? (except credentials,
>>>> of course)
>>>>
>>>> Best regards,
>>>>
>>>> Pavol Mederly
>>>> Software developerevolveum.com
>>>>
>>>> On 15.08.2016 19:39, Florin. Stingaciu wrote:
>>>>
>>>> Hello Pavol,
>>>>
>>>> Thanks for your detailed response. I tried setting up the datasource
>>>> with validationQuery set up properly for the mySQL backed I have. However,
>>>> upon service restart I get the following errors: http://pastebin.com/8d
>>>> pGN0JC
>>>>
>>>> To save you a click, it seems as though the connection is set up in
>>>> readonly mode or some other strange things happen. I've tried setting
>>>> readonly="false" in the resource definition in server.xml but that
>>>> didn't help. I will continue researching this, however any guidance would
>>>> be quite appreciated.
>>>>
>>>> Thanks!
>>>> -F
>>>>
>>>> On Mon, Aug 15, 2016 at 2:29 AM, Pavol Mederly <mederly at evolveum.com>
>>>> wrote:
>>>>
>>>>> Hello Florin,
>>>>>
>>>>> having looked at your logs, it seems that maybe explicit setting of
>>>>> validationQuery in Quartz data source setup would help. (See
>>>>> http://www.quartz-scheduler.org/documentation/quartz-2.x/con
>>>>> figuration/ConfigDataSources.html.)
>>>>>
>>>>> Unfortunately, current midPoint implementation does not allow to
>>>>> configure Quartz data source parameters. So, there are the following three
>>>>> possibilities:
>>>>>
>>>>> 1. Take an alternative route, and use application server-defined
>>>>> data source (with validationQuery set up).
>>>>> 2. Patch Task Manager implementation by adding appropriate lines
>>>>> to Quartz configuration (see LocalNodeManager.java:87-90).
>>>>> 3. Wait until we implement it - I've created an issue MID-3347
>>>>> <https://jira.evolveum.com/browse/MID-3347> for this.
>>>>>
>>>>> As for the first one (externally defined data source): Please see
>>>>> https://wiki.evolveum.com/display/midPoint/Repository+Config
>>>>> uration#RepositoryConfiguration-Datasourceconfiguration on how to
>>>>> configure midPoint repository with the data source. This data source will
>>>>> be used also by Quartz, if not overriden in <taskManager> section. It
>>>>> should work but I don't remember if someone actually tested this.
>>>>>
>>>>> Concerning autoReconnect: I have no experiences with this setting. I
>>>>> agree with you that about the hesitation of using it in production
>>>>> environment. If really needed, I'd recommend to separate midPoint
>>>>> repository configuration from Quartz configuration by using different JDBC
>>>>> URLs for the two: the standard one for the repository and the one with
>>>>> "autoReconnect=true" for Quartz. In this way, potential negative effects
>>>>> should be restricted to task management functionality only. But, overall,
>>>>> I'd suggest trying to eliminate the problem by setting validationQuery
>>>>> first.
>>>>>
>>>>> Best regards,
>>>>>
>>>>> Pavol Mederly
>>>>> Software developerevolveum.com
>>>>>
>>>>> On 14.08.2016 22:40, Florin. Stingaciu wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> I'm trying to configure an active/active configuration, and
>>>>> experiencing some issues with the Quartz scheduler. The SQL connection
>>>>> seems to timeout quite often and result in many warning messages. I'm also
>>>>> experiencing some errors -- as the timeout closes the connection, some
>>>>> processes are still trying to commit using that stale handler.
>>>>>
>>>>> Here are some relevant logs from both midPoint instances in this
>>>>> cluster: http://pastebin.com/JFjqrLnT http://pastebin.com/1vy4iYPY
>>>>>
>>>>> Also here's the relevant portion of my config.xml file:
>>>>> http://pastebin.com/WbydcCC8
>>>>>
>>>>> Following the suggestions in the warnings, I've started to look at
>>>>> autoReconnect propriety of the JDBC connector and applied it to my configs
>>>>> like so:
>>>>>
>>>>> <jdbcUrl>jdbc:mysql://SERVER:3306/midpoint_dev?autoReconnect
>>>>>> =true</jdbcUrl>
>>>>>
>>>>>
>>>>> Since implementing this change, the errors and warnings seem to have
>>>>> disappeared. I will continue to monitor the logs and ensure this actually
>>>>> the case.
>>>>>
>>>>> Reading the mysql docs, I found that this is not recommended as this
>>>>> may cause data inconsistency issues and that stale connection exceptions
>>>>> should be properly caught within the application. Namely:
>>>>>
>>>>> The use of this feature is not recommended, because it has side
>>>>>> effects related to session state and data consistency when applications
>>>>>> don't handle SQLExceptions properly, and is only designed to be used when
>>>>>> you are unable to configure your application to handle SQLExceptions
>>>>>> resulting from dead and stale connections properly.
>>>>>
>>>>>
>>>>> Do you have any recommended configuration for this scenario? I'd like
>>>>> to move my current production environment in an active active
>>>>> configuration, however as of right now I'm hesitant to do so in order to
>>>>> avoid any data corruption. Especially since it's quite difficult to test
>>>>> for data consistency issues that may arise from using autoReconnect in my
>>>>> dev environment.
>>>>>
>>>>> Thanks,
>>>>> -F
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> midPoint mailing listmidPoint at lists.evolveum.comhttp://lists.evolveum.com/mailman/listinfo/midpoint
>>>>>
>>>>> _______________________________________________ midPoint mailing list
>>>>> midPoint at lists.evolveum.com http://lists.evolveum.com/mail
>>>>> man/listinfo/midpoint
>>>>
>>>> _______________________________________________
>>>> midPoint mailing listmidPoint at lists.evolveum.comhttp://lists.evolveum.com/mailman/listinfo/midpoint
>>>>
>>>> _______________________________________________ midPoint mailing list
>>>> midPoint at lists.evolveum.com http://lists.evolveum.com/mail
>>>> man/listinfo/midpoint
>>>
>>> _______________________________________________
>>> midPoint mailing listmidPoint at lists.evolveum.comhttp://lists.evolveum.com/mailman/listinfo/midpoint
>>>
>>> _______________________________________________ midPoint mailing list
>>> midPoint at lists.evolveum.com http://lists.evolveum.com/mail
>>> man/listinfo/midpoint
>>
>> _______________________________________________
>> midPoint mailing listmidPoint at lists.evolveum.comhttp://lists.evolveum.com/mailman/listinfo/midpoint
>>
>>
>> _______________________________________________
>> midPoint mailing list
>> midPoint at lists.evolveum.com
>> http://lists.evolveum.com/mailman/listinfo/midpoint
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20160824/2f23a6ca/attachment.htm>
More information about the midPoint
mailing list