[midPoint] [midpoint] Active/Active Configuration
Pavol Mederly
mederly at evolveum.com
Tue Aug 16 00:00:53 CEST 2016
Florin,
I think I've fixed it:
https://github.com/Evolveum/midpoint/commit/c00f2979cea6f6ada0a1194b13bad51c0ecec2e3
Cannot verify as the issue isn't visible on my computer, but I am quite
convinced it should be fixed now. The suspected cause is that some
query-only transactions weren't closed properly. C3P0 closes them
automatically but other connection providers probably not
<http://stackoverflow.com/questions/19396563/does-hibernates-session-close-automatically-rollback-uncommitted-transactions>.
You can safely use the master, or you can apply the above patch to 3.4
yourself. The master is something like "3.4-support" these days
(preparing for 3.4.1), so it should be of production-level quality, or
at least near it.
(I'm going to run full test suite during the night, but I don't expect
any failures. Or you can check after approx 3 hours here
<https://travis-ci.org/Evolveum/midpoint>.)
Best regards,
Pavol Mederly
Software developer
evolveum.com
On 15.08.2016 22:57, Florin. Stingaciu wrote:
> Hey Pavol,
>
> I uploaded the log file you requested to my gdrive
> (https://drive.google.com/file/d/0B9f4-vWBsztoZy1zWThGYnpGeGs/view?usp=sharing)
> It's a trace level debug so the file is quite large. Let me know if
> anything jumps out at you.
>
> Thanks,
> -F
>
> On Mon, Aug 15, 2016 at 1:07 PM, Pavol Mederly <mederly at evolveum.com
> <mailto:mederly at evolveum.com>> wrote:
>
> Strange enough. I've replicated your setup on my computer (using
> PostgreSQL, as I have no MySQL currently installed), and it works:
> midPoint starts cleanly. I am able to add/delete repository objects.
>
> Attached are my config.xml and server.xml; they are pretty much
> the same as yours.
>
> So, I have no idea. You could try adding
> <database>mysql</database> into <repository> element in your
> config.xml, but it would most probably make no difference.
>
> As last resort, you could turn on the most detailed debugging for
> the repository on startup by adding a line like this
>
> <logger name="com.evolveum.midpoint.repo" level="TRACE" />
>
> into webapps\midpoint\WEB-INF\classes\logback.xml file and start
> midPoint. In idm.log there should be quite a detailed record of
> what's going on.
>
> Pavol Mederly
> Software developer
> evolveum.com <http://evolveum.com>
>
> On 15.08.2016 21:03, Florin. Stingaciu wrote:
>> Here's my datasource config:
>>
>> <Resource name="jdbc/mysql" auth="Container"
>> type="javax.sql.DataSource"
>> username="midpoint" password="pass"
>> url="jdbc:mysql://SERVER:3306/midpoint_dev"
>> driverClassName="com.mysql.jdbc.Driver"
>> accessToUnderlyingConnectionAllowed="true"
>> initialSize="5" maxWait="5000"
>> maxActive="30" maxIdle="5"
>> validationQuery="select 1"
>> poolPreparedStatements="true"/>
>>
>> And here's my config.xml:
>>
>> <repository>
>> <embedded>false</embedded>
>> <repositoryServiceFactoryClass>com.evolveum.midpoint.repo.sql.SqlRepositoryFactory</repositoryServiceFactoryClass>
>> <hibernateHbm2ddl>validate</hibernateHbm2ddl>
>> <hibernateDialect>com.evolveum.midpoint.repo.sql.util.MidPointMySQLDialect</hibernateDialect>
>> <dataSource>java:comp/env/jdbc/mysql</dataSource>
>> </repository>
>> <taskManager>
>> <clustered>true</clustered>
>> <jmxUsername>midpoint</jmxUsername>
>> <jmxPassword>password</jmxPassword>
>> </taskManager>
>>
>> On Mon, Aug 15, 2016 at 11:44 AM, Pavol Mederly
>> <mederly at evolveum.com <mailto:mederly at evolveum.com>> wrote:
>>
>> Hello Florin,
>>
>> this is really interesting. Please, could you also share your
>> midPoint config.xml, as well as your data source
>> configuration? (except credentials, of course)
>>
>> Best regards,
>>
>> Pavol Mederly
>> Software developer
>> evolveum.com <http://evolveum.com>
>>
>> On 15.08.2016 19:39, Florin. Stingaciu wrote:
>>> Hello Pavol,
>>>
>>> Thanks for your detailed response. I tried setting up the
>>> datasource with validationQuery set up properly for the
>>> mySQL backed I have. However, upon service restart I get the
>>> following errors: http://pastebin.com/8dpGN0JC
>>> <http://pastebin.com/8dpGN0JC>
>>>
>>> To save you a click, it seems as though the connection is
>>> set up in readonly mode or some other strange things happen.
>>> I've tried setting readonly="false" in the resource
>>> definition in server.xml but that didn't help. I will
>>> continue researching this, however any guidance would be
>>> quite appreciated.
>>>
>>> Thanks!
>>> -F
>>>
>>> On Mon, Aug 15, 2016 at 2:29 AM, Pavol Mederly
>>> <mederly at evolveum.com <mailto:mederly at evolveum.com>> wrote:
>>>
>>> Hello Florin,
>>>
>>> having looked at your logs, it seems that maybe explicit
>>> setting of validationQuery in Quartz data source setup
>>> would help. (See
>>> http://www.quartz-scheduler.org/documentation/quartz-2.x/configuration/ConfigDataSources.html
>>> <http://www.quartz-scheduler.org/documentation/quartz-2.x/configuration/ConfigDataSources.html>.)
>>>
>>> Unfortunately, current midPoint implementation does not
>>> allow to configure Quartz data source parameters. So,
>>> there are the following three possibilities:
>>>
>>> 1. Take an alternative route, and use application
>>> server-defined data source (with validationQuery set
>>> up).
>>> 2. Patch Task Manager implementation by adding
>>> appropriate lines to Quartz configuration (see
>>> LocalNodeManager.java:87-90).
>>> 3. Wait until we implement it - I've created an issue
>>> MID-3347 <https://jira.evolveum.com/browse/MID-3347>
>>> for this.
>>>
>>> As for the first one (externally defined data source):
>>> Please see
>>> https://wiki.evolveum.com/display/midPoint/Repository+Configuration#RepositoryConfiguration-Datasourceconfiguration
>>> <https://wiki.evolveum.com/display/midPoint/Repository+Configuration#RepositoryConfiguration-Datasourceconfiguration>
>>> on how to configure midPoint repository with the data
>>> source. This data source will be used also by Quartz, if
>>> not overriden in <taskManager> section. It should work
>>> but I don't remember if someone actually tested this.
>>>
>>> Concerning autoReconnect: I have no experiences with
>>> this setting. I agree with you that about the hesitation
>>> of using it in production environment. If really needed,
>>> I'd recommend to separate midPoint repository
>>> configuration from Quartz configuration by using
>>> different JDBC URLs for the two: the standard one for
>>> the repository and the one with "autoReconnect=true" for
>>> Quartz. In this way, potential negative effects should
>>> be restricted to task management functionality only.
>>> But, overall, I'd suggest trying to eliminate the
>>> problem by setting validationQuery first.
>>>
>>> Best regards,
>>>
>>> Pavol Mederly
>>> Software developer
>>> evolveum.com <http://evolveum.com>
>>>
>>> On 14.08.2016 22:40, Florin. Stingaciu wrote:
>>>> Hello,
>>>>
>>>> I'm trying to configure an active/active configuration,
>>>> and experiencing some issues with the Quartz scheduler.
>>>> The SQL connection seems to timeout quite often and
>>>> result in many warning messages. I'm also experiencing
>>>> some errors -- as the timeout closes the connection,
>>>> some processes are still trying to commit using that
>>>> stale handler.
>>>>
>>>> Here are some relevant logs from both midPoint
>>>> instances in this cluster: http://pastebin.com/JFjqrLnT
>>>> <http://pastebin.com/JFjqrLnT>
>>>> http://pastebin.com/1vy4iYPY <http://pastebin.com/1vy4iYPY>
>>>>
>>>> Also here's the relevant portion of my config.xml file:
>>>> http://pastebin.com/WbydcCC8 <http://pastebin.com/WbydcCC8>
>>>>
>>>> Following the suggestions in the warnings, I've started
>>>> to look at autoReconnect propriety of the JDBC
>>>> connector and applied it to my configs like so:
>>>>
>>>> <jdbcUrl>jdbc:mysql://SERVER:3306/midpoint_dev?autoReconnect=true</jdbcUrl>
>>>>
>>>>
>>>> Since implementing this change, the errors and warnings
>>>> seem to have disappeared. I will continue to monitor
>>>> the logs and ensure this actually the case.
>>>>
>>>> Reading the mysql docs, I found that this is not
>>>> recommended as this may cause data inconsistency issues
>>>> and that stale connection exceptions should be properly
>>>> caught within the application. Namely:
>>>>
>>>> The use of this feature is not recommended, because
>>>> it has side effects related to session state and
>>>> data consistency when applications don't handle
>>>> SQLExceptions properly, and is only designed to be
>>>> used when you are unable to configure your
>>>> application to handle SQLExceptions resulting from
>>>> dead and stale connections properly.
>>>>
>>>>
>>>> Do you have any recommended configuration for this
>>>> scenario? I'd like to move my current production
>>>> environment in an active active configuration, however
>>>> as of right now I'm hesitant to do so in order to avoid
>>>> any data corruption. Especially since it's quite
>>>> difficult to test for data consistency issues that may
>>>> arise from using autoReconnect in my dev environment.
>>>>
>>>> Thanks,
>>>> -F
>>>>
>>>>
>>>> _______________________________________________
>>>> midPoint mailing list
>>>> midPoint at lists.evolveum.com
>>>> <mailto:midPoint at lists.evolveum.com>
>>>> http://lists.evolveum.com/mailman/listinfo/midpoint
>>>> <http://lists.evolveum.com/mailman/listinfo/midpoint>
>>> _______________________________________________ midPoint
>>> mailing list midPoint at lists.evolveum.com
>>> <mailto:midPoint at lists.evolveum.com>
>>> http://lists.evolveum.com/mailman/listinfo/midpoint
>>> <http://lists.evolveum.com/mailman/listinfo/midpoint>
>>>
>>> _______________________________________________
>>> midPoint mailing list
>>> midPoint at lists.evolveum.com <mailto:midPoint at lists.evolveum.com>
>>> http://lists.evolveum.com/mailman/listinfo/midpoint
>>> <http://lists.evolveum.com/mailman/listinfo/midpoint>
>> _______________________________________________ midPoint
>> mailing list midPoint at lists.evolveum.com
>> <mailto:midPoint at lists.evolveum.com>
>> http://lists.evolveum.com/mailman/listinfo/midpoint
>> <http://lists.evolveum.com/mailman/listinfo/midpoint>
>>
>> _______________________________________________
>> midPoint mailing list
>> midPoint at lists.evolveum.com <mailto:midPoint at lists.evolveum.com>
>> http://lists.evolveum.com/mailman/listinfo/midpoint
>> <http://lists.evolveum.com/mailman/listinfo/midpoint>
> _______________________________________________ midPoint mailing
> list midPoint at lists.evolveum.com
> <mailto:midPoint at lists.evolveum.com>
> http://lists.evolveum.com/mailman/listinfo/midpoint
> <http://lists.evolveum.com/mailman/listinfo/midpoint>
>
> _______________________________________________
> midPoint mailing list
> midPoint at lists.evolveum.com
> http://lists.evolveum.com/mailman/listinfo/midpoint
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20160816/b9c11011/attachment.htm>
More information about the midPoint
mailing list