[midPoint] [midpoint] Active/Active Configuration
Pavol Mederly
mederly at evolveum.com
Tue Aug 30 00:39:55 CEST 2016
Hello Florin,
I've tried to use your setup in my environment (slightly modified), but
I was not able to replicate the exceptions. I've tried in particular to
play with the listing of shadow objects.
So we perhaps have to continue diagnosing on your side.
I suggest to try to isolate the problem. If you could use the following
config:
<repository>
<embedded>false</embedded>
<repositoryServiceFactoryClass>com.evolveum.midpoint.repo.sql.SqlRepositoryFactory</repositoryServiceFactoryClass>
<hibernateHbm2ddl>validate</hibernateHbm2ddl>
<database>mysql</database>
<jdbcUsername>...</jdbcUsername>
<jdbcPassword>...</jdbcPassword>
<jdbcUrl>jdbc:mysql://servder:3306/midpoint_dev</jdbcUrl>
</repository>
<taskManager>
<clustered>true</clustered>
<jmxUsername>midpoint</jmxUsername>
<jmxPassword>password</jmxPassword>
<dataSource>java:comp/env/jdbc/mysql</dataSource>
</taskManager>
<workflow>
<enabled>false</enabled>
</workflow>
...and try the testing. The configuration simply makes midPoint use its
own connection mechanism (c3p0) and Quartz use the tomcat's connection pool.
Best regards,
Pavol Mederly
Software developer
evolveum.com
On 29.08.2016 20:17, Florin. Stingaciu wrote:
> As an update, after having the instance up for half an hour or so with
> the above config, I got the following errors again:
> http://pastebin.com/tJJVFVsu
>
> Also, in the deployment, there's no workflows at all and very little
> user load as it's just a dev env.
>
> Thanks,
> -F
>
> On Mon, Aug 29, 2016 at 10:43 AM, Florin. Stingaciu
> <fstingaciu at mirantis.com <mailto:fstingaciu at mirantis.com>> wrote:
>
> Hey Pavol,
>
> Currently, for Quartz and miPoint, I'm using a tomcat7 defined
> datasource with the following config:
>
> <Resource name="jdbc/mysql" auth="Container"
> type="javax.sql.DataSource"
> username="midpoint" password="password"
> url="jdbc:mysql://servder:3306/midpoint_dev"
> driverClassName="com.mysql.jdbc.Driver"
> accessToUnderlyingConnectionAllowed="true"
> initialSize="5" maxWait="5000"
> maxActive="30" maxIdle="5"
> validationQuery="select 1"
> readonly="false"
> poolPreparedStatements="true"/>
>
> And here's my relevant config.xml:
>
> <repository>
> <embedded>false</embedded>
> <repositoryServiceFactoryClass>com.evolveum.midpoint.repo.sql.SqlRepositoryFactory</repositoryServiceFactoryClass>
> <hibernateHbm2ddl>validate</hibernateHbm2ddl>
> <hibernateDialect>com.evolveum.midpoint.repo.sql.util.MidPointMySQLDialect</hibernateDialect>
> <dataSource>java:comp/env/jdbc/mysql</dataSource>
> </repository>
> <taskManager>
> <clustered>true</clustered>
> <jmxUsername>midpoint</jmxUsername>
> <jmxPassword>password</jmxPassword>
> </taskManager>
>
> With this configuration, when I was trying to access the Shadows
> page, I got the errors mentioned in my previous message.
>
> Following your last email, I added the following
> <dataSource>java:comp/env/jdbc/mysql</dataSource> under
> <taskManager>. Now when I try to access the Shadows Page I get the
> following error: http://imgur.com/a/391CU
> <http://imgur.com/a/391CU>. Please note there are four photos in
> that album. Also the reason, I chose an image is due to the fact
> that this error does not appear anywhere in the logs and it's in
> the GUI only.
>
> I would also like to add that even though this error occurs,
> midPoint still continues to function. Before I added the
> <dataSource> in the taskManager section, even though I was using a
> datasource for the Repository, when I tried to access the shadows
> page, midPoint would not work anymore (due to lack of mysql
> connections) until I restarted tomcat.
>
> The last point I want to make is that from what I've seen so far,
> this issue only happens when I try to access the shadow page. I
> don't see this issue while trying to access any other type of
> repository objects.
>
> Thanks,
> -F
>
>
> On Fri, Aug 26, 2016 at 3:41 PM, Pavol Mederly
> <mederly at evolveum.com <mailto:mederly at evolveum.com>> wrote:
>
> Hello Florin,
>
> it seems that either midPoint (i.e. hibernate), Quartz
> scheduler, or Activiti BPMS use connections without properly
> closing them.
>
> First step to diagnose it would be perhaps to setup things
> like this:
>
> 1. Let midPoint and Activiti use their own data source (by
> configuring them with a JDBC URL): see Repository
> configuration
> <https://wiki.evolveum.com/display/midPoint/Repository+Configuration>
> and Workflow configuration
> <https://wiki.evolveum.com/display/midPoint/Workflow+configuration>
> wiki pages. (If you don't use approvals you can disable
> workflows altogether by setting "enabled" to "false".)
> 2. Keep Quartz schedule using tomcat data source: something like
>
> |<||taskManager||>|
> |||<||clustered||>true</||clustered||>
> <dataSource>...your data source name...</dataSource>
> |
> ...
> |</||taskManager||>
> |
> And you'll see if the problem would appear again. (Maybe you
> could even configure three Tomcat data sources; one for
> midPoint, second for Quartz, and the third for Activiti, if
> needed. But I have not done this yet; these are pure
> speculations of mine...)
>
> Best regards,
>
> Pavol Mederly
> Software developer
> evolveum.com <http://evolveum.com>
>
> On 24.08.2016 22:41, Florin. Stingaciu wrote:
>> I just rebuilt the latest snapshot
>> (git-v3.5devel-221-g9744fa0) and am experiencing the same
>> problem.
>>
>> Thanks,
>> -F
>>
>>
>> On Wed, Aug 24, 2016 at 1:24 PM, Florin. Stingaciu
>> <fstingaciu at mirantis.com <mailto:fstingaciu at mirantis.com>> wrote:
>>
>> Upon further investigation I found that this error only
>> occurs when I try to access the shadow repository via
>> Configuration->Repository Objects->Shadows
>>
>> It seems that something triggers the creation of DB
>> connections until it runs out of available connections.
>> For this particular instance, I'm at
>> git-v3.5devel-170-g26b0644.
>>
>> Thanks,
>> -F
>>
>> On Wed, Aug 24, 2016 at 1:13 PM, Florin. Stingaciu
>> <fstingaciu at mirantis.com
>> <mailto:fstingaciu at mirantis.com>> wrote:
>>
>> Hello Pavol,
>>
>> After running this configuration my dev environment
>> with two instances up for a couple of days, I started
>> to get the following error messages:
>> http://pastebin.com/nkxE9LNA
>> <http://pastebin.com/nkxE9LNA>
>>
>> It seems as though the connection pool is not
>> properly cleaned up. The database logs indicate no
>> failure and in fact my production environment is
>> hooked up to the same DB instance and is working fine.
>>
>> Any recommendations would be very appreciated.
>>
>> Thanks,
>> -F
>>
>> On Mon, Aug 15, 2016 at 3:29 PM, Florin. Stingaciu
>> <fstingaciu at mirantis.com
>> <mailto:fstingaciu at mirantis.com>> wrote:
>>
>> Awesome! Thanks for all your help today Pavol. I
>> will try and test in my environment and let you
>> know how that goes.
>>
>> Thanks again!
>> -F
>>
>> On Mon, Aug 15, 2016 at 3:00 PM, Pavol Mederly
>> <mederly at evolveum.com
>> <mailto:mederly at evolveum.com>> wrote:
>>
>> Florin,
>>
>> I think I've fixed it:
>> https://github.com/Evolveum/midpoint/commit/c00f2979cea6f6ada0a1194b13bad51c0ecec2e3
>> <https://github.com/Evolveum/midpoint/commit/c00f2979cea6f6ada0a1194b13bad51c0ecec2e3>
>>
>> Cannot verify as the issue isn't visible on
>> my computer, but I am quite convinced it
>> should be fixed now. The suspected cause is
>> that some query-only transactions weren't
>> closed properly. C3P0 closes them
>> automatically but other connection providers
>> probably not
>> <http://stackoverflow.com/questions/19396563/does-hibernates-session-close-automatically-rollback-uncommitted-transactions>.
>>
>> You can safely use the master, or you can
>> apply the above patch to 3.4 yourself. The
>> master is something like "3.4-support" these
>> days (preparing for 3.4.1), so it should be
>> of production-level quality, or at least near
>> it.
>> (I'm going to run full test suite during the
>> night, but I don't expect any failures. Or
>> you can check after approx 3 hours here
>> <https://travis-ci.org/Evolveum/midpoint>.)
>>
>> Best regards,
>>
>> Pavol Mederly
>> Software developer
>> evolveum.com <http://evolveum.com>
>>
>> On 15.08.2016 22:57, Florin. Stingaciu wrote:
>>> Hey Pavol,
>>>
>>> I uploaded the log file you requested to my
>>> gdrive
>>> (https://drive.google.com/file/d/0B9f4-vWBsztoZy1zWThGYnpGeGs/view?usp=sharing
>>> <https://drive.google.com/file/d/0B9f4-vWBsztoZy1zWThGYnpGeGs/view?usp=sharing>)
>>> It's a trace level debug so the file is
>>> quite large. Let me know if anything jumps
>>> out at you.
>>>
>>> Thanks,
>>> -F
>>>
>>> On Mon, Aug 15, 2016 at 1:07 PM, Pavol
>>> Mederly <mederly at evolveum.com
>>> <mailto:mederly at evolveum.com>> wrote:
>>>
>>> Strange enough. I've replicated your
>>> setup on my computer (using PostgreSQL,
>>> as I have no MySQL currently installed),
>>> and it works: midPoint starts cleanly. I
>>> am able to add/delete repository objects.
>>>
>>> Attached are my config.xml and
>>> server.xml; they are pretty much the
>>> same as yours.
>>>
>>> So, I have no idea. You could try adding
>>> <database>mysql</database> into
>>> <repository> element in your config.xml,
>>> but it would most probably make no
>>> difference.
>>>
>>> As last resort, you could turn on the
>>> most detailed debugging for the
>>> repository on startup by adding a line
>>> like this
>>>
>>> <logger name="com.evolveum.midpoint.re
>>> <http://com.evolveum.midpoint.re>po"
>>> level="TRACE" />
>>>
>>> into
>>> webapps\midpoint\WEB-INF\classes\logback.xml
>>> file and start midPoint. In idm.log
>>> there should be quite a detailed record
>>> of what's going on.
>>>
>>> Pavol Mederly
>>> Software developer
>>> evolveum.com <http://evolveum.com>
>>>
>>> On 15.08.2016 21:03, Florin. Stingaciu
>>> wrote:
>>>> Here's my datasource config:
>>>>
>>>> <Resource name="jdbc/mysql"
>>>> auth="Container"
>>>> type="javax.sql.DataSource"
>>>> username="midpoint" password="pass"
>>>> url="jdbc:mysql://SERVER:3306/midpoint_dev"
>>>> driverClassName="com.mysql.jdbc.Driver"
>>>> accessToUnderlyingConnectionAllowed="true"
>>>> initialSize="5" maxWait="5000"
>>>> maxActive="30" maxIdle="5"
>>>> validationQuery="select 1"
>>>> poolPreparedStatements="true"/>
>>>>
>>>> And here's my config.xml:
>>>>
>>>> <repository>
>>>> <embedded>false</embedded>
>>>> <repositoryServiceFactoryClass>com.evolveum.midpoint.repo.sql.SqlRepositoryFactory</repositoryServiceFactoryClass>
>>>> <hibernateHbm2ddl>validate</hibernateHbm2ddl>
>>>> <hibernateDialect>com.evolveum.midpoint.repo.sql.util.MidPointMySQLDialect</hibernateDialect>
>>>> <dataSource>java:comp/env/jdbc/mysql</dataSource>
>>>> </repository>
>>>> <taskManager>
>>>> <clustered>true</clustered>
>>>> <jmxUsername>midpoint</jmxUsername>
>>>> <jmxPassword>password</jmxPassword>
>>>> </taskManager>
>>>>
>>>> On Mon, Aug 15, 2016 at 11:44 AM, Pavol
>>>> Mederly <mederly at evolveum.com
>>>> <mailto:mederly at evolveum.com>> wrote:
>>>>
>>>> Hello Florin,
>>>>
>>>> this is really interesting. Please,
>>>> could you also share your midPoint
>>>> config.xml, as well as your data
>>>> source configuration? (except
>>>> credentials, of course)
>>>>
>>>> Best regards,
>>>>
>>>> Pavol Mederly
>>>> Software developer
>>>> evolveum.com <http://evolveum.com>
>>>>
>>>> On 15.08.2016 19:39, Florin.
>>>> Stingaciu wrote:
>>>>> Hello Pavol,
>>>>>
>>>>> Thanks for your detailed response.
>>>>> I tried setting up the datasource
>>>>> with validationQuery set up
>>>>> properly for the mySQL backed I
>>>>> have. However, upon service
>>>>> restart I get the following
>>>>> errors:
>>>>> http://pastebin.com/8dpGN0JC
>>>>> <http://pastebin.com/8dpGN0JC>
>>>>>
>>>>> To save you a click, it seems as
>>>>> though the connection is set up in
>>>>> readonly mode or some other
>>>>> strange things happen. I've tried
>>>>> setting readonly="false" in the
>>>>> resource definition in server.xml
>>>>> but that didn't help. I will
>>>>> continue researching this, however
>>>>> any guidance would be quite
>>>>> appreciated.
>>>>>
>>>>> Thanks!
>>>>> -F
>>>>>
>>>>> On Mon, Aug 15, 2016 at 2:29 AM,
>>>>> Pavol Mederly
>>>>> <mederly at evolveum.com
>>>>> <mailto:mederly at evolveum.com>> wrote:
>>>>>
>>>>> Hello Florin,
>>>>>
>>>>> having looked at your logs, it
>>>>> seems that maybe explicit
>>>>> setting of validationQuery in
>>>>> Quartz data source setup would
>>>>> help. (See
>>>>> http://www.quartz-scheduler.org/documentation/quartz-2.x/configuration/ConfigDataSources.html
>>>>> <http://www.quartz-scheduler.org/documentation/quartz-2.x/configuration/ConfigDataSources.html>.)
>>>>>
>>>>> Unfortunately, current
>>>>> midPoint implementation does
>>>>> not allow to configure Quartz
>>>>> data source parameters. So,
>>>>> there are the following three
>>>>> possibilities:
>>>>>
>>>>> 1. Take an alternative route,
>>>>> and use application
>>>>> server-defined data source
>>>>> (with validationQuery set up).
>>>>> 2. Patch Task Manager
>>>>> implementation by adding
>>>>> appropriate lines to
>>>>> Quartz configuration (see
>>>>> LocalNodeManager.java:87-90).
>>>>> 3. Wait until we implement it
>>>>> - I've created an issue
>>>>> MID-3347
>>>>> <https://jira.evolveum.com/browse/MID-3347>
>>>>> for this.
>>>>>
>>>>> As for the first one
>>>>> (externally defined data
>>>>> source): Please see
>>>>> https://wiki.evolveum.com/display/midPoint/Repository+Configuration#RepositoryConfiguration-Datasourceconfiguration
>>>>> <https://wiki.evolveum.com/display/midPoint/Repository+Configuration#RepositoryConfiguration-Datasourceconfiguration>
>>>>> on how to configure midPoint
>>>>> repository with the data
>>>>> source. This data source will
>>>>> be used also by Quartz, if not
>>>>> overriden in <taskManager>
>>>>> section. It should work but I
>>>>> don't remember if someone
>>>>> actually tested this.
>>>>>
>>>>> Concerning autoReconnect: I
>>>>> have no experiences with this
>>>>> setting. I agree with you that
>>>>> about the hesitation of using
>>>>> it in production environment.
>>>>> If really needed, I'd
>>>>> recommend to separate midPoint
>>>>> repository configuration from
>>>>> Quartz configuration by using
>>>>> different JDBC URLs for the
>>>>> two: the standard one for the
>>>>> repository and the one with
>>>>> "autoReconnect=true" for
>>>>> Quartz. In this way, potential
>>>>> negative effects should be
>>>>> restricted to task management
>>>>> functionality only. But,
>>>>> overall, I'd suggest trying to
>>>>> eliminate the problem by
>>>>> setting validationQuery first.
>>>>>
>>>>> Best regards,
>>>>>
>>>>> Pavol Mederly
>>>>> Software developer
>>>>> evolveum.com <http://evolveum.com>
>>>>>
>>>>> On 14.08.2016 22:40, Florin.
>>>>> Stingaciu wrote:
>>>>>> Hello,
>>>>>>
>>>>>> I'm trying to configure an
>>>>>> active/active configuration,
>>>>>> and experiencing some issues
>>>>>> with the Quartz scheduler.
>>>>>> The SQL connection seems to
>>>>>> timeout quite often and
>>>>>> result in many warning
>>>>>> messages. I'm also
>>>>>> experiencing some errors --
>>>>>> as the timeout closes the
>>>>>> connection, some processes
>>>>>> are still trying to commit
>>>>>> using that stale handler.
>>>>>>
>>>>>> Here are some relevant logs
>>>>>> from both midPoint instances
>>>>>> in this cluster:
>>>>>> http://pastebin.com/JFjqrLnT
>>>>>> <http://pastebin.com/JFjqrLnT>
>>>>>> http://pastebin.com/1vy4iYPY
>>>>>> <http://pastebin.com/1vy4iYPY>
>>>>>>
>>>>>> Also here's the relevant
>>>>>> portion of my config.xml
>>>>>> file:
>>>>>> http://pastebin.com/WbydcCC8
>>>>>> <http://pastebin.com/WbydcCC8>
>>>>>>
>>>>>> Following the suggestions in
>>>>>> the warnings, I've started to
>>>>>> look at autoReconnect
>>>>>> propriety of the JDBC
>>>>>> connector and applied it to
>>>>>> my configs like so:
>>>>>>
>>>>>> <jdbcUrl>jdbc:mysql://SERVER:3306/midpoint_dev?autoReconnect=true</jdbcUrl>
>>>>>>
>>>>>>
>>>>>> Since implementing this
>>>>>> change, the errors and
>>>>>> warnings seem to have
>>>>>> disappeared. I will continue
>>>>>> to monitor the logs and
>>>>>> ensure this actually the case.
>>>>>>
>>>>>> Reading the mysql docs, I
>>>>>> found that this is not
>>>>>> recommended as this may cause
>>>>>> data inconsistency issues and
>>>>>> that stale connection
>>>>>> exceptions should be properly
>>>>>> caught within the
>>>>>> application. Namely:
>>>>>>
>>>>>> The use of this feature
>>>>>> is not recommended,
>>>>>> because it has side
>>>>>> effects related to
>>>>>> session state and data
>>>>>> consistency when
>>>>>> applications don't handle
>>>>>> SQLExceptions properly,
>>>>>> and is only designed to
>>>>>> be used when you are
>>>>>> unable to configure your
>>>>>> application to handle
>>>>>> SQLExceptions resulting
>>>>>> from dead and stale
>>>>>> connections properly.
>>>>>>
>>>>>>
>>>>>> Do you have any recommended
>>>>>> configuration for this
>>>>>> scenario? I'd like to move my
>>>>>> current production
>>>>>> environment in an active
>>>>>> active configuration, however
>>>>>> as of right now I'm hesitant
>>>>>> to do so in order to avoid
>>>>>> any data corruption.
>>>>>> Especially since it's quite
>>>>>> difficult to test for data
>>>>>> consistency issues that may
>>>>>> arise from using
>>>>>> autoReconnect in my dev
>>>>>> environment.
>>>>>>
>>>>>> Thanks,
>>>>>> -F
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> midPoint mailing list
>>>>>> midPoint at lists.evolveum.com
>>>>>> <mailto:midPoint at lists.evolveum.com>
>>>>>> http://lists.evolveum.com/mailman/listinfo/midpoint
>>>>>> <http://lists.evolveum.com/mailman/listinfo/midpoint>
>>>>> _______________________________________________
>>>>> midPoint mailing list
>>>>> midPoint at lists.evolveum.com
>>>>> <mailto:midPoint at lists.evolveum.com>
>>>>> http://lists.evolveum.com/mailman/listinfo/midpoint
>>>>> <http://lists.evolveum.com/mailman/listinfo/midpoint>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> midPoint mailing list
>>>>> midPoint at lists.evolveum.com
>>>>> <mailto:midPoint at lists.evolveum.com>
>>>>> http://lists.evolveum.com/mailman/listinfo/midpoint
>>>>> <http://lists.evolveum.com/mailman/listinfo/midpoint>
>>>> _______________________________________________
>>>> midPoint mailing list
>>>> midPoint at lists.evolveum.com
>>>> <mailto:midPoint at lists.evolveum.com>
>>>> http://lists.evolveum.com/mailman/listinfo/midpoint
>>>> <http://lists.evolveum.com/mailman/listinfo/midpoint>
>>>>
>>>>
>>>> _______________________________________________
>>>> midPoint mailing list
>>>> midPoint at lists.evolveum.com
>>>> <mailto:midPoint at lists.evolveum.com>
>>>> http://lists.evolveum.com/mailman/listinfo/midpoint
>>>> <http://lists.evolveum.com/mailman/listinfo/midpoint>
>>> _______________________________________________
>>> midPoint mailing list
>>> midPoint at lists.evolveum.com
>>> <mailto:midPoint at lists.evolveum.com>
>>> http://lists.evolveum.com/mailman/listinfo/midpoint
>>> <http://lists.evolveum.com/mailman/listinfo/midpoint>
>>>
>>>
>>> _______________________________________________
>>> midPoint mailing list
>>> midPoint at lists.evolveum.com
>>> <mailto:midPoint at lists.evolveum.com>
>>> http://lists.evolveum.com/mailman/listinfo/midpoint
>>> <http://lists.evolveum.com/mailman/listinfo/midpoint>
>> _______________________________________________
>> midPoint mailing list
>> midPoint at lists.evolveum.com
>> <mailto:midPoint at lists.evolveum.com>
>> http://lists.evolveum.com/mailman/listinfo/midpoint
>> <http://lists.evolveum.com/mailman/listinfo/midpoint>
>>
>>
>> _______________________________________________
>> midPoint mailing list
>> midPoint at lists.evolveum.com <mailto:midPoint at lists.evolveum.com>
>> http://lists.evolveum.com/mailman/listinfo/midpoint
>> <http://lists.evolveum.com/mailman/listinfo/midpoint>
> _______________________________________________ midPoint
> mailing list midPoint at lists.evolveum.com
> <mailto:midPoint at lists.evolveum.com>
> http://lists.evolveum.com/mailman/listinfo/midpoint
> <http://lists.evolveum.com/mailman/listinfo/midpoint>
>
> _______________________________________________
> midPoint mailing list
> midPoint at lists.evolveum.com
> http://lists.evolveum.com/mailman/listinfo/midpoint
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20160830/16b464c2/attachment.htm>
More information about the midPoint
mailing list