[midPoint] [midpoint] Active/Active Configuration
Pavol Mederly
mederly at evolveum.com
Tue Aug 30 20:45:22 CEST 2016
Florin,
I see...
I am out of ideas here ... so, logged JIRA issue for this (MID-3378
<https://jira.evolveum.com/browse/MID-3378>).
Maybe someone here uses clustering + MySQL and could help you better. Or
maybe some very detailed logging, or reviewing the parameters (e.g. is
"SELECT 1" reliable way of determining connection validity? Probably
yes, but...).
If you would have any more hints, we can discuss them. I just don't know
at the moment. :(
Pavol Mederly
Software developer
evolveum.com
On 30.08.2016 20:28, Florin. Stingaciu wrote:
> Hey Pavol,
>
> Here's the relevant lines:
>
> 2016-08-30 18:18:44,443 [] [localhost-startStop-1] INFO
> (com.evolveum.midpoint.task.quartzimpl.Initializer): Task Manager
> initialization.
> 2016-08-30 18:18:44,452 [] [localhost-startStop-1] INFO
> (com.evolveum.midpoint.task.quartzimpl.Initializer): Task Manager:
> Quartz Job Store: JDBC, clustered. Threads: 10
> 2016-08-30 18:18:44,452 [] [localhost-startStop-1] INFO
> (com.evolveum.midpoint.task.quartzimpl.TaskManagerConfiguration):
> Quartz database is at java:comp/env/jdbc/mysql (a data source)
> 2016-08-30 18:18:44,461 [] [localhost-startStop-1] INFO
> (com.evolveum.midpoint.task.quartzimpl.cluster.NodeRegistrar):
> Registering this node in the repository as 001 at 100.73.170.188:20001
> <http://100.73.170.188:20001>
> 2016-08-30 18:18:44,945 [] [localhost-startStop-1] INFO
> (com.evolveum.midpoint.task.quartzimpl.execution.TaskSynchronizer):
> Synchronizing Quartz job store with midPoint repository.
> 2016-08-30 18:18:46,036 [] [localhost-startStop-1] INFO
> (com.evolveum.midpoint.task.quartzimpl.execution.TaskSynchronizer):
> Synchronization of midpoint and Quartz task store finished. 26 task(s)
> existing in midPoint repository successfully processed, resulting in 0
> updated Quartz job(s). 0 task(s) removed from Quartz job store.
> Processing of 0 task(s) failed.
> 2016-08-30 18:18:46,037 [] [localhost-startStop-1] INFO
> (com.evolveum.midpoint.task.quartzimpl.Initializer): Task Manager
> initialized
>
> Thanks,
> -F
>
> On Tue, Aug 30, 2016 at 11:17 AM, Pavol Mederly <mederly at evolveum.com
> <mailto:mederly at evolveum.com>> wrote:
>
> Florin,
>
> this is really interesting. The exception of
>
> Caused by:
> com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last
> packet successfully received from the server was 64,841,997
> milliseconds ago. The last packet sent successfully to the server
> was 64,841,998 milliseconds ago. is longer than the server
> configured value of 'wait_timeout'. You should consider either
> expiring and/or testing connection validity before use in your
> application, increasing the server configured values for client
> timeouts, or using the Connector/J connection property
> 'autoReconnect=true' to avoid this problem.
>
> occurs in the context of Quartz scheduler, where the data source
> that checks connection validity is used!
>
> When you start midPoint there should be messages like these:
>
> 2016-08-30 20:14:31,388 [] [RMI TCP Connection(5)-127.0.0.1] INFO
> (com.evolveum.midpoint.task.quartzimpl.Initializer): Task Manager:
> Quartz Job Store: JDBC, NOT clustered. Threads: 10
> 2016-08-30 20:14:31,388 [] [RMI TCP Connection(5)-127.0.0.1] INFO
> (com.evolveum.midpoint.task.quartzimpl.TaskManagerConfiguration):
> Quartz database is at jdbc:postgresql://localhost/midpoint (a JDBC
> URL)
>
> In your case there should be a reference to the data source. Could
> you please verify it is OK, and perhaps paste relevant lines here?
>
> Pavol Mederly
> Software developer
> evolveum.com <http://evolveum.com>
>
> On 30.08.2016 19:21, Florin. Stingaciu wrote:
>> Hey Pavol,
>>
>> Over night, I now have the same error messages I initially had:
>> http://pastebin.com/iWeSaswD <http://pastebin.com/iWeSaswD>
>>
>> Thank you for all your help so far. Any further recommendations
>> would be really appreciated.
>>
>> Thanks,
>> -F
>>
>> On Mon, Aug 29, 2016 at 3:54 PM, Florin. Stingaciu
>> <fstingaciu at mirantis.com <mailto:fstingaciu at mirantis.com>> wrote:
>>
>> Hey Pavol,
>>
>> I switched the repository to the configuration you mentioned,
>> and I could now access the shadow page. However (minus the
>> task manager config), this is the same configuration as I
>> previously had where I ran into errors and then made this
>> thread because of it. For now I'm not seeing any errors in
>> the log files.
>>
>> I'll update this thread once I run into issues. As of right
>> now, I'm still definitely not comfortable moving to active
>> active considering all these issues. Have you guys seen
>> similar issues before? Or do you believe I've just
>> misconfigured something?
>>
>> Thanks,
>> -F
>>
>>
>>
>> On Mon, Aug 29, 2016 at 3:39 PM, Pavol Mederly
>> <mederly at evolveum.com <mailto:mederly at evolveum.com>> wrote:
>>
>> Hello Florin,
>>
>> I've tried to use your setup in my environment (slightly
>> modified), but I was not able to replicate the
>> exceptions. I've tried in particular to play with the
>> listing of shadow objects.
>>
>> So we perhaps have to continue diagnosing on your side.
>>
>> I suggest to try to isolate the problem. If you could use
>> the following config:
>>
>> <repository>
>> <embedded>false</embedded>
>> <repositoryServiceFactoryClass>com.evolveum.midpoint.repo.sql.SqlRepositoryFactory</repositoryServiceFactoryClass>
>> <hibernateHbm2ddl>validate</hibernateHbm2ddl>
>> <database>mysql</database>
>> <jdbcUsername>...</jdbcUsername>
>> <jdbcPassword>...</jdbcPassword>
>> <jdbcUrl>jdbc:mysql://servder:3306/midpoint_dev</jdbcUrl>
>> </repository>
>> <taskManager>
>> <clustered>true</clustered>
>> <jmxUsername>midpoint</jmxUsername>
>> <jmxPassword>password</jmxPassword>
>> <dataSource>java:comp/env/jdbc/mysql</dataSource>
>> </taskManager>
>> <workflow>
>> <enabled>false</enabled>
>> </workflow>
>>
>> ...and try the testing. The configuration simply makes
>> midPoint use its own connection mechanism (c3p0) and
>> Quartz use the tomcat's connection pool.
>> Best regards,
>>
>> Pavol Mederly
>> Software developer
>> evolveum.com <http://evolveum.com>
>>
>> On 29.08.2016 20:17, Florin. Stingaciu wrote:
>>> As an update, after having the instance up for half an
>>> hour or so with the above config, I got the following
>>> errors again: http://pastebin.com/tJJVFVsu
>>> <http://pastebin.com/tJJVFVsu>
>>>
>>> Also, in the deployment, there's no workflows at all and
>>> very little user load as it's just a dev env.
>>>
>>> Thanks,
>>> -F
>>>
>>> On Mon, Aug 29, 2016 at 10:43 AM, Florin. Stingaciu
>>> <fstingaciu at mirantis.com
>>> <mailto:fstingaciu at mirantis.com>> wrote:
>>>
>>> Hey Pavol,
>>>
>>> Currently, for Quartz and miPoint, I'm using a
>>> tomcat7 defined datasource with the following config:
>>>
>>> <Resource name="jdbc/mysql" auth="Container"
>>> type="javax.sql.DataSource"
>>> username="midpoint" password="password"
>>> url="jdbc:mysql://servder:3306/midpoint_dev"
>>> driverClassName="com.mysql.jdbc.Driver"
>>> accessToUnderlyingConnectionAllowed="true"
>>> initialSize="5" maxWait="5000"
>>> maxActive="30" maxIdle="5"
>>> validationQuery="select 1"
>>> readonly="false"
>>> poolPreparedStatements="true"/>
>>>
>>> And here's my relevant config.xml:
>>>
>>> <repository>
>>> <embedded>false</embedded>
>>> <repositoryServiceFactoryClass>com.evolveum.midpoint.repo.sql.SqlRepositoryFactory</repositoryServiceFactoryClass>
>>> <hibernateHbm2ddl>validate</hibernateHbm2ddl>
>>> <hibernateDialect>com.evolveum.midpoint.repo.sql.util.MidPointMySQLDialect</hibernateDialect>
>>> <dataSource>java:comp/env/jdbc/mysql</dataSource>
>>> </repository>
>>> <taskManager>
>>> <clustered>true</clustered>
>>> <jmxUsername>midpoint</jmxUsername>
>>> <jmxPassword>password</jmxPassword>
>>> </taskManager>
>>>
>>> With this configuration, when I was trying to access
>>> the Shadows page, I got the errors mentioned in my
>>> previous message.
>>>
>>> Following your last email, I added the following
>>> <dataSource>java:comp/env/jdbc/mysql</dataSource>
>>> under <taskManager>. Now when I try to access the
>>> Shadows Page I get the following error:
>>> http://imgur.com/a/391CU <http://imgur.com/a/391CU>.
>>> Please note there are four photos in that album.
>>> Also the reason, I chose an image is due to the fact
>>> that this error does not appear anywhere in the logs
>>> and it's in the GUI only.
>>>
>>> I would also like to add that even though this error
>>> occurs, midPoint still continues to function. Before
>>> I added the <dataSource> in the taskManager section,
>>> even though I was using a datasource for the
>>> Repository, when I tried to access the shadows page,
>>> midPoint would not work anymore (due to lack of
>>> mysql connections) until I restarted tomcat.
>>>
>>> The last point I want to make is that from what I've
>>> seen so far, this issue only happens when I try to
>>> access the shadow page. I don't see this issue while
>>> trying to access any other type of repository objects.
>>>
>>> Thanks,
>>> -F
>>>
>>>
>>> On Fri, Aug 26, 2016 at 3:41 PM, Pavol Mederly
>>> <mederly at evolveum.com <mailto:mederly at evolveum.com>>
>>> wrote:
>>>
>>> Hello Florin,
>>>
>>> it seems that either midPoint (i.e. hibernate),
>>> Quartz scheduler, or Activiti BPMS use
>>> connections without properly closing them.
>>>
>>> First step to diagnose it would be perhaps to
>>> setup things like this:
>>>
>>> 1. Let midPoint and Activiti use their own data
>>> source (by configuring them with a JDBC
>>> URL): see Repository configuration
>>> <https://wiki.evolveum.com/display/midPoint/Repository+Configuration>
>>> and Workflow configuration
>>> <https://wiki.evolveum.com/display/midPoint/Workflow+configuration>
>>> wiki pages. (If you don't use approvals you
>>> can disable workflows altogether by setting
>>> "enabled" to "false".)
>>> 2. Keep Quartz schedule using tomcat data
>>> source: something like
>>>
>>> |<||taskManager||>|
>>> |||<||clustered||>true</||clustered||>
>>> <dataSource>...your data source name...</dataSource>
>>> |
>>> ...
>>> |</||taskManager||>
>>> |
>>> And you'll see if the problem would appear
>>> again. (Maybe you could even configure three
>>> Tomcat data sources; one for midPoint, second
>>> for Quartz, and the third for Activiti, if
>>> needed. But I have not done this yet; these are
>>> pure speculations of mine...)
>>>
>>> Best regards,
>>>
>>> Pavol Mederly
>>> Software developer
>>> evolveum.com <http://evolveum.com>
>>>
>>> On 24.08.2016 22:41, Florin. Stingaciu wrote:
>>>> I just rebuilt the latest snapshot
>>>> (git-v3.5devel-221-g9744fa0) and am
>>>> experiencing the same problem.
>>>>
>>>> Thanks,
>>>> -F
>>>>
>>>>
>>>> On Wed, Aug 24, 2016 at 1:24 PM, Florin.
>>>> Stingaciu <fstingaciu at mirantis.com
>>>> <mailto:fstingaciu at mirantis.com>> wrote:
>>>>
>>>> Upon further investigation I found that
>>>> this error only occurs when I try to access
>>>> the shadow repository via
>>>> Configuration->Repository Objects->Shadows
>>>>
>>>> It seems that something triggers the
>>>> creation of DB connections until it runs
>>>> out of available connections. For this
>>>> particular instance, I'm at
>>>> git-v3.5devel-170-g26b0644.
>>>>
>>>> Thanks,
>>>> -F
>>>>
>>>> On Wed, Aug 24, 2016 at 1:13 PM, Florin.
>>>> Stingaciu <fstingaciu at mirantis.com
>>>> <mailto:fstingaciu at mirantis.com>> wrote:
>>>>
>>>> Hello Pavol,
>>>>
>>>> After running this configuration my dev
>>>> environment with two instances up for a
>>>> couple of days, I started to get the
>>>> following error messages:
>>>> http://pastebin.com/nkxE9LNA
>>>> <http://pastebin.com/nkxE9LNA>
>>>>
>>>> It seems as though the connection pool
>>>> is not properly cleaned up. The
>>>> database logs indicate no failure and
>>>> in fact my production environment is
>>>> hooked up to the same DB instance and
>>>> is working fine.
>>>>
>>>> Any recommendations would be very
>>>> appreciated.
>>>>
>>>> Thanks,
>>>> -F
>>>>
>>>> On Mon, Aug 15, 2016 at 3:29 PM,
>>>> Florin. Stingaciu
>>>> <fstingaciu at mirantis.com
>>>> <mailto:fstingaciu at mirantis.com>> wrote:
>>>>
>>>> Awesome! Thanks for all your help
>>>> today Pavol. I will try and test in
>>>> my environment and let you know how
>>>> that goes.
>>>>
>>>> Thanks again!
>>>> -F
>>>>
>>>> On Mon, Aug 15, 2016 at 3:00 PM,
>>>> Pavol Mederly <mederly at evolveum.com
>>>> <mailto:mederly at evolveum.com>> wrote:
>>>>
>>>> Florin,
>>>>
>>>> I think I've fixed it:
>>>> https://github.com/Evolveum/midpoint/commit/c00f2979cea6f6ada0a1194b13bad51c0ecec2e3
>>>> <https://github.com/Evolveum/midpoint/commit/c00f2979cea6f6ada0a1194b13bad51c0ecec2e3>
>>>>
>>>> Cannot verify as the issue
>>>> isn't visible on my computer,
>>>> but I am quite convinced it
>>>> should be fixed now. The
>>>> suspected cause is that some
>>>> query-only transactions weren't
>>>> closed properly. C3P0 closes
>>>> them automatically but other
>>>> connection providers probably
>>>> not
>>>> <http://stackoverflow.com/questions/19396563/does-hibernates-session-close-automatically-rollback-uncommitted-transactions>.
>>>>
>>>> You can safely use the master,
>>>> or you can apply the above
>>>> patch to 3.4 yourself. The
>>>> master is something like
>>>> "3.4-support" these days
>>>> (preparing for 3.4.1), so it
>>>> should be of production-level
>>>> quality, or at least near it.
>>>> (I'm going to run full test
>>>> suite during the night, but I
>>>> don't expect any failures. Or
>>>> you can check after approx 3
>>>> hours here
>>>> <https://travis-ci.org/Evolveum/midpoint>.)
>>>>
>>>> Best regards,
>>>>
>>>> Pavol Mederly
>>>> Software developer
>>>> evolveum.com <http://evolveum.com>
>>>>
>>>> On 15.08.2016 22:57, Florin.
>>>> Stingaciu wrote:
>>>>> Hey Pavol,
>>>>>
>>>>> I uploaded the log file you
>>>>> requested to my gdrive
>>>>> (https://drive.google.com/file/d/0B9f4-vWBsztoZy1zWThGYnpGeGs/view?usp=sharing
>>>>> <https://drive.google.com/file/d/0B9f4-vWBsztoZy1zWThGYnpGeGs/view?usp=sharing>)
>>>>> It's a trace level debug so
>>>>> the file is quite large. Let
>>>>> me know if anything jumps out
>>>>> at you.
>>>>>
>>>>> Thanks,
>>>>> -F
>>>>>
>>>>> On Mon, Aug 15, 2016 at 1:07
>>>>> PM, Pavol Mederly
>>>>> <mederly at evolveum.com
>>>>> <mailto:mederly at evolveum.com>>
>>>>> wrote:
>>>>>
>>>>> Strange enough. I've
>>>>> replicated your setup on
>>>>> my computer (using
>>>>> PostgreSQL, as I have no
>>>>> MySQL currently
>>>>> installed), and it works:
>>>>> midPoint starts cleanly. I
>>>>> am able to add/delete
>>>>> repository objects.
>>>>>
>>>>> Attached are my config.xml
>>>>> and server.xml; they are
>>>>> pretty much the same as yours.
>>>>>
>>>>> So, I have no idea. You
>>>>> could try adding
>>>>> <database>mysql</database>
>>>>> into <repository> element
>>>>> in your config.xml, but it
>>>>> would most probably make
>>>>> no difference.
>>>>>
>>>>> As last resort, you could
>>>>> turn on the most detailed
>>>>> debugging for the
>>>>> repository on startup by
>>>>> adding a line like this
>>>>>
>>>>> <logger
>>>>> name="com.evolveum.midpoint.re
>>>>> <http://com.evolveum.midpoint.re>po"
>>>>> level="TRACE" />
>>>>>
>>>>> into
>>>>> webapps\midpoint\WEB-INF\classes\logback.xml
>>>>> file and start midPoint.
>>>>> In idm.log there should be
>>>>> quite a detailed record of
>>>>> what's going on.
>>>>>
>>>>> Pavol Mederly
>>>>> Software developer
>>>>> evolveum.com
>>>>> <http://evolveum.com>
>>>>>
>>>>> On 15.08.2016 21:03,
>>>>> Florin. Stingaciu wrote:
>>>>>> Here's my datasource config:
>>>>>>
>>>>>> <Resource
>>>>>> name="jdbc/mysql"
>>>>>> auth="Container"
>>>>>> type="javax.sql.DataSource"
>>>>>> username="midpoint"
>>>>>> password="pass"
>>>>>> url="jdbc:mysql://SERVER:3306/midpoint_dev"
>>>>>> driverClassName="com.mysql.jdbc.Driver"
>>>>>> accessToUnderlyingConnectionAllowed="true"
>>>>>> initialSize="5"
>>>>>> maxWait="5000"
>>>>>> maxActive="30" maxIdle="5"
>>>>>> validationQuery="select 1"
>>>>>> poolPreparedStatements="true"/>
>>>>>>
>>>>>> And here's my config.xml:
>>>>>>
>>>>>> <repository>
>>>>>> <embedded>false</embedded>
>>>>>> <repositoryServiceFactoryClass>com.evolveum.midpoint.repo.sql.SqlRepositoryFactory</repositoryServiceFactoryClass>
>>>>>> <hibernateHbm2ddl>validate</hibernateHbm2ddl>
>>>>>> <hibernateDialect>com.evolveum.midpoint.repo.sql.util.MidPointMySQLDialect</hibernateDialect>
>>>>>> <dataSource>java:comp/env/jdbc/mysql</dataSource>
>>>>>> </repository>
>>>>>> <taskManager>
>>>>>> <clustered>true</clustered>
>>>>>> <jmxUsername>midpoint</jmxUsername>
>>>>>> <jmxPassword>password</jmxPassword>
>>>>>> </taskManager>
>>>>>>
>>>>>> On Mon, Aug 15, 2016 at
>>>>>> 11:44 AM, Pavol Mederly
>>>>>> <mederly at evolveum.com
>>>>>> <mailto:mederly at evolveum.com>>
>>>>>> wrote:
>>>>>>
>>>>>> Hello Florin,
>>>>>>
>>>>>> this is really
>>>>>> interesting. Please,
>>>>>> could you also share
>>>>>> your midPoint
>>>>>> config.xml, as well
>>>>>> as your data source
>>>>>> configuration?
>>>>>> (except credentials,
>>>>>> of course)
>>>>>>
>>>>>> Best regards,
>>>>>>
>>>>>> Pavol Mederly
>>>>>> Software developer
>>>>>> evolveum.com
>>>>>> <http://evolveum.com>
>>>>>>
>>>>>> On 15.08.2016 19:39,
>>>>>> Florin. Stingaciu wrote:
>>>>>>> Hello Pavol,
>>>>>>>
>>>>>>> Thanks for your
>>>>>>> detailed response. I
>>>>>>> tried setting up the
>>>>>>> datasource with
>>>>>>> validationQuery set
>>>>>>> up properly for the
>>>>>>> mySQL backed I have.
>>>>>>> However, upon
>>>>>>> service restart I
>>>>>>> get the following
>>>>>>> errors:
>>>>>>> http://pastebin.com/8dpGN0JC
>>>>>>> <http://pastebin.com/8dpGN0JC>
>>>>>>>
>>>>>>> To save you a click,
>>>>>>> it seems as though
>>>>>>> the connection is
>>>>>>> set up in readonly
>>>>>>> mode or some other
>>>>>>> strange things
>>>>>>> happen. I've tried
>>>>>>> setting
>>>>>>> readonly="false" in
>>>>>>> the resource
>>>>>>> definition in
>>>>>>> server.xml but that
>>>>>>> didn't help. I will
>>>>>>> continue researching
>>>>>>> this, however any
>>>>>>> guidance would be
>>>>>>> quite appreciated.
>>>>>>>
>>>>>>> Thanks!
>>>>>>> -F
>>>>>>>
>>>>>>> On Mon, Aug 15, 2016
>>>>>>> at 2:29 AM, Pavol
>>>>>>> Mederly
>>>>>>> <mederly at evolveum.com
>>>>>>> <mailto:mederly at evolveum.com>>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hello Florin,
>>>>>>>
>>>>>>> having looked at
>>>>>>> your logs, it
>>>>>>> seems that maybe
>>>>>>> explicit setting
>>>>>>> of
>>>>>>> validationQuery
>>>>>>> in Quartz data
>>>>>>> source setup
>>>>>>> would help. (See
>>>>>>> http://www.quartz-scheduler.org/documentation/quartz-2.x/configuration/ConfigDataSources.html
>>>>>>> <http://www.quartz-scheduler.org/documentation/quartz-2.x/configuration/ConfigDataSources.html>.)
>>>>>>>
>>>>>>> Unfortunately,
>>>>>>> current midPoint
>>>>>>> implementation
>>>>>>> does not allow
>>>>>>> to configure
>>>>>>> Quartz data
>>>>>>> source
>>>>>>> parameters. So,
>>>>>>> there are the
>>>>>>> following three
>>>>>>> possibilities:
>>>>>>>
>>>>>>> 1. Take an
>>>>>>> alternative
>>>>>>> route, and
>>>>>>> use
>>>>>>> application
>>>>>>> server-defined
>>>>>>> data source
>>>>>>> (with
>>>>>>> validationQuery
>>>>>>> set up).
>>>>>>> 2. Patch Task
>>>>>>> Manager
>>>>>>> implementation
>>>>>>> by adding
>>>>>>> appropriate
>>>>>>> lines to
>>>>>>> Quartz
>>>>>>> configuration
>>>>>>> (see
>>>>>>> LocalNodeManager.java:87-90).
>>>>>>> 3. Wait until
>>>>>>> we implement
>>>>>>> it - I've
>>>>>>> created an
>>>>>>> issue
>>>>>>> MID-3347
>>>>>>> <https://jira.evolveum.com/browse/MID-3347>
>>>>>>> for this.
>>>>>>>
>>>>>>> As for the first
>>>>>>> one (externally
>>>>>>> defined data
>>>>>>> source): Please
>>>>>>> see
>>>>>>> https://wiki.evolveum.com/display/midPoint/Repository+Configuration#RepositoryConfiguration-Datasourceconfiguration
>>>>>>> <https://wiki.evolveum.com/display/midPoint/Repository+Configuration#RepositoryConfiguration-Datasourceconfiguration>
>>>>>>> on how to
>>>>>>> configure
>>>>>>> midPoint
>>>>>>> repository with
>>>>>>> the data source.
>>>>>>> This data source
>>>>>>> will be used
>>>>>>> also by Quartz,
>>>>>>> if not overriden
>>>>>>> in <taskManager>
>>>>>>> section. It
>>>>>>> should work but
>>>>>>> I don't remember
>>>>>>> if someone
>>>>>>> actually tested
>>>>>>> this.
>>>>>>>
>>>>>>> Concerning
>>>>>>> autoReconnect: I
>>>>>>> have no
>>>>>>> experiences with
>>>>>>> this setting. I
>>>>>>> agree with you
>>>>>>> that about the
>>>>>>> hesitation of
>>>>>>> using it in
>>>>>>> production
>>>>>>> environment. If
>>>>>>> really needed,
>>>>>>> I'd recommend to
>>>>>>> separate
>>>>>>> midPoint
>>>>>>> repository
>>>>>>> configuration
>>>>>>> from Quartz
>>>>>>> configuration by
>>>>>>> using different
>>>>>>> JDBC URLs for
>>>>>>> the two: the
>>>>>>> standard one for
>>>>>>> the repository
>>>>>>> and the one with
>>>>>>> "autoReconnect=true"
>>>>>>> for Quartz. In
>>>>>>> this way,
>>>>>>> potential
>>>>>>> negative effects
>>>>>>> should be
>>>>>>> restricted to
>>>>>>> task management
>>>>>>> functionality
>>>>>>> only. But,
>>>>>>> overall, I'd
>>>>>>> suggest trying
>>>>>>> to eliminate the
>>>>>>> problem by
>>>>>>> setting
>>>>>>> validationQuery
>>>>>>> first.
>>>>>>>
>>>>>>> Best regards,
>>>>>>>
>>>>>>> Pavol Mederly
>>>>>>> Software developer
>>>>>>> evolveum.com
>>>>>>> <http://evolveum.com>
>>>>>>>
>>>>>>> On 14.08.2016
>>>>>>> 22:40, Florin.
>>>>>>> Stingaciu wrote:
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> I'm trying to
>>>>>>>> configure an
>>>>>>>> active/active
>>>>>>>> configuration,
>>>>>>>> and
>>>>>>>> experiencing
>>>>>>>> some issues
>>>>>>>> with the Quartz
>>>>>>>> scheduler. The
>>>>>>>> SQL connection
>>>>>>>> seems to
>>>>>>>> timeout quite
>>>>>>>> often and
>>>>>>>> result in many
>>>>>>>> warning
>>>>>>>> messages. I'm
>>>>>>>> also
>>>>>>>> experiencing
>>>>>>>> some errors --
>>>>>>>> as the timeout
>>>>>>>> closes the
>>>>>>>> connection,
>>>>>>>> some processes
>>>>>>>> are still
>>>>>>>> trying to
>>>>>>>> commit using
>>>>>>>> that stale
>>>>>>>> handler.
>>>>>>>>
>>>>>>>> Here are some
>>>>>>>> relevant logs
>>>>>>>> from both
>>>>>>>> midPoint
>>>>>>>> instances in
>>>>>>>> this cluster:
>>>>>>>> http://pastebin.com/JFjqrLnT
>>>>>>>> <http://pastebin.com/JFjqrLnT>
>>>>>>>> http://pastebin.com/1vy4iYPY
>>>>>>>> <http://pastebin.com/1vy4iYPY>
>>>>>>>>
>>>>>>>> Also here's the
>>>>>>>> relevant
>>>>>>>> portion of my
>>>>>>>> config.xml
>>>>>>>> file:
>>>>>>>> http://pastebin.com/WbydcCC8
>>>>>>>> <http://pastebin.com/WbydcCC8>
>>>>>>>>
>>>>>>>> Following the
>>>>>>>> suggestions in
>>>>>>>> the warnings,
>>>>>>>> I've started to
>>>>>>>> look at
>>>>>>>> autoReconnect
>>>>>>>> propriety of
>>>>>>>> the JDBC
>>>>>>>> connector and
>>>>>>>> applied it to
>>>>>>>> my configs like so:
>>>>>>>>
>>>>>>>> <jdbcUrl>jdbc:mysql://SERVER:3306/midpoint_dev?autoReconnect=true</jdbcUrl>
>>>>>>>>
>>>>>>>>
>>>>>>>> Since
>>>>>>>> implementing
>>>>>>>> this change,
>>>>>>>> the errors and
>>>>>>>> warnings seem
>>>>>>>> to have
>>>>>>>> disappeared. I
>>>>>>>> will continue
>>>>>>>> to monitor the
>>>>>>>> logs and ensure
>>>>>>>> this actually
>>>>>>>> the case.
>>>>>>>>
>>>>>>>> Reading the
>>>>>>>> mysql docs, I
>>>>>>>> found that this
>>>>>>>> is not
>>>>>>>> recommended as
>>>>>>>> this may cause
>>>>>>>> data
>>>>>>>> inconsistency
>>>>>>>> issues and that
>>>>>>>> stale
>>>>>>>> connection
>>>>>>>> exceptions
>>>>>>>> should be
>>>>>>>> properly caught
>>>>>>>> within the
>>>>>>>> application.
>>>>>>>> Namely:
>>>>>>>>
>>>>>>>> The use of
>>>>>>>> this
>>>>>>>> feature is
>>>>>>>> not
>>>>>>>> recommended,
>>>>>>>> because it
>>>>>>>> has side
>>>>>>>> effects
>>>>>>>> related to
>>>>>>>> session
>>>>>>>> state and
>>>>>>>> data
>>>>>>>> consistency
>>>>>>>> when
>>>>>>>> applications
>>>>>>>> don't
>>>>>>>> handle
>>>>>>>> SQLExceptions
>>>>>>>> properly,
>>>>>>>> and is only
>>>>>>>> designed to
>>>>>>>> be used
>>>>>>>> when you
>>>>>>>> are unable
>>>>>>>> to
>>>>>>>> configure
>>>>>>>> your
>>>>>>>> application
>>>>>>>> to handle
>>>>>>>> SQLExceptions
>>>>>>>> resulting
>>>>>>>> from dead
>>>>>>>> and stale
>>>>>>>> connections
>>>>>>>> properly.
>>>>>>>>
>>>>>>>>
>>>>>>>> Do you have any
>>>>>>>> recommended
>>>>>>>> configuration
>>>>>>>> for this
>>>>>>>> scenario? I'd
>>>>>>>> like to move my
>>>>>>>> current
>>>>>>>> production
>>>>>>>> environment in
>>>>>>>> an active
>>>>>>>> active
>>>>>>>> configuration,
>>>>>>>> however as of
>>>>>>>> right now I'm
>>>>>>>> hesitant to do
>>>>>>>> so in order to
>>>>>>>> avoid any data
>>>>>>>> corruption.
>>>>>>>> Especially
>>>>>>>> since it's
>>>>>>>> quite difficult
>>>>>>>> to test for
>>>>>>>> data
>>>>>>>> consistency
>>>>>>>> issues that may
>>>>>>>> arise from
>>>>>>>> using
>>>>>>>> autoReconnect
>>>>>>>> in my dev
>>>>>>>> environment.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> -F
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> midPoint mailing list
>>>>>>>> midPoint at lists.evolveum.com
>>>>>>>> <mailto:midPoint at lists.evolveum.com>
>>>>>>>> http://lists.evolveum.com/mailman/listinfo/midpoint
>>>>>>>> <http://lists.evolveum.com/mailman/listinfo/midpoint>
>>>>>>> _______________________________________________
>>>>>>> midPoint mailing
>>>>>>> list
>>>>>>> midPoint at lists.evolveum.com
>>>>>>> <mailto:midPoint at lists.evolveum.com>
>>>>>>> http://lists.evolveum.com/mailman/listinfo/midpoint
>>>>>>> <http://lists.evolveum.com/mailman/listinfo/midpoint>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> midPoint mailing list
>>>>>>> midPoint at lists.evolveum.com
>>>>>>> <mailto:midPoint at lists.evolveum.com>
>>>>>>> http://lists.evolveum.com/mailman/listinfo/midpoint
>>>>>>> <http://lists.evolveum.com/mailman/listinfo/midpoint>
>>>>>> _______________________________________________
>>>>>> midPoint mailing list
>>>>>> midPoint at lists.evolveum.com
>>>>>> <mailto:midPoint at lists.evolveum.com>
>>>>>> http://lists.evolveum.com/mailman/listinfo/midpoint
>>>>>> <http://lists.evolveum.com/mailman/listinfo/midpoint>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> midPoint mailing list
>>>>>> midPoint at lists.evolveum.com
>>>>>> <mailto:midPoint at lists.evolveum.com>
>>>>>> http://lists.evolveum.com/mailman/listinfo/midpoint
>>>>>> <http://lists.evolveum.com/mailman/listinfo/midpoint>
>>>>> _______________________________________________
>>>>> midPoint mailing list
>>>>> midPoint at lists.evolveum.com
>>>>> <mailto:midPoint at lists.evolveum.com>
>>>>> http://lists.evolveum.com/mailman/listinfo/midpoint
>>>>> <http://lists.evolveum.com/mailman/listinfo/midpoint>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> midPoint mailing list
>>>>> midPoint at lists.evolveum.com
>>>>> <mailto:midPoint at lists.evolveum.com>
>>>>> http://lists.evolveum.com/mailman/listinfo/midpoint
>>>>> <http://lists.evolveum.com/mailman/listinfo/midpoint>
>>>> _______________________________________________
>>>> midPoint mailing list
>>>> midPoint at lists.evolveum.com
>>>> <mailto:midPoint at lists.evolveum.com>
>>>> http://lists.evolveum.com/mailman/listinfo/midpoint
>>>> <http://lists.evolveum.com/mailman/listinfo/midpoint>
>>>>
>>>>
>>>> _______________________________________________
>>>> midPoint mailing list
>>>> midPoint at lists.evolveum.com
>>>> <mailto:midPoint at lists.evolveum.com>
>>>> http://lists.evolveum.com/mailman/listinfo/midpoint
>>>> <http://lists.evolveum.com/mailman/listinfo/midpoint>
>>> _______________________________________________
>>> midPoint mailing list
>>> midPoint at lists.evolveum.com
>>> <mailto:midPoint at lists.evolveum.com>
>>> http://lists.evolveum.com/mailman/listinfo/midpoint
>>> <http://lists.evolveum.com/mailman/listinfo/midpoint>
>>>
>>>
>>> _______________________________________________
>>> midPoint mailing list
>>> midPoint at lists.evolveum.com
>>> <mailto:midPoint at lists.evolveum.com>
>>> http://lists.evolveum.com/mailman/listinfo/midpoint
>>> <http://lists.evolveum.com/mailman/listinfo/midpoint>
>> _______________________________________________ midPoint
>> mailing list midPoint at lists.evolveum.com
>> <mailto:midPoint at lists.evolveum.com>
>> http://lists.evolveum.com/mailman/listinfo/midpoint
>> <http://lists.evolveum.com/mailman/listinfo/midpoint>
>>
>> _______________________________________________
>> midPoint mailing list
>> midPoint at lists.evolveum.com <mailto:midPoint at lists.evolveum.com>
>> http://lists.evolveum.com/mailman/listinfo/midpoint
>> <http://lists.evolveum.com/mailman/listinfo/midpoint>
> _______________________________________________ midPoint mailing
> list midPoint at lists.evolveum.com
> <mailto:midPoint at lists.evolveum.com>
> http://lists.evolveum.com/mailman/listinfo/midpoint
> <http://lists.evolveum.com/mailman/listinfo/midpoint>
>
> _______________________________________________
> midPoint mailing list
> midPoint at lists.evolveum.com
> http://lists.evolveum.com/mailman/listinfo/midpoint
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20160830/ba103e2e/attachment.htm>
More information about the midPoint
mailing list