<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>Hello Matt,</p>
<p>I have no direct experience with deploying midPoint with large
number of users, but let me give you a few hints:</p>
<p>1) If GUI is dead, maybe REST would allow you to check the task's
state; at
<a class="moz-txt-link-freetext" href="http://localhost:8080/midpoint/ws/rest/tasks/">http://localhost:8080/midpoint/ws/rest/tasks/</a><oid>, e.g.
<a class="moz-txt-link-freetext" href="http://localhost:8080/midpoint/ws/rest/tasks/00000000-0000-0000-0000-000000000005">http://localhost:8080/midpoint/ws/rest/tasks/00000000-0000-0000-0000-000000000005</a></p>
<p>2) If REST is unresponsive as well, or if there's need to know
more about what's going on, I sometimes use jvisualvm - to see
e.g. heap utilization, thread state, even collecting thread dump,
etc. Maybe you could use it to see if there's a memleak somewhere
(as Radovan said).</p>
<p>Its output looks like this:</p>
<img src="cid:part1.C04B14B9.A4C696E1@evolveum.com" alt=""><br>
<br>
Best regards,<br>
<pre class="moz-signature" cols="72">Pavol Mederly
Software developer
evolveum.com
</pre>
<div class="moz-cite-prefix">On 26.08.2016 17:25, Mencel, Matt
wrote:<br>
</div>
<blockquote
cite="mid:CABpVbHL-1P4tmeZ=oSrijb+YN3cgscSzF_TniDGaaPjFC=T9ww@mail.gmail.com"
type="cite">
<div dir="ltr">New entries in the idm.log appeared about 40
minutes after it stopped logging. My -Xmx setting in JAVA_OPTS
is set to 4096.
<div><br>
</div>
<div>Matt<br>
<div><br>
</div>
<div>
<div>2016-08-26 10:06:51,393 []
[C3P0PooledConnectionPoolManager[identityToken->2tu6q89i1j961p54hnz75|180fb7dc]-AdminTaskTimer]
WARN (com.mchange.v2.async.ThreadPoolAsynchronousRunner):
com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@2f41b4ad
-- APPARENT DEADLOCK!!! Creating emergency threads for
unassigned pending tasks!</div>
<div>2016-08-26 10:12:30,979 [] [NioProcessor-1] WARN
(org.apache.directory.ldap.client.api.LdapNetworkConnection):
Java heap space</div>
<div>java.lang.OutOfMemoryError: Java heap space</div>
<div> at
org.apache.directory.api.util.Strings.utf8ToString(Strings.java:724)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.util.Strings.toLowerCase(Strings.java:1890)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.ldap.model.entry.DefaultAttribute.setUpId(DefaultAttribute.java:637)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.ldap.model.entry.DefaultAttribute.setUpId(DefaultAttribute.java:575)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.ldap.model.entry.DefaultAttribute.<init>(DefaultAttribute.java:192)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.ldap.codec.decorators.SearchResultEntryDecorator.addAttribute(SearchResultEntryDecorator.java:112)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.ldap.codec.actions.searchResultEntry.AddAttributeType.action(AddAttributeType.java:87)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.ldap.codec.actions.searchResultEntry.AddAttributeType.action(AddAttributeType.java:48)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.asn1.ber.grammar.AbstractGrammar.executeAction(AbstractGrammar.java:136)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.asn1.ber.Asn1Decoder.treatTLVDoneState(Asn1Decoder.java:600)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.asn1.ber.Asn1Decoder.decode(Asn1Decoder.java:736)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.ldap.codec.protocol.mina.LdapProtocolDecoder.decode(LdapProtocolDecoder.java:145)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.ldap.codec.protocol.mina.LdapProtocolDecoder.decode(LdapProtocolDecoder.java:94)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:230)
~[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:542)
[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:48)
[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:943)
[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:109)
[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:542)
[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:535)
[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:697)
[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:651)
[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:640)
[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.polling.AbstractPollingIoProcessor.access$600(AbstractPollingIoProcessor.java:68)
[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1097)
[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
[mina-core-2.0.13.jar:na]</div>
<div> at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_101]</div>
<div> at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_101]</div>
<div> at java.lang.Thread.run(Thread.java:745)
[na:1.8.0_101]</div>
<div>2016-08-26 10:12:30,980 [] [NioProcessor-1] WARN
(org.apache.directory.ldap.client.api.LdapNetworkConnection):
Java heap space</div>
<div>java.lang.OutOfMemoryError: Java heap space</div>
<div> at
org.apache.directory.api.util.Strings.utf8ToString(Strings.java:724)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.util.Strings.toLowerCase(Strings.java:1890)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.ldap.model.entry.DefaultAttribute.setUpId(DefaultAttribute.java:637)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.ldap.model.entry.DefaultAttribute.setUpId(DefaultAttribute.java:575)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.ldap.model.entry.DefaultAttribute.<init>(DefaultAttribute.java:192)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.ldap.codec.decorators.SearchResultEntryDecorator.addAttribute(SearchResultEntryDecorator.java:112)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.ldap.codec.actions.searchResultEntry.AddAttributeType.action(AddAttributeType.java:87)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.ldap.codec.actions.searchResultEntry.AddAttributeType.action(AddAttributeType.java:48)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.asn1.ber.grammar.AbstractGrammar.executeAction(AbstractGrammar.java:136)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.asn1.ber.Asn1Decoder.treatTLVDoneState(Asn1Decoder.java:600)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.asn1.ber.Asn1Decoder.decode(Asn1Decoder.java:736)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.ldap.codec.protocol.mina.LdapProtocolDecoder.decode(LdapProtocolDecoder.java:145)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.ldap.codec.protocol.mina.LdapProtocolDecoder.decode(LdapProtocolDecoder.java:94)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:230)
~[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:542)
[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:48)
[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:943)
~[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:109)
~[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:542)
[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:535)
[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:697)
[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:651)
[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:640)
[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.polling.AbstractPollingIoProcessor.access$600(AbstractPollingIoProcessor.java:68)
[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1097)
[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
[mina-core-2.0.13.jar:na]</div>
<div> at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_101]</div>
<div> at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_101]</div>
<div> at java.lang.Thread.run(Thread.java:745)
[na:1.8.0_101]</div>
<div>2016-08-26 10:12:30,981 [] [NioProcessor-2] WARN
(org.apache.directory.ldap.client.api.LdapNetworkConnection):
Java heap space</div>
<div>java.lang.OutOfMemoryError: Java heap space</div>
<div> at
java.nio.charset.CharsetEncoder.<init>(CharsetEncoder.java:233)
~[na:1.8.0_101]</div>
<div> at
sun.nio.cs.UTF_8$Encoder.<init>(UTF_8.java:558)
~[na:1.8.0_101]</div>
<div> at
sun.nio.cs.UTF_8$Encoder.<init>(UTF_8.java:554)
~[na:1.8.0_101]</div>
<div> at sun.nio.cs.UTF_8.newEncoder(UTF_8.java:72)
~[na:1.8.0_101]</div>
<div> at
java.lang.StringCoding.encode(StringCoding.java:348)
~[na:1.8.0_101]</div>
<div> at java.lang.String.getBytes(String.java:941)
~[na:1.8.0_101]</div>
<div> at
org.apache.directory.api.util.Strings.getBytesUtf8(Strings.java:1578)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.ldap.model.entry.StringValue.<init>(StringValue.java:93)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.ldap.model.entry.DefaultAttribute.createStringValue(DefaultAttribute.java:93)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.ldap.model.entry.DefaultAttribute.add(DefaultAttribute.java:1064)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.ldap.codec.decorators.SearchResultEntryDecorator.addAttributeValue(SearchResultEntryDecorator.java:127)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.ldap.codec.actions.searchResultEntry.StoreSearchResultAttributeValue.action(StoreSearchResultAttributeValue.java:100)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.ldap.codec.actions.searchResultEntry.StoreSearchResultAttributeValue.action(StoreSearchResultAttributeValue.java:44)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.asn1.ber.grammar.AbstractGrammar.executeAction(AbstractGrammar.java:136)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.asn1.ber.Asn1Decoder.treatTLVDoneState(Asn1Decoder.java:600)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.asn1.ber.Asn1Decoder.decode(Asn1Decoder.java:736)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.ldap.codec.protocol.mina.LdapProtocolDecoder.decode(LdapProtocolDecoder.java:145)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.ldap.codec.protocol.mina.LdapProtocolDecoder.decode(LdapProtocolDecoder.java:94)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:230)
~[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:542)
~[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:48)
~[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:943)
~[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:109)
~[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:542)
~[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:535)
~[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:697)
~[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:651)
~[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:640)
~[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.polling.AbstractPollingIoProcessor.access$600(AbstractPollingIoProcessor.java:68)
~[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1097)
~[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
~[mina-core-2.0.13.jar:na]</div>
<div> at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_101]</div>
<div>2016-08-26 10:12:30,982 [] [NioProcessor-2] WARN
(org.apache.directory.ldap.client.api.LdapNetworkConnection):
Java heap space</div>
<div>java.lang.OutOfMemoryError: Java heap space</div>
<div> at
java.nio.charset.CharsetEncoder.<init>(CharsetEncoder.java:233)
~[na:1.8.0_101]</div>
<div> at
sun.nio.cs.UTF_8$Encoder.<init>(UTF_8.java:558)
~[na:1.8.0_101]</div>
<div> at
sun.nio.cs.UTF_8$Encoder.<init>(UTF_8.java:554)
~[na:1.8.0_101]</div>
<div> at sun.nio.cs.UTF_8.newEncoder(UTF_8.java:72)
~[na:1.8.0_101]</div>
<div> at
java.lang.StringCoding.encode(StringCoding.java:348)
~[na:1.8.0_101]</div>
<div> at java.lang.String.getBytes(String.java:941)
~[na:1.8.0_101]</div>
<div> at
org.apache.directory.api.util.Strings.getBytesUtf8(Strings.java:1578)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.ldap.model.entry.StringValue.<init>(StringValue.java:93)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.ldap.model.entry.DefaultAttribute.createStringValue(DefaultAttribute.java:93)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.ldap.model.entry.DefaultAttribute.add(DefaultAttribute.java:1064)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.ldap.codec.decorators.SearchResultEntryDecorator.addAttributeValue(SearchResultEntryDecorator.java:127)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.ldap.codec.actions.searchResultEntry.StoreSearchResultAttributeValue.action(StoreSearchResultAttributeValue.java:100)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.ldap.codec.actions.searchResultEntry.StoreSearchResultAttributeValue.action(StoreSearchResultAttributeValue.java:44)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.asn1.ber.grammar.AbstractGrammar.executeAction(AbstractGrammar.java:136)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.asn1.ber.Asn1Decoder.treatTLVDoneState(Asn1Decoder.java:600)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.asn1.ber.Asn1Decoder.decode(Asn1Decoder.java:736)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.ldap.codec.protocol.mina.LdapProtocolDecoder.decode(LdapProtocolDecoder.java:145)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.directory.api.ldap.codec.protocol.mina.LdapProtocolDecoder.decode(LdapProtocolDecoder.java:94)
~[api-all-1.0.0-RC1.jar:1.0.0-RC1]</div>
<div> at
org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:230)
~[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:542)
~[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:48)
~[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:943)
~[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:109)
~[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:542)
~[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:535)
~[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:697)
~[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:651)
~[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:640)
~[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.polling.AbstractPollingIoProcessor.access$600(AbstractPollingIoProcessor.java:68)
~[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1097)
~[mina-core-2.0.13.jar:na]</div>
<div> at
org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
~[mina-core-2.0.13.jar:na]</div>
<div> at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_101]</div>
<div>2016-08-26 10:13:22,931 []
[C3P0PooledConnectionPoolManager[identityToken->2tu6q89i1j961p54hnz75|180fb7dc]-AdminTaskTimer]
WARN (com.mchange.v2.async.ThreadPoolAsynchronousRunner):
com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@2f41b4ad
-- APPARENT DEADLOCK!!! Complete Status:</div>
<div> Managed Threads: 3</div>
<div> Active Threads: 3</div>
<div> Active Tasks:</div>
<div>
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@5b637151</div>
<div> on thread:
C3P0PooledConnectionPoolManager[identityToken->2tu6q89i1j961p54hnz75|180fb7dc]-HelperThread-#2</div>
<div>
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@79a5524c</div>
<div> on thread:
C3P0PooledConnectionPoolManager[identityToken->2tu6q89i1j961p54hnz75|180fb7dc]-HelperThread-#0</div>
<div>
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@a6cc06e</div>
<div> on thread:
C3P0PooledConnectionPoolManager[identityToken->2tu6q89i1j961p54hnz75|180fb7dc]-HelperThread-#1</div>
<div> Pending Tasks:</div>
<div>
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@716cf0fa</div>
<div>
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@3145fe7f</div>
<div>
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@62a1fd29</div>
<div>
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@e223ea3</div>
<div>
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@185ff4bd</div>
<div>
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@6f757cf</div>
<div>
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@6b52ba8d</div>
<div>
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@749a4662</div>
<div>
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@1307ae50</div>
<div>
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@2f72cad0</div>
<div>
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@510a9b61</div>
<div>
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@77057e74</div>
<div>
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@595e2bea</div>
<div>
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@60c34e5b</div>
<div>
com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask@659710b1</div>
<div>Pool thread stack traces:</div>
<div>
Thread[C3P0PooledConnectionPoolManager[identityToken->2tu6q89i1j961p54hnz75|180fb7dc]-HelperThread-#2,5,main]</div>
<div>
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:674)</div>
<div>
Thread[C3P0PooledConnectionPoolManager[identityToken->2tu6q89i1j961p54hnz75|180fb7dc]-HelperThread-#1,5,main]</div>
<div>
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:674)</div>
<div>
Thread[C3P0PooledConnectionPoolManager[identityToken->2tu6q89i1j961p54hnz75|180fb7dc]-HelperThread-#0,5,main]</div>
<div>
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:672)</div>
<div><br>
</div>
<div><br>
</div>
<div>2016-08-26 10:13:51,355 [] [Thread-9] ERROR
(org.activiti.engine.impl.jobexecutor.AcquireJobsRunnableImpl):
exception during job acquisition: Java heap space</div>
<div>java.lang.OutOfMemoryError: Java heap space</div>
<div> at
java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1019)
~[na:1.8.0_101]</div>
<div> at
java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
~[na:1.8.0_101]</div>
<div> at
sun.util.resources.ParallelListResourceBundle.loadLookupTablesIfNecessary(ParallelListResourceBundle.java:169)
~[na:1.8.0_101]</div>
<div> at
sun.util.resources.ParallelListResourceBundle.handleKeySet(ParallelListResourceBundle.java:134)
~[na:1.8.0_101]</div>
<div> at
sun.util.resources.ParallelListResourceBundle.keySet(ParallelListResourceBundle.java:143)
~[na:1.8.0_101]</div>
<div> at
sun.util.resources.ParallelListResourceBundle.containsKey(ParallelListResourceBundle.java:129)
~[na:1.8.0_101]</div>
<div> at
sun.util.resources.ParallelListResourceBundle$KeySet.contains(ParallelListResourceBundle.java:208)
~[na:1.8.0_101]</div>
<div> at
sun.util.resources.ParallelListResourceBundle.containsKey(ParallelListResourceBundle.java:129)
~[na:1.8.0_101]</div>
<div> at
sun.util.resources.ParallelListResourceBundle$KeySet.contains(ParallelListResourceBundle.java:208)
~[na:1.8.0_101]</div>
<div> at
sun.util.resources.ParallelListResourceBundle.containsKey(ParallelListResourceBundle.java:129)
~[na:1.8.0_101]</div>
<div> at
java.text.DateFormatSymbols.initializeData(DateFormatSymbols.java:702)
~[na:1.8.0_101]</div>
<div> at
java.text.DateFormatSymbols.<init>(DateFormatSymbols.java:146)
~[na:1.8.0_101]</div>
<div> at
sun.util.locale.provider.DateFormatSymbolsProviderImpl.getInstance(DateFormatSymbolsProviderImpl.java:85)
~[na:1.8.0_101]</div>
<div> at
java.text.DateFormatSymbols.getProviderInstance(DateFormatSymbols.java:359)
~[na:1.8.0_101]</div>
<div> at
java.text.DateFormatSymbols.getInstanceRef(DateFormatSymbols.java:349)
~[na:1.8.0_101]</div>
<div> at
java.text.SimpleDateFormat.<init>(SimpleDateFormat.java:603)
~[na:1.8.0_101]</div>
<div> at
com.mysql.jdbc.PreparedStatement.setTimestampInternal(PreparedStatement.java:4224)
~[mysql-connector-java-5.1.39-bin.jar:5.1.39]</div>
<div> at
com.mysql.jdbc.PreparedStatement.setTimestamp(PreparedStatement.java:4183)
~[mysql-connector-java-5.1.39-bin.jar:5.1.39]</div>
<div> at
org.apache.ibatis.type.DateTypeHandler.setNonNullParameter(DateTypeHandler.java:33)
~[mybatis-3.3.0.jar:3.3.0]</div>
<div> at
org.apache.ibatis.type.DateTypeHandler.setNonNullParameter(DateTypeHandler.java:28)
~[mybatis-3.3.0.jar:3.3.0]</div>
<div> at
org.apache.ibatis.type.BaseTypeHandler.setParameter(BaseTypeHandler.java:53)
~[mybatis-3.3.0.jar:3.3.0]</div>
<div> at
org.apache.ibatis.type.UnknownTypeHandler.setNonNullParameter(UnknownTypeHandler.java:45)
~[mybatis-3.3.0.jar:3.3.0]</div>
<div> at
org.apache.ibatis.type.BaseTypeHandler.setParameter(BaseTypeHandler.java:53)
~[mybatis-3.3.0.jar:3.3.0]</div>
<div> at
org.apache.ibatis.scripting.defaults.DefaultParameterHandler.setParameters(DefaultParameterHandler.java:87)
~[mybatis-3.3.0.jar:3.3.0]</div>
<div> at
org.apache.ibatis.executor.statement.PreparedStatementHandler.parameterize(PreparedStatementHandler.java:85)
~[mybatis-3.3.0.jar:3.3.0]</div>
<div> at
org.apache.ibatis.executor.statement.RoutingStatementHandler.parameterize(RoutingStatementHandler.java:63)
~[mybatis-3.3.0.jar:3.3.0]</div>
<div> at
org.apache.ibatis.executor.SimpleExecutor.prepareStatement(SimpleExecutor.java:77)
~[mybatis-3.3.0.jar:3.3.0]</div>
<div> at
org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:61)
~[mybatis-3.3.0.jar:3.3.0]</div>
<div> at
org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:303)
~[mybatis-3.3.0.jar:3.3.0]</div>
<div> at
org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:154)
~[mybatis-3.3.0.jar:3.3.0]</div>
<div> at
org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:102)
~[mybatis-3.3.0.jar:3.3.0]</div>
<div> at
org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:82)
~[mybatis-3.3.0.jar:3.3.0]</div>
</div>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Aug 26, 2016 at 10:05 AM,
Mencel, Matt <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:mr-mencel@wiu.edu" target="_blank">mr-mencel@wiu.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">I just attempted my first import from LDAP.
I have a condition that limits the import to a single
department of about 80 accounts, but the import still
processes all the others.
<div><br>
</div>
<div>The import processed nearly 30000 accounts. At just
over 9000 the web UI became unresponsive. It's been 30
minutes and I still can't do anything in the UI.</div>
<div><br>
</div>
<div>My tomcat java process is still eating up the CPU,
but nothing has logged in the idm.log for the last 30
minutes. Is there some other place I can look to see
what this process is doing?</div>
<div><br>
</div>
<div>Here's the processes from htop...</div>
<div><br>
</div>
<div>
<div><font face="monospace, monospace">111390 tomcat_mi
20 0 9163M 5052M 15752 S 591. 42.1 2h08:45
/usr/bin/java -Djava.util.logging.config.<wbr>file=/opt/tomcat_midpoint/<wbr>conf/logging.properties
-Djava.util.logging.manager=<wbr>org.apache.juli.<wbr>ClassLoaderLogManager
-Djdk.tls.ephe</font></div>
<div><font face="monospace, monospace">111404 tomcat_mi
20 0 9163M 5052M 15752 R 98.3 42.1 19:45.81
/usr/bin/java -Djava.util.logging.config.<wbr>file=/opt/tomcat_midpoint/<wbr>conf/logging.properties
-Djava.util.logging.manager=<wbr>org.apache.juli.<wbr>ClassLoaderLogManager
-Djdk.tls.ephe</font></div>
<div><font face="monospace, monospace">111407 tomcat_mi
20 0 9163M 5052M 15752 R 98.3 42.1 19:42.20
/usr/bin/java -Djava.util.logging.config.<wbr>file=/opt/tomcat_midpoint/<wbr>conf/logging.properties
-Djava.util.logging.manager=<wbr>org.apache.juli.<wbr>ClassLoaderLogManager
-Djdk.tls.ephe</font></div>
<div><font face="monospace, monospace">111405 tomcat_mi
20 0 9163M 5052M 15752 R 99.0 42.1 19:42.24
/usr/bin/java -Djava.util.logging.config.<wbr>file=/opt/tomcat_midpoint/<wbr>conf/logging.properties
-Djava.util.logging.manager=<wbr>org.apache.juli.<wbr>ClassLoaderLogManager
-Djdk.tls.ephe</font></div>
<div><font face="monospace, monospace">111406 tomcat_mi
20 0 9163M 5052M 15752 R 98.3 42.1 19:47.07
/usr/bin/java -Djava.util.logging.config.<wbr>file=/opt/tomcat_midpoint/<wbr>conf/logging.properties
-Djava.util.logging.manager=<wbr>org.apache.juli.<wbr>ClassLoaderLogManager
-Djdk.tls.ephe</font></div>
<div><font face="monospace, monospace">111402 tomcat_mi
20 0 9163M 5052M 15752 R 97.7 42.1 19:44.50
/usr/bin/java -Djava.util.logging.config.<wbr>file=/opt/tomcat_midpoint/<wbr>conf/logging.properties
-Djava.util.logging.manager=<wbr>org.apache.juli.<wbr>ClassLoaderLogManager
-Djdk.tls.ephe</font></div>
<div><font face="monospace, monospace">111403 tomcat_mi
20 0 9163M 5052M 15752 R 98.3 42.1 19:43.86
/usr/bin/java -Djava.util.logging.config.<wbr>file=/opt/tomcat_midpoint/<wbr>conf/logging.properties
-Djava.util.logging.manager=<wbr>org.apache.juli.<wbr>ClassLoaderLogManager
-Djdk.tls.ephe</font></div>
<div><font face="monospace, monospace">111408 tomcat_mi
20 0 9163M 5052M 15752 S 1.3 42.1 0:53.12
/usr/bin/java -Djava.util.logging.config.<wbr>file=/opt/tomcat_midpoint/<wbr>conf/logging.properties
-Djava.util.logging.manager=<wbr>org.apache.juli.<wbr>ClassLoaderLogManager
-Djdk.tls.ephe</font></div>
</div>
<div><font face="monospace, monospace"><br>
</font></div>
<div><font face="arial, helvetica, sans-serif">Details on
one of the processes....</font></div>
<div><font face="arial, helvetica, sans-serif"># ps aux |
grep 11390</font></div>
<div><font face="monospace, monospace"><br>
</font></div>
<div><font face="monospace, monospace">tomcat_+ 111390
204 42.1 9383488 5173404 ? Ssl 08:57 133:18
/usr/bin/java -Djava.util.logging.config.<wbr>file=/opt/tomcat_midpoint/<wbr>conf/logging.properties
-Djava.util.logging.manager=<wbr>org.apache.juli.<wbr>ClassLoaderLogManager
-Djdk.tls.ephemeralDHKeySize=<wbr>2048
-Dmidpoint.home=/opt/midpoint
-Djavax.net.ssl.trustStore=/<wbr>opt/midpoint/keystore.jceks
-Djavax.net.ssl.<wbr>trustStoreType=jceks -Xms512m
-XX:MaxPermSize=1024m -Xss1m -Xmx4096m -classpath
/opt/tomcat_midpoint/bin/<wbr>bootstrap.jar:/opt/tomcat_<wbr>midpoint/bin/tomcat-juli.jar
-Dcatalina.base=/opt/tomcat_<wbr>midpoint
-Dcatalina.home=/opt/tomcat_<wbr>midpoint
-Djava.io.tmpdir=/opt/tomcat_<wbr>midpoint/temp
org.apache.catalina.startup.<wbr>Bootstrap start<br>
</font></div>
<div><font face="monospace, monospace"><br>
</font></div>
<div><font face="arial, helvetica, sans-serif">Are those
threads still just busy processing the records?</font></div>
<div><font face="arial, helvetica, sans-serif"><br>
</font></div>
<div><font face="arial, helvetica, sans-serif">Thanks,</font></div>
<div><font face="arial, helvetica, sans-serif">Matt</font></div>
</div>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
midPoint mailing list
<a class="moz-txt-link-abbreviated" href="mailto:midPoint@lists.evolveum.com">midPoint@lists.evolveum.com</a>
<a class="moz-txt-link-freetext" href="http://lists.evolveum.com/mailman/listinfo/midpoint">http://lists.evolveum.com/mailman/listinfo/midpoint</a>
</pre>
</blockquote>
<br>
</body>
</html>