[midPoint] High CPU Utilization After Import Attempt - UI Unresponsive
Pavol Mederly
mederly at evolveum.com
Fri Aug 26 17:39:59 CEST 2016
Hello Matt,
I have no direct experience with deploying midPoint with large number of
users, but let me give you a few hints:
1) If GUI is dead, maybe REST would allow you to check the task's state;
at http://localhost:8080/midpoint/ws/rest/tasks/<oid>, e.g.
http://localhost:8080/midpoint/ws/rest/tasks/00000000-0000-0000-0000-000000000005
2) If REST is unresponsive as well, or if there's need to know more
about what's going on, I sometimes use jvisualvm - to see e.g. heap
utilization, thread state, even collecting thread dump, etc. Maybe you
could use it to see if there's a memleak somewhere (as Radovan said).
Its output looks like this:
Best regards,
Pavol Mederly
Software developer
evolveum.com
On 26.08.2016 17:25, Mencel, Matt wrote:
> New entries in the idm.log appeared about 40 minutes after it stopped
> logging. My -Xmx setting in JAVA_OPTS is set to 4096.
>
> Matt
>
> 2016-08-26 10:06:51,393 []
> [C3P0PooledConnectionPoolManager[identityToken->2tu6q89i1j961p54hnz75|180fb7dc]-AdminTaskTimer]
> WARN (com.mchange.v2.async.ThreadPoolAsynchronousRunner):
> com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector at 2f41b4ad
> -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned
> pending tasks!
> 2016-08-26 10:12:30,979 [] [NioProcessor-1] WARN
> (org.apache.directory.ldap.client.api.LdapNetworkConnection): Java
> heap space
> java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.directory.api.util.Strings.utf8ToString(Strings.java:724)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.util.Strings.toLowerCase(Strings.java:1890)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.ldap.model.entry.DefaultAttribute.setUpId(DefaultAttribute.java:637)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.ldap.model.entry.DefaultAttribute.setUpId(DefaultAttribute.java:575)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.ldap.model.entry.DefaultAttribute.<init>(DefaultAttribute.java:192)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.ldap.codec.decorators.SearchResultEntryDecorator.addAttribute(SearchResultEntryDecorator.java:112)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.ldap.codec.actions.searchResultEntry.AddAttributeType.action(AddAttributeType.java:87)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.ldap.codec.actions.searchResultEntry.AddAttributeType.action(AddAttributeType.java:48)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.asn1.ber.grammar.AbstractGrammar.executeAction(AbstractGrammar.java:136)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.asn1.ber.Asn1Decoder.treatTLVDoneState(Asn1Decoder.java:600)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.asn1.ber.Asn1Decoder.decode(Asn1Decoder.java:736)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.ldap.codec.protocol.mina.LdapProtocolDecoder.decode(LdapProtocolDecoder.java:145)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.ldap.codec.protocol.mina.LdapProtocolDecoder.decode(LdapProtocolDecoder.java:94)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:230)
> ~[mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:542)
> [mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:48)
> [mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:943)
> [mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:109)
> [mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:542)
> [mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:535)
> [mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:697)
> [mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:651)
> [mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:640)
> [mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.polling.AbstractPollingIoProcessor.access$600(AbstractPollingIoProcessor.java:68)
> [mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1097)
> [mina-core-2.0.13.jar:na]
> at
> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
> [mina-core-2.0.13.jar:na]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_101]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_101]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
> 2016-08-26 10:12:30,980 [] [NioProcessor-1] WARN
> (org.apache.directory.ldap.client.api.LdapNetworkConnection): Java
> heap space
> java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.directory.api.util.Strings.utf8ToString(Strings.java:724)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.util.Strings.toLowerCase(Strings.java:1890)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.ldap.model.entry.DefaultAttribute.setUpId(DefaultAttribute.java:637)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.ldap.model.entry.DefaultAttribute.setUpId(DefaultAttribute.java:575)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.ldap.model.entry.DefaultAttribute.<init>(DefaultAttribute.java:192)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.ldap.codec.decorators.SearchResultEntryDecorator.addAttribute(SearchResultEntryDecorator.java:112)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.ldap.codec.actions.searchResultEntry.AddAttributeType.action(AddAttributeType.java:87)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.ldap.codec.actions.searchResultEntry.AddAttributeType.action(AddAttributeType.java:48)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.asn1.ber.grammar.AbstractGrammar.executeAction(AbstractGrammar.java:136)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.asn1.ber.Asn1Decoder.treatTLVDoneState(Asn1Decoder.java:600)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.asn1.ber.Asn1Decoder.decode(Asn1Decoder.java:736)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.ldap.codec.protocol.mina.LdapProtocolDecoder.decode(LdapProtocolDecoder.java:145)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.ldap.codec.protocol.mina.LdapProtocolDecoder.decode(LdapProtocolDecoder.java:94)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:230)
> ~[mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:542)
> [mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:48)
> [mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:943)
> ~[mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:109)
> ~[mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:542)
> [mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:535)
> [mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:697)
> [mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:651)
> [mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:640)
> [mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.polling.AbstractPollingIoProcessor.access$600(AbstractPollingIoProcessor.java:68)
> [mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1097)
> [mina-core-2.0.13.jar:na]
> at
> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
> [mina-core-2.0.13.jar:na]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_101]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_101]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
> 2016-08-26 10:12:30,981 [] [NioProcessor-2] WARN
> (org.apache.directory.ldap.client.api.LdapNetworkConnection): Java
> heap space
> java.lang.OutOfMemoryError: Java heap space
> at
> java.nio.charset.CharsetEncoder.<init>(CharsetEncoder.java:233)
> ~[na:1.8.0_101]
> at sun.nio.cs.UTF_8$Encoder.<init>(UTF_8.java:558) ~[na:1.8.0_101]
> at sun.nio.cs.UTF_8$Encoder.<init>(UTF_8.java:554) ~[na:1.8.0_101]
> at sun.nio.cs.UTF_8.newEncoder(UTF_8.java:72) ~[na:1.8.0_101]
> at java.lang.StringCoding.encode(StringCoding.java:348)
> ~[na:1.8.0_101]
> at java.lang.String.getBytes(String.java:941) ~[na:1.8.0_101]
> at
> org.apache.directory.api.util.Strings.getBytesUtf8(Strings.java:1578)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.ldap.model.entry.StringValue.<init>(StringValue.java:93)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.ldap.model.entry.DefaultAttribute.createStringValue(DefaultAttribute.java:93)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.ldap.model.entry.DefaultAttribute.add(DefaultAttribute.java:1064)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.ldap.codec.decorators.SearchResultEntryDecorator.addAttributeValue(SearchResultEntryDecorator.java:127)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.ldap.codec.actions.searchResultEntry.StoreSearchResultAttributeValue.action(StoreSearchResultAttributeValue.java:100)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.ldap.codec.actions.searchResultEntry.StoreSearchResultAttributeValue.action(StoreSearchResultAttributeValue.java:44)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.asn1.ber.grammar.AbstractGrammar.executeAction(AbstractGrammar.java:136)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.asn1.ber.Asn1Decoder.treatTLVDoneState(Asn1Decoder.java:600)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.asn1.ber.Asn1Decoder.decode(Asn1Decoder.java:736)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.ldap.codec.protocol.mina.LdapProtocolDecoder.decode(LdapProtocolDecoder.java:145)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.ldap.codec.protocol.mina.LdapProtocolDecoder.decode(LdapProtocolDecoder.java:94)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:230)
> ~[mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:542)
> ~[mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:48)
> ~[mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:943)
> ~[mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:109)
> ~[mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:542)
> ~[mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:535)
> ~[mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:697)
> ~[mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:651)
> ~[mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:640)
> ~[mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.polling.AbstractPollingIoProcessor.access$600(AbstractPollingIoProcessor.java:68)
> ~[mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1097)
> ~[mina-core-2.0.13.jar:na]
> at
> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
> ~[mina-core-2.0.13.jar:na]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_101]
> 2016-08-26 10:12:30,982 [] [NioProcessor-2] WARN
> (org.apache.directory.ldap.client.api.LdapNetworkConnection): Java
> heap space
> java.lang.OutOfMemoryError: Java heap space
> at
> java.nio.charset.CharsetEncoder.<init>(CharsetEncoder.java:233)
> ~[na:1.8.0_101]
> at sun.nio.cs.UTF_8$Encoder.<init>(UTF_8.java:558) ~[na:1.8.0_101]
> at sun.nio.cs.UTF_8$Encoder.<init>(UTF_8.java:554) ~[na:1.8.0_101]
> at sun.nio.cs.UTF_8.newEncoder(UTF_8.java:72) ~[na:1.8.0_101]
> at java.lang.StringCoding.encode(StringCoding.java:348)
> ~[na:1.8.0_101]
> at java.lang.String.getBytes(String.java:941) ~[na:1.8.0_101]
> at
> org.apache.directory.api.util.Strings.getBytesUtf8(Strings.java:1578)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.ldap.model.entry.StringValue.<init>(StringValue.java:93)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.ldap.model.entry.DefaultAttribute.createStringValue(DefaultAttribute.java:93)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.ldap.model.entry.DefaultAttribute.add(DefaultAttribute.java:1064)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.ldap.codec.decorators.SearchResultEntryDecorator.addAttributeValue(SearchResultEntryDecorator.java:127)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.ldap.codec.actions.searchResultEntry.StoreSearchResultAttributeValue.action(StoreSearchResultAttributeValue.java:100)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.ldap.codec.actions.searchResultEntry.StoreSearchResultAttributeValue.action(StoreSearchResultAttributeValue.java:44)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.asn1.ber.grammar.AbstractGrammar.executeAction(AbstractGrammar.java:136)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.asn1.ber.Asn1Decoder.treatTLVDoneState(Asn1Decoder.java:600)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.asn1.ber.Asn1Decoder.decode(Asn1Decoder.java:736)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.ldap.codec.protocol.mina.LdapProtocolDecoder.decode(LdapProtocolDecoder.java:145)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.directory.api.ldap.codec.protocol.mina.LdapProtocolDecoder.decode(LdapProtocolDecoder.java:94)
> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
> at
> org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:230)
> ~[mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:542)
> ~[mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1300(DefaultIoFilterChain.java:48)
> ~[mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:943)
> ~[mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:109)
> ~[mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:542)
> ~[mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:535)
> ~[mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:697)
> ~[mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:651)
> ~[mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:640)
> ~[mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.polling.AbstractPollingIoProcessor.access$600(AbstractPollingIoProcessor.java:68)
> ~[mina-core-2.0.13.jar:na]
> at
> org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1097)
> ~[mina-core-2.0.13.jar:na]
> at
> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
> ~[mina-core-2.0.13.jar:na]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_101]
> 2016-08-26 10:13:22,931 []
> [C3P0PooledConnectionPoolManager[identityToken->2tu6q89i1j961p54hnz75|180fb7dc]-AdminTaskTimer]
> WARN (com.mchange.v2.async.ThreadPoolAsynchronousRunner):
> com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector at 2f41b4ad
> -- APPARENT DEADLOCK!!! Complete Status:
> Managed Threads: 3
> Active Threads: 3
> Active Tasks:
> com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask at 5b637151
> on thread:
> C3P0PooledConnectionPoolManager[identityToken->2tu6q89i1j961p54hnz75|180fb7dc]-HelperThread-#2
> com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask at 79a5524c
> on thread:
> C3P0PooledConnectionPoolManager[identityToken->2tu6q89i1j961p54hnz75|180fb7dc]-HelperThread-#0
> com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask at a6cc06e
> on thread:
> C3P0PooledConnectionPoolManager[identityToken->2tu6q89i1j961p54hnz75|180fb7dc]-HelperThread-#1
> Pending Tasks:
> com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask at 716cf0fa
> com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask at 3145fe7f
> com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask at 62a1fd29
> com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask at e223ea3
> com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask at 185ff4bd
> com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask at 6f757cf
> com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask at 6b52ba8d
> com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask at 749a4662
> com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask at 1307ae50
> com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask at 2f72cad0
> com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask at 510a9b61
> com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask at 77057e74
> com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask at 595e2bea
> com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask at 60c34e5b
> com.mchange.v2.resourcepool.BasicResourcePool$AsyncTestIdleResourceTask at 659710b1
> Pool thread stack traces:
> Thread[C3P0PooledConnectionPoolManager[identityToken->2tu6q89i1j961p54hnz75|180fb7dc]-HelperThread-#2,5,main]
> com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:674)
> Thread[C3P0PooledConnectionPoolManager[identityToken->2tu6q89i1j961p54hnz75|180fb7dc]-HelperThread-#1,5,main]
> com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:674)
> Thread[C3P0PooledConnectionPoolManager[identityToken->2tu6q89i1j961p54hnz75|180fb7dc]-HelperThread-#0,5,main]
> com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:672)
>
>
> 2016-08-26 10:13:51,355 [] [Thread-9] ERROR
> (org.activiti.engine.impl.jobexecutor.AcquireJobsRunnableImpl):
> exception during job acquisition: Java heap space
> java.lang.OutOfMemoryError: Java heap space
> at
> java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1019)
> ~[na:1.8.0_101]
> at
> java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
> ~[na:1.8.0_101]
> at
> sun.util.resources.ParallelListResourceBundle.loadLookupTablesIfNecessary(ParallelListResourceBundle.java:169)
> ~[na:1.8.0_101]
> at
> sun.util.resources.ParallelListResourceBundle.handleKeySet(ParallelListResourceBundle.java:134)
> ~[na:1.8.0_101]
> at
> sun.util.resources.ParallelListResourceBundle.keySet(ParallelListResourceBundle.java:143)
> ~[na:1.8.0_101]
> at
> sun.util.resources.ParallelListResourceBundle.containsKey(ParallelListResourceBundle.java:129)
> ~[na:1.8.0_101]
> at
> sun.util.resources.ParallelListResourceBundle$KeySet.contains(ParallelListResourceBundle.java:208)
> ~[na:1.8.0_101]
> at
> sun.util.resources.ParallelListResourceBundle.containsKey(ParallelListResourceBundle.java:129)
> ~[na:1.8.0_101]
> at
> sun.util.resources.ParallelListResourceBundle$KeySet.contains(ParallelListResourceBundle.java:208)
> ~[na:1.8.0_101]
> at
> sun.util.resources.ParallelListResourceBundle.containsKey(ParallelListResourceBundle.java:129)
> ~[na:1.8.0_101]
> at
> java.text.DateFormatSymbols.initializeData(DateFormatSymbols.java:702)
> ~[na:1.8.0_101]
> at
> java.text.DateFormatSymbols.<init>(DateFormatSymbols.java:146)
> ~[na:1.8.0_101]
> at
> sun.util.locale.provider.DateFormatSymbolsProviderImpl.getInstance(DateFormatSymbolsProviderImpl.java:85)
> ~[na:1.8.0_101]
> at
> java.text.DateFormatSymbols.getProviderInstance(DateFormatSymbols.java:359)
> ~[na:1.8.0_101]
> at
> java.text.DateFormatSymbols.getInstanceRef(DateFormatSymbols.java:349)
> ~[na:1.8.0_101]
> at
> java.text.SimpleDateFormat.<init>(SimpleDateFormat.java:603)
> ~[na:1.8.0_101]
> at
> com.mysql.jdbc.PreparedStatement.setTimestampInternal(PreparedStatement.java:4224)
> ~[mysql-connector-java-5.1.39-bin.jar:5.1.39]
> at
> com.mysql.jdbc.PreparedStatement.setTimestamp(PreparedStatement.java:4183)
> ~[mysql-connector-java-5.1.39-bin.jar:5.1.39]
> at
> org.apache.ibatis.type.DateTypeHandler.setNonNullParameter(DateTypeHandler.java:33)
> ~[mybatis-3.3.0.jar:3.3.0]
> at
> org.apache.ibatis.type.DateTypeHandler.setNonNullParameter(DateTypeHandler.java:28)
> ~[mybatis-3.3.0.jar:3.3.0]
> at
> org.apache.ibatis.type.BaseTypeHandler.setParameter(BaseTypeHandler.java:53)
> ~[mybatis-3.3.0.jar:3.3.0]
> at
> org.apache.ibatis.type.UnknownTypeHandler.setNonNullParameter(UnknownTypeHandler.java:45)
> ~[mybatis-3.3.0.jar:3.3.0]
> at
> org.apache.ibatis.type.BaseTypeHandler.setParameter(BaseTypeHandler.java:53)
> ~[mybatis-3.3.0.jar:3.3.0]
> at
> org.apache.ibatis.scripting.defaults.DefaultParameterHandler.setParameters(DefaultParameterHandler.java:87)
> ~[mybatis-3.3.0.jar:3.3.0]
> at
> org.apache.ibatis.executor.statement.PreparedStatementHandler.parameterize(PreparedStatementHandler.java:85)
> ~[mybatis-3.3.0.jar:3.3.0]
> at
> org.apache.ibatis.executor.statement.RoutingStatementHandler.parameterize(RoutingStatementHandler.java:63)
> ~[mybatis-3.3.0.jar:3.3.0]
> at
> org.apache.ibatis.executor.SimpleExecutor.prepareStatement(SimpleExecutor.java:77)
> ~[mybatis-3.3.0.jar:3.3.0]
> at
> org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:61)
> ~[mybatis-3.3.0.jar:3.3.0]
> at
> org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:303)
> ~[mybatis-3.3.0.jar:3.3.0]
> at
> org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:154)
> ~[mybatis-3.3.0.jar:3.3.0]
> at
> org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:102)
> ~[mybatis-3.3.0.jar:3.3.0]
> at
> org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:82)
> ~[mybatis-3.3.0.jar:3.3.0]
>
> On Fri, Aug 26, 2016 at 10:05 AM, Mencel, Matt <mr-mencel at wiu.edu
> <mailto:mr-mencel at wiu.edu>> wrote:
>
> I just attempted my first import from LDAP. I have a condition
> that limits the import to a single department of about 80
> accounts, but the import still processes all the others.
>
> The import processed nearly 30000 accounts. At just over 9000 the
> web UI became unresponsive. It's been 30 minutes and I still
> can't do anything in the UI.
>
> My tomcat java process is still eating up the CPU, but nothing has
> logged in the idm.log for the last 30 minutes. Is there some
> other place I can look to see what this process is doing?
>
> Here's the processes from htop...
>
> 111390 tomcat_mi 20 0 9163M 5052M 15752 S 591. 42.1 2h08:45
> /usr/bin/java
> -Djava.util.logging.config.file=/opt/tomcat_midpoint/conf/logging.properties
> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
> -Djdk.tls.ephe
> 111404 tomcat_mi 20 0 9163M 5052M 15752 R 98.3 42.1 19:45.81
> /usr/bin/java
> -Djava.util.logging.config.file=/opt/tomcat_midpoint/conf/logging.properties
> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
> -Djdk.tls.ephe
> 111407 tomcat_mi 20 0 9163M 5052M 15752 R 98.3 42.1 19:42.20
> /usr/bin/java
> -Djava.util.logging.config.file=/opt/tomcat_midpoint/conf/logging.properties
> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
> -Djdk.tls.ephe
> 111405 tomcat_mi 20 0 9163M 5052M 15752 R 99.0 42.1 19:42.24
> /usr/bin/java
> -Djava.util.logging.config.file=/opt/tomcat_midpoint/conf/logging.properties
> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
> -Djdk.tls.ephe
> 111406 tomcat_mi 20 0 9163M 5052M 15752 R 98.3 42.1 19:47.07
> /usr/bin/java
> -Djava.util.logging.config.file=/opt/tomcat_midpoint/conf/logging.properties
> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
> -Djdk.tls.ephe
> 111402 tomcat_mi 20 0 9163M 5052M 15752 R 97.7 42.1 19:44.50
> /usr/bin/java
> -Djava.util.logging.config.file=/opt/tomcat_midpoint/conf/logging.properties
> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
> -Djdk.tls.ephe
> 111403 tomcat_mi 20 0 9163M 5052M 15752 R 98.3 42.1 19:43.86
> /usr/bin/java
> -Djava.util.logging.config.file=/opt/tomcat_midpoint/conf/logging.properties
> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
> -Djdk.tls.ephe
> 111408 tomcat_mi 20 0 9163M 5052M 15752 S 1.3 42.1 0:53.12
> /usr/bin/java
> -Djava.util.logging.config.file=/opt/tomcat_midpoint/conf/logging.properties
> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
> -Djdk.tls.ephe
>
> Details on one of the processes....
> # ps aux | grep 11390
>
> tomcat_+ 111390 204 42.1 9383488 5173404 ? Ssl 08:57 133:18
> /usr/bin/java
> -Djava.util.logging.config.file=/opt/tomcat_midpoint/conf/logging.properties
> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
> -Djdk.tls.ephemeralDHKeySize=2048 -Dmidpoint.home=/opt/midpoint
> -Djavax.net.ssl.trustStore=/opt/midpoint/keystore.jceks
> -Djavax.net.ssl.trustStoreType=jceks -Xms512m
> -XX:MaxPermSize=1024m -Xss1m -Xmx4096m -classpath
> /opt/tomcat_midpoint/bin/bootstrap.jar:/opt/tomcat_midpoint/bin/tomcat-juli.jar
> -Dcatalina.base=/opt/tomcat_midpoint
> -Dcatalina.home=/opt/tomcat_midpoint
> -Djava.io.tmpdir=/opt/tomcat_midpoint/temp
> org.apache.catalina.startup.Bootstrap start
>
> Are those threads still just busy processing the records?
>
> Thanks,
> Matt
>
>
>
>
> _______________________________________________
> midPoint mailing list
> midPoint at lists.evolveum.com
> http://lists.evolveum.com/mailman/listinfo/midpoint
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20160826/fc31214e/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: kficpbebihkpdopd.png
Type: image/png
Size: 50643 bytes
Desc: not available
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20160826/fc31214e/attachment.png>
More information about the midPoint
mailing list