[midPoint] High CPU Utilization After Import Attempt - UI Unresponsive
Mencel, Matt
mr-mencel at wiu.edu
Fri Aug 26 19:27:49 CEST 2016
My Midpoint vm currently has 6 vCPUs and 12GB of RAM. I'm running the task
with 8 threads.
I've gotten remote JMX setup and am keeping an eye on it now with
visualvm. If it happens again I'll try to take a heapdump and see if I can
find a memory leak.
Matt
On Fri, Aug 26, 2016 at 12:20 PM, Jason Everling <jeverling at bshp.edu> wrote:
> Yeah, so if you go to 'Configuration' -> 'Shadow Details' and choose your
> resource, you will see all accounts.
>
> How many vCPUs does your machine have? You can also try and see if giving
> it more worker threads would help, for our resources that have every single
> person I utilize it and it does help speed up the recon tasks,
>
> https://wiki.evolveum.com/display/midPoint/Parallelized+background+tasks
>
> Sample from one of my task xmls,
>
> <name>Reconcile: Some Database</name>
> <extension xmlns:mext="http://midpoint.evolveum.com/xml/ns/public/
> model/extension-3">
> <mext:kind>account</mext:kind>
> <mext:workerThreads>4</mext:workerThreads>
> </extension>
>
> JASON
>
> On Fri, Aug 26, 2016 at 11:33 AM, Mencel, Matt <mr-mencel at wiu.edu> wrote:
>
>> Yep, see below...
>>
>> <synchronization>
>> <objectSynchronization>
>> <kind>account</kind>
>> <intent>person</intent>
>> <focusType>c:UserType</focusType>
>> <enabled>true</enabled>
>> <!-- Only Sync uTech from LDAP -->
>> <condition>
>> <script>
>> <code>
>> tmp = basic.getAttributeValue(shadow, '
>> http://midpoint.evolveum.com/xml/ns/public/resource/instance-3',
>> 'department'); return (tmp == 'University Technology' || tmp == 'AIMS')
>> </code>
>> </script>
>> </condition>
>> <correlation>
>> <q:equal>
>> <q:path>c:name</q:path>
>> <expression>
>> <path>$account/attributes/icfs:name</path>
>> </expression>
>> </q:equal>
>> </correlation>
>> <reaction>
>> <situation>linked</situation>
>> <action>
>> <handlerUri>http://midpoint.ev
>> olveum.com/xml/ns/public/model/action-3#modifyUser</handlerUri>
>> </action>
>> </reaction>
>> <reaction>
>> <situation>deleted</situation>
>> <action>
>> <handlerUri>http://midpoint.ev
>> olveum.com/xml/ns/public/model/action-3#unlinkAccount</handlerUri>
>> </action>
>> </reaction>
>> <reaction>
>> <situation>unlinked</situation>
>> <action>
>> <handlerUri>http://midpoint.ev
>> olveum.com/xml/ns/public/model/action-3#linkAccount</handlerUri>
>> </action>
>> </reaction>
>> <reaction>
>> <situation>unmatched</situation>
>> <action>
>> <handlerUri>http://midpoint.ev
>> olveum.com/xml/ns/public/model/action-3#addUser</handlerUri>
>> </action>
>> </reaction>
>> </objectSynchronization>
>> </synchronization>
>>
>> On Fri, Aug 26, 2016 at 11:30 AM, Jason Everling <jeverling at bshp.edu>
>> wrote:
>>
>>> So, I noticed this, did you use a condition in the objectSynchronization
>>> part? If so, it will still create 'shadows' for every item. I have a
>>> condition to only sync certain groups but if I go to the debug pages and
>>> view the shadows for the resource I can see all my AD groups. I am assuming
>>> this should be correct so that it would not create a new account on the
>>> resource that already exist but did not know about.
>>>
>>>
>>>
>>> JASON
>>>
>>> On Fri, Aug 26, 2016 at 10:39 AM, Pavol Mederly <mederly at evolveum.com>
>>> wrote:
>>>
>>>> Hello Matt,
>>>>
>>>> I have no direct experience with deploying midPoint with large number
>>>> of users, but let me give you a few hints:
>>>>
>>>> 1) If GUI is dead, maybe REST would allow you to check the task's
>>>> state; at http://localhost:8080/midpoint/ws/rest/tasks/<oid>, e.g.
>>>> http://localhost:8080/midpoint/ws/rest/tasks/00000000-0000-0
>>>> 000-0000-000000000005
>>>>
>>>> 2) If REST is unresponsive as well, or if there's need to know more
>>>> about what's going on, I sometimes use jvisualvm - to see e.g. heap
>>>> utilization, thread state, even collecting thread dump, etc. Maybe you
>>>> could use it to see if there's a memleak somewhere (as Radovan said).
>>>>
>>>> Its output looks like this:
>>>>
>>>>
>>>> Best regards,
>>>>
>>>> Pavol Mederly
>>>> Software developerevolveum.com
>>>>
>>>> On 26.08.2016 17:25, Mencel, Matt wrote:
>>>>
>>>> New entries in the idm.log appeared about 40 minutes after it stopped
>>>> logging. My -Xmx setting in JAVA_OPTS is set to 4096.
>>>>
>>>> Matt
>>>>
>>>> 2016-08-26 10:06:51,393 [] [C3P0PooledConnectionPoolManag
>>>> er[identityToken->2tu6q89i1j961p54hnz75|180fb7dc]-AdminTaskTimer] WARN
>>>> (com.mchange.v2.async.ThreadPoolAsynchronousRunner):
>>>> com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDe
>>>> tector at 2f41b4ad -- APPARENT DEADLOCK!!! Creating emergency threads for
>>>> unassigned pending tasks!
>>>> 2016-08-26 10:12:30,979 [] [NioProcessor-1] WARN
>>>> (org.apache.directory.ldap.client.api.LdapNetworkConnection): Java
>>>> heap space
>>>> java.lang.OutOfMemoryError: Java heap space
>>>> at org.apache.directory.api.util.Strings.utf8ToString(Strings.java:724)
>>>> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.util.Strings.toLowerCase(Strings.java:1890)
>>>> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.ldap.model.entry.DefaultAttribute.s
>>>> etUpId(DefaultAttribute.java:637) ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.ldap.model.entry.DefaultAttribute.s
>>>> etUpId(DefaultAttribute.java:575) ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.ldap.model.entry.DefaultAttribute.<
>>>> init>(DefaultAttribute.java:192) ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.ldap.codec.decorators.SearchResultE
>>>> ntryDecorator.addAttribute(SearchResultEntryDecorator.java:112)
>>>> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.ldap.codec.actions.searchResultEntr
>>>> y.AddAttributeType.action(AddAttributeType.java:87)
>>>> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.ldap.codec.actions.searchResultEntr
>>>> y.AddAttributeType.action(AddAttributeType.java:48)
>>>> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.asn1.ber.grammar.AbstractGrammar.ex
>>>> ecuteAction(AbstractGrammar.java:136) ~[api-all-1.0.0-RC1.jar:1.0.0-
>>>> RC1]
>>>> at org.apache.directory.api.asn1.ber.Asn1Decoder.treatTLVDoneState(Asn1Decoder.java:600)
>>>> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.asn1.ber.Asn1Decoder.decode(Asn1Decoder.java:736)
>>>> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.ldap.codec.protocol.mina.LdapProtoc
>>>> olDecoder.decode(LdapProtocolDecoder.java:145)
>>>> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.ldap.codec.protocol.mina.LdapProtoc
>>>> olDecoder.decode(LdapProtocolDecoder.java:94)
>>>> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.mina.filter.codec.ProtocolCodecFilter.messageRece
>>>> ived(ProtocolCodecFilter.java:230) ~[mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNe
>>>> xtMessageReceived(DefaultIoFilterChain.java:542)
>>>> [mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.filterchain.DefaultIoFilterChain.access
>>>> $1300(DefaultIoFilterChain.java:48) [mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryI
>>>> mpl$1.messageReceived(DefaultIoFilterChain.java:943)
>>>> [mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:109)
>>>> [mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNe
>>>> xtMessageReceived(DefaultIoFilterChain.java:542)
>>>> [mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMe
>>>> ssageReceived(DefaultIoFilterChain.java:535) [mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.polling.AbstractPollingIoProcessor.read
>>>> (AbstractPollingIoProcessor.java:697) [mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.polling.AbstractPollingIoProcessor.proc
>>>> ess(AbstractPollingIoProcessor.java:651) [mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.polling.AbstractPollingIoProcessor.proc
>>>> ess(AbstractPollingIoProcessor.java:640) [mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.polling.AbstractPollingIoProcessor.acce
>>>> ss$600(AbstractPollingIoProcessor.java:68) [mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.polling.AbstractPollingIoProcessor$Proc
>>>> essor.run(AbstractPollingIoProcessor.java:1097)
>>>> [mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
>>>> [mina-core-2.0.13.jar:na]
>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> [na:1.8.0_101]
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> [na:1.8.0_101]
>>>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
>>>> 2016-08-26 10:12:30,980 [] [NioProcessor-1] WARN
>>>> (org.apache.directory.ldap.client.api.LdapNetworkConnection): Java
>>>> heap space
>>>> java.lang.OutOfMemoryError: Java heap space
>>>> at org.apache.directory.api.util.Strings.utf8ToString(Strings.java:724)
>>>> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.util.Strings.toLowerCase(Strings.java:1890)
>>>> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.ldap.model.entry.DefaultAttribute.s
>>>> etUpId(DefaultAttribute.java:637) ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.ldap.model.entry.DefaultAttribute.s
>>>> etUpId(DefaultAttribute.java:575) ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.ldap.model.entry.DefaultAttribute.<
>>>> init>(DefaultAttribute.java:192) ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.ldap.codec.decorators.SearchResultE
>>>> ntryDecorator.addAttribute(SearchResultEntryDecorator.java:112)
>>>> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.ldap.codec.actions.searchResultEntr
>>>> y.AddAttributeType.action(AddAttributeType.java:87)
>>>> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.ldap.codec.actions.searchResultEntr
>>>> y.AddAttributeType.action(AddAttributeType.java:48)
>>>> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.asn1.ber.grammar.AbstractGrammar.ex
>>>> ecuteAction(AbstractGrammar.java:136) ~[api-all-1.0.0-RC1.jar:1.0.0-
>>>> RC1]
>>>> at org.apache.directory.api.asn1.ber.Asn1Decoder.treatTLVDoneState(Asn1Decoder.java:600)
>>>> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.asn1.ber.Asn1Decoder.decode(Asn1Decoder.java:736)
>>>> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.ldap.codec.protocol.mina.LdapProtoc
>>>> olDecoder.decode(LdapProtocolDecoder.java:145)
>>>> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.ldap.codec.protocol.mina.LdapProtoc
>>>> olDecoder.decode(LdapProtocolDecoder.java:94)
>>>> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.mina.filter.codec.ProtocolCodecFilter.messageRece
>>>> ived(ProtocolCodecFilter.java:230) ~[mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNe
>>>> xtMessageReceived(DefaultIoFilterChain.java:542)
>>>> [mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.filterchain.DefaultIoFilterChain.access
>>>> $1300(DefaultIoFilterChain.java:48) [mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryI
>>>> mpl$1.messageReceived(DefaultIoFilterChain.java:943)
>>>> ~[mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:109)
>>>> ~[mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNe
>>>> xtMessageReceived(DefaultIoFilterChain.java:542)
>>>> [mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMe
>>>> ssageReceived(DefaultIoFilterChain.java:535) [mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.polling.AbstractPollingIoProcessor.read
>>>> (AbstractPollingIoProcessor.java:697) [mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.polling.AbstractPollingIoProcessor.proc
>>>> ess(AbstractPollingIoProcessor.java:651) [mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.polling.AbstractPollingIoProcessor.proc
>>>> ess(AbstractPollingIoProcessor.java:640) [mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.polling.AbstractPollingIoProcessor.acce
>>>> ss$600(AbstractPollingIoProcessor.java:68) [mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.polling.AbstractPollingIoProcessor$Proc
>>>> essor.run(AbstractPollingIoProcessor.java:1097)
>>>> [mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
>>>> [mina-core-2.0.13.jar:na]
>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> [na:1.8.0_101]
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> [na:1.8.0_101]
>>>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
>>>> 2016-08-26 10:12:30,981 [] [NioProcessor-2] WARN
>>>> (org.apache.directory.ldap.client.api.LdapNetworkConnection): Java
>>>> heap space
>>>> java.lang.OutOfMemoryError: Java heap space
>>>> at java.nio.charset.CharsetEncoder.<init>(CharsetEncoder.java:233)
>>>> ~[na:1.8.0_101]
>>>> at sun.nio.cs.UTF_8$Encoder.<init>(UTF_8.java:558)
>>>> ~[na:1.8.0_101]
>>>> at sun.nio.cs.UTF_8$Encoder.<init>(UTF_8.java:554)
>>>> ~[na:1.8.0_101]
>>>> at sun.nio.cs.UTF_8.newEncoder(UTF_8.java:72) ~[na:1.8.0_101]
>>>> at java.lang.StringCoding.encode(StringCoding.java:348)
>>>> ~[na:1.8.0_101]
>>>> at java.lang.String.getBytes(String.java:941) ~[na:1.8.0_101]
>>>> at org.apache.directory.api.util.Strings.getBytesUtf8(Strings.java:1578)
>>>> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.ldap.model.entry.StringValue.<init>(StringValue.java:93)
>>>> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.ldap.model.entry.DefaultAttribute.c
>>>> reateStringValue(DefaultAttribute.java:93)
>>>> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.ldap.model.entry.DefaultAttribute.add(DefaultAttribute.java:1064)
>>>> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.ldap.codec.decorators.SearchResultE
>>>> ntryDecorator.addAttributeValue(SearchResultEntryDecorator.java:127)
>>>> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.ldap.codec.actions.searchResultEntr
>>>> y.StoreSearchResultAttributeValue.action(StoreSearchResultAttributeValue.java:100)
>>>> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.ldap.codec.actions.searchResultEntr
>>>> y.StoreSearchResultAttributeValue.action(StoreSearchResultAttributeValue.java:44)
>>>> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.asn1.ber.grammar.AbstractGrammar.ex
>>>> ecuteAction(AbstractGrammar.java:136) ~[api-all-1.0.0-RC1.jar:1.0.0-
>>>> RC1]
>>>> at org.apache.directory.api.asn1.ber.Asn1Decoder.treatTLVDoneState(Asn1Decoder.java:600)
>>>> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.asn1.ber.Asn1Decoder.decode(Asn1Decoder.java:736)
>>>> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.ldap.codec.protocol.mina.LdapProtoc
>>>> olDecoder.decode(LdapProtocolDecoder.java:145)
>>>> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.ldap.codec.protocol.mina.LdapProtoc
>>>> olDecoder.decode(LdapProtocolDecoder.java:94)
>>>> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.mina.filter.codec.ProtocolCodecFilter.messageRece
>>>> ived(ProtocolCodecFilter.java:230) ~[mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNe
>>>> xtMessageReceived(DefaultIoFilterChain.java:542)
>>>> ~[mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.filterchain.DefaultIoFilterChain.access
>>>> $1300(DefaultIoFilterChain.java:48) ~[mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryI
>>>> mpl$1.messageReceived(DefaultIoFilterChain.java:943)
>>>> ~[mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:109)
>>>> ~[mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNe
>>>> xtMessageReceived(DefaultIoFilterChain.java:542)
>>>> ~[mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMe
>>>> ssageReceived(DefaultIoFilterChain.java:535) ~[mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.polling.AbstractPollingIoProcessor.read
>>>> (AbstractPollingIoProcessor.java:697) ~[mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.polling.AbstractPollingIoProcessor.proc
>>>> ess(AbstractPollingIoProcessor.java:651) ~[mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.polling.AbstractPollingIoProcessor.proc
>>>> ess(AbstractPollingIoProcessor.java:640) ~[mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.polling.AbstractPollingIoProcessor.acce
>>>> ss$600(AbstractPollingIoProcessor.java:68) ~[mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.polling.AbstractPollingIoProcessor$Proc
>>>> essor.run(AbstractPollingIoProcessor.java:1097)
>>>> ~[mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
>>>> ~[mina-core-2.0.13.jar:na]
>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> ~[na:1.8.0_101]
>>>> 2016-08-26 10:12:30,982 [] [NioProcessor-2] WARN
>>>> (org.apache.directory.ldap.client.api.LdapNetworkConnection): Java
>>>> heap space
>>>> java.lang.OutOfMemoryError: Java heap space
>>>> at java.nio.charset.CharsetEncoder.<init>(CharsetEncoder.java:233)
>>>> ~[na:1.8.0_101]
>>>> at sun.nio.cs.UTF_8$Encoder.<init>(UTF_8.java:558)
>>>> ~[na:1.8.0_101]
>>>> at sun.nio.cs.UTF_8$Encoder.<init>(UTF_8.java:554)
>>>> ~[na:1.8.0_101]
>>>> at sun.nio.cs.UTF_8.newEncoder(UTF_8.java:72) ~[na:1.8.0_101]
>>>> at java.lang.StringCoding.encode(StringCoding.java:348)
>>>> ~[na:1.8.0_101]
>>>> at java.lang.String.getBytes(String.java:941) ~[na:1.8.0_101]
>>>> at org.apache.directory.api.util.Strings.getBytesUtf8(Strings.java:1578)
>>>> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.ldap.model.entry.StringValue.<init>(StringValue.java:93)
>>>> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.ldap.model.entry.DefaultAttribute.c
>>>> reateStringValue(DefaultAttribute.java:93)
>>>> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.ldap.model.entry.DefaultAttribute.add(DefaultAttribute.java:1064)
>>>> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.ldap.codec.decorators.SearchResultE
>>>> ntryDecorator.addAttributeValue(SearchResultEntryDecorator.java:127)
>>>> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.ldap.codec.actions.searchResultEntr
>>>> y.StoreSearchResultAttributeValue.action(StoreSearchResultAttributeValue.java:100)
>>>> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.ldap.codec.actions.searchResultEntr
>>>> y.StoreSearchResultAttributeValue.action(StoreSearchResultAttributeValue.java:44)
>>>> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.asn1.ber.grammar.AbstractGrammar.ex
>>>> ecuteAction(AbstractGrammar.java:136) ~[api-all-1.0.0-RC1.jar:1.0.0-
>>>> RC1]
>>>> at org.apache.directory.api.asn1.ber.Asn1Decoder.treatTLVDoneState(Asn1Decoder.java:600)
>>>> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.asn1.ber.Asn1Decoder.decode(Asn1Decoder.java:736)
>>>> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.ldap.codec.protocol.mina.LdapProtoc
>>>> olDecoder.decode(LdapProtocolDecoder.java:145)
>>>> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.directory.api.ldap.codec.protocol.mina.LdapProtoc
>>>> olDecoder.decode(LdapProtocolDecoder.java:94)
>>>> ~[api-all-1.0.0-RC1.jar:1.0.0-RC1]
>>>> at org.apache.mina.filter.codec.ProtocolCodecFilter.messageRece
>>>> ived(ProtocolCodecFilter.java:230) ~[mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNe
>>>> xtMessageReceived(DefaultIoFilterChain.java:542)
>>>> ~[mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.filterchain.DefaultIoFilterChain.access
>>>> $1300(DefaultIoFilterChain.java:48) ~[mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryI
>>>> mpl$1.messageReceived(DefaultIoFilterChain.java:943)
>>>> ~[mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:109)
>>>> ~[mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNe
>>>> xtMessageReceived(DefaultIoFilterChain.java:542)
>>>> ~[mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMe
>>>> ssageReceived(DefaultIoFilterChain.java:535) ~[mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.polling.AbstractPollingIoProcessor.read
>>>> (AbstractPollingIoProcessor.java:697) ~[mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.polling.AbstractPollingIoProcessor.proc
>>>> ess(AbstractPollingIoProcessor.java:651) ~[mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.polling.AbstractPollingIoProcessor.proc
>>>> ess(AbstractPollingIoProcessor.java:640) ~[mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.polling.AbstractPollingIoProcessor.acce
>>>> ss$600(AbstractPollingIoProcessor.java:68) ~[mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.core.polling.AbstractPollingIoProcessor$Proc
>>>> essor.run(AbstractPollingIoProcessor.java:1097)
>>>> ~[mina-core-2.0.13.jar:na]
>>>> at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
>>>> ~[mina-core-2.0.13.jar:na]
>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> ~[na:1.8.0_101]
>>>> 2016-08-26 10:13:22,931 [] [C3P0PooledConnectionPoolManag
>>>> er[identityToken->2tu6q89i1j961p54hnz75|180fb7dc]-AdminTaskTimer] WARN
>>>> (com.mchange.v2.async.ThreadPoolAsynchronousRunner):
>>>> com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDe
>>>> tector at 2f41b4ad -- APPARENT DEADLOCK!!! Complete Status:
>>>> Managed Threads: 3
>>>> Active Threads: 3
>>>> Active Tasks:
>>>> com.mchange.v2.resourcepool.Ba
>>>> sicResourcePool$AsyncTestIdleResourceTask at 5b637151
>>>> on thread: C3P0PooledConnectionPoolManage
>>>> r[identityToken->2tu6q89i1j961p54hnz75|180fb7dc]-HelperThread-#2
>>>> com.mchange.v2.resourcepool.Ba
>>>> sicResourcePool$AsyncTestIdleResourceTask at 79a5524c
>>>> on thread: C3P0PooledConnectionPoolManage
>>>> r[identityToken->2tu6q89i1j961p54hnz75|180fb7dc]-HelperThread-#0
>>>> com.mchange.v2.resourcepool.Ba
>>>> sicResourcePool$AsyncTestIdleResourceTask at a6cc06e
>>>> on thread: C3P0PooledConnectionPoolManage
>>>> r[identityToken->2tu6q89i1j961p54hnz75|180fb7dc]-HelperThread-#1
>>>> Pending Tasks:
>>>> com.mchange.v2.resourcepool.Ba
>>>> sicResourcePool$AsyncTestIdleResourceTask at 716cf0fa
>>>> com.mchange.v2.resourcepool.Ba
>>>> sicResourcePool$AsyncTestIdleResourceTask at 3145fe7f
>>>> com.mchange.v2.resourcepool.Ba
>>>> sicResourcePool$AsyncTestIdleResourceTask at 62a1fd29
>>>> com.mchange.v2.resourcepool.Ba
>>>> sicResourcePool$AsyncTestIdleResourceTask at e223ea3
>>>> com.mchange.v2.resourcepool.Ba
>>>> sicResourcePool$AsyncTestIdleResourceTask at 185ff4bd
>>>> com.mchange.v2.resourcepool.Ba
>>>> sicResourcePool$AsyncTestIdleResourceTask at 6f757cf
>>>> com.mchange.v2.resourcepool.Ba
>>>> sicResourcePool$AsyncTestIdleResourceTask at 6b52ba8d
>>>> com.mchange.v2.resourcepool.Ba
>>>> sicResourcePool$AsyncTestIdleResourceTask at 749a4662
>>>> com.mchange.v2.resourcepool.Ba
>>>> sicResourcePool$AsyncTestIdleResourceTask at 1307ae50
>>>> com.mchange.v2.resourcepool.Ba
>>>> sicResourcePool$AsyncTestIdleResourceTask at 2f72cad0
>>>> com.mchange.v2.resourcepool.Ba
>>>> sicResourcePool$AsyncTestIdleResourceTask at 510a9b61
>>>> com.mchange.v2.resourcepool.Ba
>>>> sicResourcePool$AsyncTestIdleResourceTask at 77057e74
>>>> com.mchange.v2.resourcepool.Ba
>>>> sicResourcePool$AsyncTestIdleResourceTask at 595e2bea
>>>> com.mchange.v2.resourcepool.Ba
>>>> sicResourcePool$AsyncTestIdleResourceTask at 60c34e5b
>>>> com.mchange.v2.resourcepool.Ba
>>>> sicResourcePool$AsyncTestIdleResourceTask at 659710b1
>>>> Pool thread stack traces:
>>>> Thread[C3P0PooledConnectionPoolManager[identityToken->2tu6q8
>>>> 9i1j961p54hnz75|180fb7dc]-HelperThread-#2,5,main]
>>>> com.mchange.v2.async.ThreadPoo
>>>> lAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRun
>>>> ner.java:674)
>>>> Thread[C3P0PooledConnectionPoolManager[identityToken->2tu6q8
>>>> 9i1j961p54hnz75|180fb7dc]-HelperThread-#1,5,main]
>>>> com.mchange.v2.async.ThreadPoo
>>>> lAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRun
>>>> ner.java:674)
>>>> Thread[C3P0PooledConnectionPoolManager[identityToken->2tu6q8
>>>> 9i1j961p54hnz75|180fb7dc]-HelperThread-#0,5,main]
>>>> com.mchange.v2.async.ThreadPoo
>>>> lAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRun
>>>> ner.java:672)
>>>>
>>>>
>>>> 2016-08-26 10:13:51,355 [] [Thread-9] ERROR
>>>> (org.activiti.engine.impl.jobexecutor.AcquireJobsRunnableImpl):
>>>> exception during job acquisition: Java heap space
>>>> java.lang.OutOfMemoryError: Java heap space
>>>> at java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1019)
>>>> ~[na:1.8.0_101]
>>>> at java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
>>>> ~[na:1.8.0_101]
>>>> at sun.util.resources.ParallelListResourceBundle.loadLookupTabl
>>>> esIfNecessary(ParallelListResourceBundle.java:169) ~[na:1.8.0_101]
>>>> at sun.util.resources.ParallelListResourceBundle.handleKeySet(P
>>>> arallelListResourceBundle.java:134) ~[na:1.8.0_101]
>>>> at sun.util.resources.ParallelListResourceBundle.keySet(ParallelListResourceBundle.java:143)
>>>> ~[na:1.8.0_101]
>>>> at sun.util.resources.ParallelListResourceBundle.containsKey(Pa
>>>> rallelListResourceBundle.java:129) ~[na:1.8.0_101]
>>>> at sun.util.resources.ParallelListResourceBundle$KeySet.contain
>>>> s(ParallelListResourceBundle.java:208) ~[na:1.8.0_101]
>>>> at sun.util.resources.ParallelListResourceBundle.containsKey(Pa
>>>> rallelListResourceBundle.java:129) ~[na:1.8.0_101]
>>>> at sun.util.resources.ParallelListResourceBundle$KeySet.contain
>>>> s(ParallelListResourceBundle.java:208) ~[na:1.8.0_101]
>>>> at sun.util.resources.ParallelListResourceBundle.containsKey(Pa
>>>> rallelListResourceBundle.java:129) ~[na:1.8.0_101]
>>>> at java.text.DateFormatSymbols.initializeData(DateFormatSymbols.java:702)
>>>> ~[na:1.8.0_101]
>>>> at java.text.DateFormatSymbols.<init>(DateFormatSymbols.java:146)
>>>> ~[na:1.8.0_101]
>>>> at sun.util.locale.provider.DateFormatSymbolsProviderImpl.getIn
>>>> stance(DateFormatSymbolsProviderImpl.java:85) ~[na:1.8.0_101]
>>>> at java.text.DateFormatSymbols.getProviderInstance(DateFormatSymbols.java:359)
>>>> ~[na:1.8.0_101]
>>>> at java.text.DateFormatSymbols.getInstanceRef(DateFormatSymbols.java:349)
>>>> ~[na:1.8.0_101]
>>>> at java.text.SimpleDateFormat.<init>(SimpleDateFormat.java:603)
>>>> ~[na:1.8.0_101]
>>>> at com.mysql.jdbc.PreparedStatement.setTimestampInternal(PreparedStatement.java:4224)
>>>> ~[mysql-connector-java-5.1.39-bin.jar:5.1.39]
>>>> at com.mysql.jdbc.PreparedStatement.setTimestamp(PreparedStatement.java:4183)
>>>> ~[mysql-connector-java-5.1.39-bin.jar:5.1.39]
>>>> at org.apache.ibatis.type.DateTypeHandler.setNonNullParameter(DateTypeHandler.java:33)
>>>> ~[mybatis-3.3.0.jar:3.3.0]
>>>> at org.apache.ibatis.type.DateTypeHandler.setNonNullParameter(DateTypeHandler.java:28)
>>>> ~[mybatis-3.3.0.jar:3.3.0]
>>>> at org.apache.ibatis.type.BaseTypeHandler.setParameter(BaseTypeHandler.java:53)
>>>> ~[mybatis-3.3.0.jar:3.3.0]
>>>> at org.apache.ibatis.type.UnknownTypeHandler.setNonNullParameter(UnknownTypeHandler.java:45)
>>>> ~[mybatis-3.3.0.jar:3.3.0]
>>>> at org.apache.ibatis.type.BaseTypeHandler.setParameter(BaseTypeHandler.java:53)
>>>> ~[mybatis-3.3.0.jar:3.3.0]
>>>> at org.apache.ibatis.scripting.defaults.DefaultParameterHandler
>>>> .setParameters(DefaultParameterHandler.java:87)
>>>> ~[mybatis-3.3.0.jar:3.3.0]
>>>> at org.apache.ibatis.executor.statement.PreparedStatementHandle
>>>> r.parameterize(PreparedStatementHandler.java:85)
>>>> ~[mybatis-3.3.0.jar:3.3.0]
>>>> at org.apache.ibatis.executor.statement.RoutingStatementHandler
>>>> .parameterize(RoutingStatementHandler.java:63)
>>>> ~[mybatis-3.3.0.jar:3.3.0]
>>>> at org.apache.ibatis.executor.SimpleExecutor.prepareStatement(SimpleExecutor.java:77)
>>>> ~[mybatis-3.3.0.jar:3.3.0]
>>>> at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:61)
>>>> ~[mybatis-3.3.0.jar:3.3.0]
>>>> at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:303)
>>>> ~[mybatis-3.3.0.jar:3.3.0]
>>>> at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:154)
>>>> ~[mybatis-3.3.0.jar:3.3.0]
>>>> at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:102)
>>>> ~[mybatis-3.3.0.jar:3.3.0]
>>>> at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:82)
>>>> ~[mybatis-3.3.0.jar:3.3.0]
>>>>
>>>> On Fri, Aug 26, 2016 at 10:05 AM, Mencel, Matt <mr-mencel at wiu.edu>
>>>> wrote:
>>>>
>>>>> I just attempted my first import from LDAP. I have a condition that
>>>>> limits the import to a single department of about 80 accounts, but the
>>>>> import still processes all the others.
>>>>>
>>>>> The import processed nearly 30000 accounts. At just over 9000 the web
>>>>> UI became unresponsive. It's been 30 minutes and I still can't do anything
>>>>> in the UI.
>>>>>
>>>>> My tomcat java process is still eating up the CPU, but nothing has
>>>>> logged in the idm.log for the last 30 minutes. Is there some other place I
>>>>> can look to see what this process is doing?
>>>>>
>>>>> Here's the processes from htop...
>>>>>
>>>>> 111390 tomcat_mi 20 0 9163M 5052M 15752 S 591. 42.1 2h08:45
>>>>> /usr/bin/java -Djava.util.logging.config.fil
>>>>> e=/opt/tomcat_midpoint/conf/logging.properties
>>>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>>>> -Djdk.tls.ephe
>>>>> 111404 tomcat_mi 20 0 9163M 5052M 15752 R 98.3 42.1 19:45.81
>>>>> /usr/bin/java -Djava.util.logging.config.fil
>>>>> e=/opt/tomcat_midpoint/conf/logging.properties
>>>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>>>> -Djdk.tls.ephe
>>>>> 111407 tomcat_mi 20 0 9163M 5052M 15752 R 98.3 42.1 19:42.20
>>>>> /usr/bin/java -Djava.util.logging.config.fil
>>>>> e=/opt/tomcat_midpoint/conf/logging.properties
>>>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>>>> -Djdk.tls.ephe
>>>>> 111405 tomcat_mi 20 0 9163M 5052M 15752 R 99.0 42.1 19:42.24
>>>>> /usr/bin/java -Djava.util.logging.config.fil
>>>>> e=/opt/tomcat_midpoint/conf/logging.properties
>>>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>>>> -Djdk.tls.ephe
>>>>> 111406 tomcat_mi 20 0 9163M 5052M 15752 R 98.3 42.1 19:47.07
>>>>> /usr/bin/java -Djava.util.logging.config.fil
>>>>> e=/opt/tomcat_midpoint/conf/logging.properties
>>>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>>>> -Djdk.tls.ephe
>>>>> 111402 tomcat_mi 20 0 9163M 5052M 15752 R 97.7 42.1 19:44.50
>>>>> /usr/bin/java -Djava.util.logging.config.fil
>>>>> e=/opt/tomcat_midpoint/conf/logging.properties
>>>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>>>> -Djdk.tls.ephe
>>>>> 111403 tomcat_mi 20 0 9163M 5052M 15752 R 98.3 42.1 19:43.86
>>>>> /usr/bin/java -Djava.util.logging.config.fil
>>>>> e=/opt/tomcat_midpoint/conf/logging.properties
>>>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>>>> -Djdk.tls.ephe
>>>>> 111408 tomcat_mi 20 0 9163M 5052M 15752 S 1.3 42.1 0:53.12
>>>>> /usr/bin/java -Djava.util.logging.config.fil
>>>>> e=/opt/tomcat_midpoint/conf/logging.properties
>>>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>>>> -Djdk.tls.ephe
>>>>>
>>>>> Details on one of the processes....
>>>>> # ps aux | grep 11390
>>>>>
>>>>> tomcat_+ 111390 204 42.1 9383488 5173404 ? Ssl 08:57 133:18
>>>>> /usr/bin/java -Djava.util.logging.config.fil
>>>>> e=/opt/tomcat_midpoint/conf/logging.properties
>>>>> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>>>>> -Djdk.tls.ephemeralDHKeySize=2048 -Dmidpoint.home=/opt/midpoint
>>>>> -Djavax.net.ssl.trustStore=/opt/midpoint/keystore.jceks
>>>>> -Djavax.net.ssl.trustStoreType=jceks -Xms512m -XX:MaxPermSize=1024m
>>>>> -Xss1m -Xmx4096m -classpath /opt/tomcat_midpoint/bin/boots
>>>>> trap.jar:/opt/tomcat_midpoint/bin/tomcat-juli.jar
>>>>> -Dcatalina.base=/opt/tomcat_midpoint -Dcatalina.home=/opt/tomcat_midpoint
>>>>> -Djava.io.tmpdir=/opt/tomcat_midpoint/temp
>>>>> org.apache.catalina.startup.Bootstrap start
>>>>>
>>>>> Are those threads still just busy processing the records?
>>>>>
>>>>> Thanks,
>>>>> Matt
>>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> midPoint mailing listmidPoint at lists.evolveum.comhttp://lists.evolveum.com/mailman/listinfo/midpoint
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> midPoint mailing list
>>>> midPoint at lists.evolveum.com
>>>> http://lists.evolveum.com/mailman/listinfo/midpoint
>>>>
>>>>
>>>
>>>
>>>
>>> CONFIDENTIALITY NOTICE:
>>> This e-mail together with any attachments is proprietary and
>>> confidential; intended for only the recipient(s) named above and may
>>> contain information that is privileged. You should not retain, copy or use
>>> this e-mail or any attachments for any purpose, or disclose all or any part
>>> of the contents to any person. Any views or opinions expressed in this
>>> e-mail are those of the author and do not represent those of the Baptist
>>> School of Health Professions. If you have received this e-mail in error, or
>>> are not the named recipient(s), you are hereby notified that any review,
>>> dissemination, distribution or copying of this communication is prohibited
>>> by the sender and to do so might constitute a violation of the Electronic
>>> Communications Privacy Act, 18 U.S.C. section 2510-2521. Please immediately
>>> notify the sender and delete this e-mail and any attachments from your
>>> computer.
>>>
>>> _______________________________________________
>>> midPoint mailing list
>>> midPoint at lists.evolveum.com
>>> http://lists.evolveum.com/mailman/listinfo/midpoint
>>>
>>>
>>
>> _______________________________________________
>> midPoint mailing list
>> midPoint at lists.evolveum.com
>> http://lists.evolveum.com/mailman/listinfo/midpoint
>>
>>
>
>
>
> CONFIDENTIALITY NOTICE:
> This e-mail together with any attachments is proprietary and confidential;
> intended for only the recipient(s) named above and may contain information
> that is privileged. You should not retain, copy or use this e-mail or any
> attachments for any purpose, or disclose all or any part of the contents to
> any person. Any views or opinions expressed in this e-mail are those of the
> author and do not represent those of the Baptist School of Health
> Professions. If you have received this e-mail in error, or are not the
> named recipient(s), you are hereby notified that any review, dissemination,
> distribution or copying of this communication is prohibited by the sender
> and to do so might constitute a violation of the Electronic Communications
> Privacy Act, 18 U.S.C. section 2510-2521. Please immediately notify the
> sender and delete this e-mail and any attachments from your computer.
>
> _______________________________________________
> midPoint mailing list
> midPoint at lists.evolveum.com
> http://lists.evolveum.com/mailman/listinfo/midpoint
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20160826/97eb953f/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: kficpbebihkpdopd.png
Type: image/png
Size: 50643 bytes
Desc: not available
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20160826/97eb953f/attachment.png>
More information about the midPoint
mailing list