<html>
  <head>
    <meta content="text/html; charset=utf-8" http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    Yes, it is. But the memory consumption of the LDAP connector should
    be proportional to schema size and number of connector instances. It
    should NOT be proportional to the number of LDAP entries.<br>
    <br>
    The memory consumption should normally increase at the beginning of
    the import. But after first few entries are imported the memory
    consumption should stay more or less constant. If the heap size
    continually increases during import it is most likely midpoint bug,
    connector bug or Apache Directory API bug.<br>
    <br>
    <pre class="moz-signature" cols="72">-- 
Radovan Semancik
Software Architect
evolveum.com
</pre>
    <br>
    <br>
    <div class="moz-cite-prefix">On 08/25/2016 04:28 PM, Ivan Noris
      wrote:<br>
    </div>
    <blockquote
      cite="mid:1076769599.116183.1472135299965.JavaMail.zimbra@evolveum.com"
      type="cite">
      <div style="font-family: times new roman, new york, times, serif;
        font-size: 12pt; color: #000000">
        <div>I have a feeling that LDAP connector might be a little
          resource consuming for some resources with big schemas etc.<br>
        </div>
        <div>Ivan<br>
        </div>
        <div><br>
        </div>
        <hr id="zwchr">
        <blockquote style="border-left:2px solid
#1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From:
          </b>"Jason Everling" <a class="moz-txt-link-rfc2396E" href="mailto:jeverling@bshp.edu"><jeverling@bshp.edu></a><br>
          <b>To: </b>"midPoint General Discussion"
          <a class="moz-txt-link-rfc2396E" href="mailto:midpoint@lists.evolveum.com"><midpoint@lists.evolveum.com></a><br>
          <b>Sent: </b>Thursday, August 25, 2016 5:15:33 PM<br>
          <b>Subject: </b>Re: [midPoint] Running out of Java Heap Space<br>
          <div><br>
          </div>
          <div dir="ltr">I had the same issue when I imported around 20k
            accounts from one of our databases in the beginning,
            although, we have more than one java application running on
            this server, 64bit jdk, 
            <div><br>
            </div>
            <div>-Xms512m -Xmx4096m -XX:PermSize=512m
              -XX:MaxPermSize=1024m<br>
            </div>
          </div>
          <div class="gmail_extra"><br clear="all">
            <div>
              <div class="gmail_signature">
                <div dir="ltr">JASON</div>
              </div>
            </div>
            <br>
            <div class="gmail_quote">On Thu, Aug 25, 2016 at 1:36 AM,
              Ivan Noris <span dir="ltr"><<a moz-do-not-send="true"
                  href="mailto:Ivan.Noris@evolveum.com" target="_blank">Ivan.Noris@evolveum.com</a>></span>
              wrote:<br>
              <blockquote class="gmail_quote" style="margin:0 0 0
                .8ex;border-left:1px #ccc solid;padding-left:1ex">
                <div>
                  <div style="font-family:times new roman,new
                    york,times,serif;font-size:12pt;color:#000000">
                    <div>Hi Matt,<br>
                    </div>
                    <div>please try to increase -Xmx to at least 2048m
                      and see if it helps.<br>
                    </div>
                    <div><br>
                    </div>
                    <div>Ivan<br>
                    </div>
                    <div><br>
                    </div>
                    <hr>
                    <blockquote style="border-left:2px solid
#1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><b>From:
                      </b>"Matt Mencel" <<a moz-do-not-send="true"
                        href="mailto:mr-mencel@wiu.edu" target="_blank">mr-mencel@wiu.edu</a>><br>
                      <b>To: </b>"midPoint General Discussion" <<a
                        moz-do-not-send="true"
                        href="mailto:midpoint@lists.evolveum.com"
                        target="_blank">midpoint@lists.evolveum.com</a>><br>
                      <b>Sent: </b>Thursday, August 25, 2016 12:01:33
                      AM<br>
                      <b>Subject: </b>[midPoint] Running out of Java
                      Heap Space
                      <div>
                        <div class="h5"><br>
                          <div><br>
                          </div>
                          <div dir="ltr">Is there any further
                            recommendations for setting the Java memory
                            requirements?  My JAVA_OPTS is set to this.
                            <div><br>
                            </div>
                            <div><span
                                style="font-family:monospace,monospace">JAVA_OPTS="$JAVA_OPTS
                                $JSSE_OPTS -Dmidpoint.home=/opt/midpoint
-Djavax.net.ssl.trustStore=/opt/midpoint/keystore.jceks
                                -Djavax.net.ssl.trustStoreType=jceks
                                -XX:MaxPermSize=1024m -Xss1m -Xmx1024m"<br>
                              </span>
                              <div><br>
                              </div>
                              <div>I'm running into the error below
                                after an LDAP import task suspends due
                                to an error.  I have yet to resolve the
                                error, see my earlier post about how to
                                ignore an LDAP attribute on import. 
                                I've got plenty of memory on the VM, but
                                I may need to tune my Java even further.</div>
                              <div><br>
                              </div>
                              <div>Thanks,</div>
                              <div>Matt</div>
                              <div><br>
                              </div>
                              <div><br>
                              </div>
                              <div>
                                <div>2016-08-24 13:29:56,835 []
                                  [midPointScheduler_Worker-4] INFO
                                  (com.evolveum.midpoint.task.quartzimpl.execution.JobExecutor):
                                  Task encountered temporary error,
                                  suspending it. Task =
                                  Task(id:1472058712002-0-1, name:LDAP
                                  Import,
                                  oid:bfeccb74-5aa2-40f2-a107-18e120b765fc)</div>
                                <div>2016-08-24 13:29:56,836 []
                                  [midPointScheduler_Worker-4] INFO
                                  (com.evolveum.midpoint.task.quartzimpl.TaskManagerQuartzImpl):
                                  Suspending tasks
                                  [Task(id:1472058712002-0-1, name:LDAP
                                  Import,
                                  oid:bfeccb74-5aa2-40f2-a107-18e120b765fc)];
                                  do not stop tasks.</div>
                                <div>2016-08-24 13:51:21,162 []
                                  [NioProcessor-1] WARN
                                  (org.apache.directory.ldap.client.api.LdapNetworkConnection):
                                  Java heap space</div>
                                <div>java.lang.OutOfMemoryError: Java
                                  heap space</div>
                                <div>2016-08-24 13:52:46,208 []
                                  [NioProcessor-1] WARN
                                  (org.apache.directory.ldap.client.api.LdapNetworkConnection):
                                  Java heap space</div>
                                <div>java.lang.OutOfMemoryError: Java
                                  heap space</div>
                                <div>2016-08-24 14:01:27,961 []
                                  [ClusterManagerThread] ERROR
                                  (com.evolveum.midpoint.task.quartzimpl.cluster.ClusterManager):
                                  Unexpected exception in ClusterManager
                                  thread; continuing execution., reason:
                                  Java heap space (class
                                  java.lang.OutOfMemoryError)</div>
                                <div>2016-08-24 14:02:59,710 []
                                  [midPointScheduler_Worker-5] ERROR
                                  (org.quartz.core.JobRunShell): Job
                                  DEFAULT.00000000-0000-0000-0000-000000000006
                                  threw an unhandled Exception:</div>
                                <div>java.lang.OutOfMemoryError: Java
                                  heap space</div>
                                <div>2016-08-24 14:03:01,507 []
                                  [NioProcessor-2] WARN
                                  (org.apache.directory.ldap.client.api.LdapNetworkConnection):
                                  Java heap space</div>
                                <div>java.lang.OutOfMemoryError: Java
                                  heap space</div>
                                <div>2016-08-24 14:03:03,490 []
                                  [NioProcessor-2] WARN
                                  (org.apache.directory.ldap.client.api.LdapNetworkConnection):
                                  Java heap space</div>
                                <div>java.lang.OutOfMemoryError: Java
                                  heap space</div>
                                <div>2016-08-24 14:03:03,490 []
                                  [midPointScheduler_Worker-5] ERROR
                                  (org.quartz.core.ErrorLogger): Job
                                  (DEFAULT.00000000-0000-0000-0000-000000000006
                                  threw an exception.</div>
                                <div>org.quartz.SchedulerException: Job
                                  threw an unhandled exception.</div>
                                <div>        at
                                  org.quartz.core.JobRunShell.run(JobRunShell.java:224)
                                  ~[quartz-2.1.3.jar:na]</div>
                                <div>        at
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
                                  [quartz-2.1.3.jar:na]</div>
                                <div>Caused by:
                                  java.lang.OutOfMemoryError: Java heap
                                  space</div>
                                <div>2016-08-24 14:29:17,649 []
                                  [Thread-9] ERROR
                                  (org.activiti.engine.impl.jobexecutor.AcquireJobsRunnableImpl):
                                  exception during job acquisition: Java
                                  heap space</div>
                                <div>java.lang.OutOfMemoryError: Java
                                  heap space</div>
                                <div>        at
                                  sun.reflect.GeneratedConstructorAccessor5.newInstance(Unknown
                                  Source) ~[na:na]</div>
                                <div>        at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
                                  ~[na:1.8.0_101]</div>
                                <div>        at
                                  java.lang.reflect.Constructor.newInstance(Constructor.java:423)
                                  ~[na:1.8.0_101]</div>
                                <div>        at
                                  com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
~[mysql-connector-java-5.1.39-bin.jar:5.1.39]</div>
                                <div>        at
                                  com.mysql.jdbc.ResultSetImpl.getInstance(ResultSetImpl.java:319)
~[mysql-connector-java-5.1.39-bin.jar:5.1.39]</div>
                                <div>        at
                                  com.mysql.jdbc.MysqlIO.buildResultSetWithUpdates(MysqlIO.java:3187)
~[mysql-connector-java-5.1.39-bin.jar:5.1.39]</div>
                                <div>        at
                                  com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:3087)
~[mysql-connector-java-5.1.39-bin.jar:5.1.39]</div>
                                <div>        at
                                  com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:2336)
~[mysql-connector-java-5.1.39-bin.jar:5.1.39]</div>
                                <div>        at
                                  com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2729)
~[mysql-connector-java-5.1.39-bin.jar:5.1.39]</div>
                                <div>        at
                                  com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2545)
~[mysql-connector-java-5.1.39-bin.jar:5.1.39]</div>
                                <div>        at
                                  com.mysql.jdbc.ConnectionImpl.setAutoCommit(ConnectionImpl.java:4842)
~[mysql-connector-java-5.1.39-bin.jar:5.1.39]</div>
                                <div>        at
                                  sun.reflect.GeneratedMethodAccessor254.invoke(Unknown
                                  Source) ~[na:na]</div>
                                <div>        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                                  ~[na:1.8.0_101]</div>
                                <div>        at
                                  java.lang.reflect.Method.invoke(Method.java:498)
                                  ~[na:1.8.0_101]</div>
                                <div>        at
org.apache.ibatis.datasource.pooled.PooledConnection.invoke(PooledConnection.java:245)
                                  ~[mybatis-3.3.0.jar:3.3.0]</div>
                                <div>        at
                                  com.sun.proxy.$Proxy106.setAutoCommit(Unknown
                                  Source) ~[na:na]</div>
                                <div>        at
org.apache.ibatis.transaction.jdbc.JdbcTransaction.setDesiredAutoCommit(JdbcTransaction.java:104)
                                  ~[mybatis-3.3.0.jar:3.3.0]</div>
                                <div>        at
org.apache.ibatis.transaction.jdbc.JdbcTransaction.openConnection(JdbcTransaction.java:144)
                                  ~[mybatis-3.3.0.jar:3.3.0]</div>
                                <div>        at
org.apache.ibatis.transaction.jdbc.JdbcTransaction.getConnection(JdbcTransaction.java:62)
                                  ~[mybatis-3.3.0.jar:3.3.0]</div>
                                <div>        at
org.apache.ibatis.executor.BaseExecutor.getConnection(BaseExecutor.java:315)
                                  ~[mybatis-3.3.0.jar:3.3.0]</div>
                                <div>        at
org.apache.ibatis.executor.SimpleExecutor.prepareStatement(SimpleExecutor.java:75)
                                  ~[mybatis-3.3.0.jar:3.3.0]</div>
                                <div>        at
org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:61)
                                  ~[mybatis-3.3.0.jar:3.3.0]</div>
                                <div>        at
org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:303)
                                  ~[mybatis-3.3.0.jar:3.3.0]</div>
                                <div>        at
                                  org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:154)
                                  ~[mybatis-3.3.0.jar:3.3.0]</div>
                                <div>        at
org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:102)
                                  ~[mybatis-3.3.0.jar:3.3.0]</div>
                                <div>        at
org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:82)
                                  ~[mybatis-3.3.0.jar:3.3.0]</div>
                                <div>        at
org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:120)
                                  ~[mybatis-3.3.0.jar:3.3.0]</div>
                                <div>        at
org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:113)
                                  ~[mybatis-3.3.0.jar:3.3.0]</div>
                                <div>        at
org.activiti.engine.impl.db.DbSqlSession.selectListWithRawParameter(DbSqlSession.java:438)
~[activiti-engine-5.19.0.2.jar:5.19.0.2]</div>
                                <div>        at
org.activiti.engine.impl.db.DbSqlSession.selectList(DbSqlSession.java:429)
~[activiti-engine-5.19.0.2.jar:5.19.0.2]</div>
                                <div>        at
org.activiti.engine.impl.db.DbSqlSession.selectList(DbSqlSession.java:424)
~[activiti-engine-5.19.0.2.jar:5.19.0.2]</div>
                                <div>        at
org.activiti.engine.impl.db.DbSqlSession.selectList(DbSqlSession.java:411)
~[activiti-engine-5.19.0.2.jar:5.19.0.2]</div>
                              </div>
                              <div><br>
                              </div>
                            </div>
                          </div>
                          <br>
                        </div>
                      </div>
                      _______________________________________________<br>
                      midPoint mailing list<br>
                      <a moz-do-not-send="true"
                        href="mailto:midPoint@lists.evolveum.com"
                        target="_blank">midPoint@lists.evolveum.com</a><br>
                      <a moz-do-not-send="true"
                        href="http://lists.evolveum.com/mailman/listinfo/midpoint"
                        target="_blank">http://lists.evolveum.com/mailman/listinfo/midpoint</a><span
                        class="HOEnZb"><span style="color: #888888;"
                          data-mce-style="color: #888888;"
                          color="#888888"><br>
                        </span></span></blockquote>
                    <span class="HOEnZb"><span style="color: #888888;"
                        data-mce-style="color: #888888;" color="#888888">
                        <div><br>
                          <div><br>
                          </div>
                        </div>
                        <div><br>
                        </div>
                        <div>-- <br>
                        </div>
                        <div><span></span>Ivan Noris<br>
                          Senior Identity Engineer<br>
                          <a moz-do-not-send="true"
                            href="http://evolveum.com" target="_blank">evolveum.com</a><span></span><br>
                        </div>
                      </span></span></div>
                </div>
                <br>
                _______________________________________________<br>
                midPoint mailing list<br>
                <a moz-do-not-send="true"
                  href="mailto:midPoint@lists.evolveum.com"
                  target="_blank">midPoint@lists.evolveum.com</a><br>
                <a moz-do-not-send="true"
                  href="http://lists.evolveum.com/mailman/listinfo/midpoint"
                  rel="noreferrer" target="_blank">http://lists.evolveum.com/mailman/listinfo/midpoint</a><br>
                <br>
              </blockquote>
            </div>
            <br>
          </div>
          <br>
          <span style="font-size: small;" data-mce-style="font-size:
            small;" size="2"><br>
            <div><br>
            </div>
            CONFIDENTIALITY NOTICE:<br>
            This e-mail together with any attachments is proprietary and
            confidential; intended for only the recipient(s) named above
            and may contain information that is privileged. You should
            not retain, copy or use this e-mail or any attachments for
            any purpose, or disclose all or any part of the contents to
            any person. Any views or opinions expressed in this e-mail
            are those of the author and do not represent those of the
            Baptist School of Health Professions. If you have received
            this e-mail in error, or are not the named recipient(s), you
            are hereby notified that any review, dissemination,
            distribution or copying of this communication is prohibited
            by the sender and to do so might constitute a violation of
            the Electronic Communications Privacy Act, 18 U.S.C. section
            2510-2521. Please immediately notify the sender and delete
            this e-mail and any attachments from your computer. </span><br>
          <div><br>
          </div>
          _______________________________________________<br>
          midPoint mailing list<br>
          <a class="moz-txt-link-abbreviated" href="mailto:midPoint@lists.evolveum.com">midPoint@lists.evolveum.com</a><br>
          <a class="moz-txt-link-freetext" href="http://lists.evolveum.com/mailman/listinfo/midpoint">http://lists.evolveum.com/mailman/listinfo/midpoint</a><br>
        </blockquote>
        <div><br>
          <br>
        </div>
        <div><br>
        </div>
        <div>-- <br>
        </div>
        <div><span name="x"></span>Ivan Noris<br>
          Senior Identity Engineer<br>
          evolveum.com<span name="x"></span><br>
        </div>
      </div>
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <br>
      <pre wrap="">_______________________________________________
midPoint mailing list
<a class="moz-txt-link-abbreviated" href="mailto:midPoint@lists.evolveum.com">midPoint@lists.evolveum.com</a>
<a class="moz-txt-link-freetext" href="http://lists.evolveum.com/mailman/listinfo/midpoint">http://lists.evolveum.com/mailman/listinfo/midpoint</a>
</pre>
    </blockquote>
    <br>
  </body>
</html>