[midPoint] [External] RE: Unable to deploy midPoint 4.8 with postgresql using Docker

lazy tech huutm at lazytech.biz.vn
Wed Nov 22 10:26:35 CET 2023 

Hi Drew, 

The guide worked fine. Thanks for your support.



Huutm,







---- Trên Wed, 22 Nov 2023 13:38:22 +0700 Drew Roberts <aroberts at apu.edu> đã viết ---



Hey Huttm & Deshak,

After reviewing this further I realized I missed a few things that were in my config and are essential to make this work. I really hope there is a simpler way to do a native postgresql database because the way I got it working is ridiculous for the new recommended default option. Anyways, here's a more complete guide to my maniac method:

Add these variables to your midPoint container:

MP_SET_midpoint_repository_type=native

MP_SET_midpoint_repository_jdbcUrl=jdbc:postgresql://db.at.your.address:5432/midpoint

MP_SET_midpoint_repository_jdbcUsername=midpoint

MP_SET_midpoint_repository_jdbcPassword_FILE=/run/secrets/mp_database_password.txt

MP_SET_midpoint_keystore_keyStorePassword_FILE=/run/secrets/mp_keystore_password.txt


Make sure your midPoint container does NOT have these variables:

- REPO_DATABASE_TYPE=postgresql     

- REPO_HOST=midpoint_data     

- REPO_DATABASE=midpoint_db     

- REPO_USER=midpoint     

- REPO_PASSWORD_FILE=/run/secrets/mp_database_password.txt

- MP_KEYSTORE_PASSWORD_FILE=/run/secrets/mp_keystore_password.txt


Create a config.xml file and put it in /opt/midpoint/var on your midPoint container (you could probably fill in these variables instead of using the MP_SET stuff above if you want. I prefer to use environment variables when I can): <?xml version="1.0"?> <!--

  ~ Copyright (C) 2010-2021 Evolveum and contributors

  ~

  ~ This work is dual-licensed under the Apache License 2.0

  ~ and European Union Public License. See LICENSE file for details.

  -->


<!--

Example config.xml for Native PG repository.

To use it:

1. Copy it to $MIDPOINT_HOME, e.g. under midpoint.../var directory in the extracted archive.

2. Rename the copied file in $MIDPOINT_HOME to config.xml.

3. Check DB connection parameters and change them as needed.

4. Start midPoint (bin/start.sh) and observe the success in $MIDPOINT_HOME/log/midpoint.log.


For more info see:


* midPoint home:

https://docs.evolveum.com/midpoint/reference/deployment/midpoint-home-directory/


* Overriding config.xml parameters from command line:

https://docs.evolveum.com/midpoint/reference/deployment/midpoint-home-directory/overriding-config-xml-parameters/


* Repository configuration:

https://docs.evolveum.com/midpoint/reference/repository/configuration/


* Canonical config.xml auto-created if not present in $MIDPOINT_HOME (master branch):

https://github.com/Evolveum/midpoint/blob/master/repo/system-init/src/main/resources/config.xml

-->




<configuration>
    <midpoint>

        <webApplication>

            <importFolder>${midpoint.home}/import</importFolder>

        </webApplication>

        <repository>

            <type>native</type>

            <jdbcUrl>jdbc:postgresql://localhost:5432/midpoint</jdbcUrl>

            <jdbcUsername>midpoint</jdbcUsername>

            <jdbcPassword>password</jdbcPassword>

        </repository>

        <audit>

            <auditService>

                <auditServiceFactoryClass>com.evolveum.midpoint.audit.impl.LoggerAuditServiceFactory</auditServiceFactoryClass>

            </auditService>

            <auditService>

                <auditServiceFactoryClass>com.evolveum.midpoint.repo.sqale.audit.SqaleAuditServiceFactory</auditServiceFactoryClass>

            </auditService>

        </audit>

        <icf>

            <scanClasspath>true</scanClasspath>

            <scanDirectory>${midpoint.home}/icf-connectors</scanDirectory>

        </icf>

        <keystore>

            <keyStorePath>${midpoint.home}/keystore.jceks</keyStorePath>

            <keyStorePassword>changeit</keyStorePassword>

            <encryptionKeyAlias>default</encryptionKeyAlias>

        </keystore>

    </midpoint>

</configuration>




Set things so your midPoint container doesn't boot til after your database is configured

Recreate
 this logic somehow on the database container for the first launch 
(maybe make an init container to run these commands or something?):
sudo su - postgres
psql
CREATE USER midpoint WITH PASSWORD '<PASSWORD>' LOGIN SUPERUSER; 
CREATE
 DATABASE midpointdev WITH OWNER = midpoint ENCODING = 'UTF8' TABLESPACE
 = pg_default LC_COLLATE = 'en_US.UTF-8' LC_CTYPE = 'en_US.UTF-8' 
CONNECTION LIMIT = -1;
\q

curl https://raw.githubusercontent.com/Evolveum/midpoint/support-4.8/config/sql/native/postgres.sql --output postgres.sql
curl https://raw.githubusercontent.com/Evolveum/midpoint/support-4.8/config/sql/native/postgres-audit.sql --output postgres-audit.sql
curl https://raw.githubusercontent.com/Evolveum/midpoint/support-4.8/config/sql/native/postgres-quartz.sql --output postgres-quartz.sql

psql -h localhost -d midpointdev -U midpoint -f postgres.sql -f postgres-audit.sql -f postgres-quartz.sql




I hope this time I've included everything you need to get it going. Apologies for the incomplete information before.





On Tue, Nov 21, 2023 at 9:37 PM lazy tech <mailto:huutm at lazytech.biz.vn> wrote:






-- 
Drew Roberts  |  Systems Administrator II
IT Platform Engineering, Azusa Pacific University
https://apu.edu 





Hello Drew,

I followed the same steps as outlined in your guide, but I am still encountering an error. Could you provide further assistance?​



2023-11-22 12:05:30     ... 94 common frames omitted 

2023-11-22 12:05:30 Caused by: com.evolveum.midpoint.util.exception.SystemException: AuditServiceFactory implementation class com.evolveum.midpoint.repo.sql.SqlAuditServiceFactory failed to initialize: Couldn't find AuditServiceFactory for class com.evolveum.midpoint.repo.sql.SqlAuditServiceFactory

2023-11-22 12:05:30     at com.evolveum.midpoint.init.AuditFactory.init(AuditFactory.java:84)

2023-11-22 12:05:30     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

2023-11-22 12:05:30     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)

2023-11-22 12:05:30     at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

2023-11-22 12:05:30     at java.base/java.lang.reflect.Method.invoke(Method.java:568)

2023-11-22 12:05:30     at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMethod.invoke(InitDestroyAnnotationBeanPostProcessor.java:457)

2023-11-22 12:05:30     at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:401)

2023-11-22 12:05:30     at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:219)

2023-11-22 12:05:30     ... 115 common frames omitted

2023-11-22 12:05:30 Caused by: com.evolveum.midpoint.util.exception.SystemException: Couldn't find AuditServiceFactory for class com.evolveum.midpoint.repo.sql.SqlAuditServiceFactory

2023-11-22 12:05:30     at com.evolveum.midpoint.init.AuditFactory.getFactory(AuditFactory.java:97)

2023-11-22 12:05:30     at com.evolveum.midpoint.init.AuditFactory.init(AuditFactory.java:75)

2023-11-22 12:05:30     ... 122 common frames omitted

2023-11-22 12:05:30 ERROR initializing midPoint: org.springframework.context.ApplicationContextException: Unable to start web server

2023-11-22 12:05:30 ROOT cause: com.evolveum.midpoint.util.exception.SystemException: Couldn't find AuditServiceFactory for class com.evolveum.midpoint.repo.sql.SqlAuditServiceFactory

2023-11-22 12:05:30 See midpoint.log for more details.





Thanks,

Huutm







---- Trên Tue, 14 Nov 2023 23:42:18 +0700 Drew Roberts via midPoint <mailto:midpoint at lists.evolveum.com> đã viết ---



Hey Deshak,



That's the error you should get if you only did the first step I gave you. Run the commands I gave you in step 3 on the database, then reboot the midpoint container and your issue should be resolved.



On Tue, Nov 14, 2023 at 4:45 AM BHATNAGAR Deshak <mailto:deshak.bhatnagar at thalesgroup.com> wrote:

THALES GROUP LIMITED DISTRIBUTION to email recipients 

 

Hello Drew,

 

Thanks for the swift reply and the possible solution.

 

I tried the same as explained below but faced another error:

 

ERROR initializing midPoint: org.springframework.context.ApplicationContextException: Unable to start web server

ROOT cause: org.postgresql.util.PSQLException: ERROR: relation "m_global_metadata" does not exist

  Position: 31

 

I have attached the complete error below in the error 14 Nov 2023 file.

 

It would be great if you can share the docker-compose & and any other adjustments or changes you have made to your setup, so I can get a better understanding of it. 

 

Kind regards,

Deshak Bhatnagar 

From: Drew Roberts <mailto:aroberts at apu.edu> 
 Sent: Saturday, November 11, 2023 3:50 AM
 To: midPoint General Discussion <mailto:midpoint at lists.evolveum.com>
 Cc: BHATNAGAR Deshak <mailto:deshak.bhatnagar at thalesgroup.com>
 Subject: Re: [External] Re: [midPoint] Unable to deploy midPoint 4.8 with postgresql using Docker

 

Hey Deshak,


 


If you take a look at the https://docs.evolveum.com/midpoint/release/4.8/ and use your browser's Find function with the term "native", a couple results in you'll see:


Generic Repository with PostgreSQL is not supported, if you are using PostgreSQL with generic repository, please migrate to https://docs.evolveum.com/midpoint/reference/master/repository/native-postgresql/. 




I believe the issue you're experiencing is due to this change. My environment and setup are slightly different from yours so your mileage may vary but here are my two cents:


Add a MP_SET_midpoint_repository_type=native environment variable to your midPoint container

Set things so your midPoint container doesn't boot til after your database is configured

Recreate this logic somehow on the database container for the first launch (maybe make an init container to run these commands or something?):


sudo su - postgres
 psql
 CREATE USER midpoint WITH PASSWORD '<PASSWORD>' LOGIN SUPERUSER; 
 CREATE DATABASE midpointdev WITH OWNER = midpoint ENCODING = 'UTF8' TABLESPACE = pg_default LC_COLLATE = 'en_US.UTF-8' LC_CTYPE = 'en_US.UTF-8' CONNECTION LIMIT = -1;
 \q
 
 curl https://raw.githubusercontent.com/Evolveum/midpoint/support-4.8/config/sql/native/postgres.sql --output postgres.sql
 curl https://raw.githubusercontent.com/Evolveum/midpoint/support-4.8/config/sql/native/postgres-audit.sql --output postgres-audit.sql
 curl https://raw.githubusercontent.com/Evolveum/midpoint/support-4.8/config/sql/native/postgres-quartz.sql --output postgres-quartz.sql
 
 psql -h localhost -d midpointdev -U midpoint -f postgres.sql -f postgres-audit.sql -f postgres-quartz.sql



 

On Fri, Nov 10, 2023 at 8:37 AM BHATNAGAR Deshak via midPoint <mailto:midpoint at lists.evolveum.com> wrote:


THALES GROUP LIMITED DISTRIBUTION to email recipients 

 

Added the docker-compose for reference.

 

Thanks

From: BHATNAGAR Deshak 
 Sent: Friday, November 10, 2023 9:43 PM
 To: mailto:midpoint at lists.evolveum.com
 Subject: Unable to deploy midPoint 4.8 with postgresql using Docker


 

THALES GROUP LIMITED DISTRIBUTION to email recipients 

 

Hello midpoint community,

I am deploying the Evolveum midpoint 4.8 with postgresql using Docker but I am facing the below mentioned error:

 

ERROR initializing midPoint: org.springframework.context.ApplicationContextException: Unable to start web server

ROOT cause: com.evolveum.midpoint.util.exception.SystemException: DB script (/sql/postgresql-4.6-all.sql) couldn't be found

 

This above error occurs because the system is not able to create required database tables, the log file is attached below (log file (error))

 

I was previously using the LTS version midpoint 4.4.4 and used the same docker-compose file with changes to midpoint 4.8 image and postgresql-16 image.

 

I have tried with all the recommended version of both midpoint 4.8 as well as postgresql db.

 

But the error remains the same in all cases.

 

Also, a similar error was reported on the Jira of Evolveum regarding the schema validation in which the database tables were not created: https://jira.evolveum.com/projects/MID/issues/MID-9258?filter=doneissues  

 

As per the conclusion drawn after my troubleshooting. 

I found out that the config/sql file in midpoint 4.8 repository on Github do not have the above-mentioned file in the error i.e., postgresql-4.6-all.sql (https://github.com/Evolveum/midpoint/tree/v4.8/config/sql/native). 

 

In-comparison, all of the previous versions of midpoint on the Github had the file (postgresql-4.6-all.sql) in their respective repositories, for example: midpoint 4.7.2 had the
 file postgresql-4.6-all.sql in it on the Github (https://github.com/Evolveum/midpoint/tree/v4.7.2/config/sql/native-new). 

 

So, I suspect that the issue is caused due to the file being missing in the repositories. 

 

Please resolve the above issue or let me know the way that I can resolve this. 

 

docker-compose is also attached for the reference.

 

Thanks.

 

Kind regards,

Deshak Bhatnagar



_______________________________________________
 midPoint mailing list
 mailto:midPoint at lists.evolveum.com
 https://lists.evolveum.com/mailman/listinfo/midpoint





-- 
Drew Roberts  |  Systems Administrator II
 IT Platform Engineering, Azusa Pacific University
 https://apu.edu 
 
 










-- 
Drew Roberts  |  Systems Administrator II
IT Platform Engineering, Azusa Pacific University
https://apu.edu 




_______________________________________________
midPoint mailing list
mailto:midPoint at lists.evolveum.com
https://lists.evolveum.com/mailman/listinfo/midpoint
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20231122/27309da9/attachment-0001.htm>

More information about the midPoint mailing list