[midPoint] [Newsletter] Re: [Newsletter] Re: [Newsletter] Re: [Newsletter] Re: [Newsletter] Re: [Newsletter] Re: Migrating from 3.7.x to 3.8 with custom schema

Chris Woods Chris.Woods at rohde-schwarz.com
Tue Jul 31 13:41:44 CEST 2018


Hi Pavol,

pretty sure. I’ll have another attempt when I get back from holiday. Thanks for checking – it’s only our PoC environment anyway, so if the worst comes to the worst I can reimport everything from the resources.

Cheers,
Chris.

From: midPoint <midpoint-bounces at lists.evolveum.com> On Behalf Of Pavol Mederly
Sent: Monday, July 23, 2018 7:52 PM
To: midpoint at lists.evolveum.com
Subject: *EXT* [Newsletter] Re: [midPoint] [Newsletter] Re: [Newsletter] Re: [Newsletter] Re: [Newsletter] Re: [Newsletter] Re: Migrating from 3.7.x to 3.8 with custom schema


Hello Chris,

the spelling difference is OK.

I have now used your XSD in my case and run the export. It is as it should be:

<service xmlns="http://midpoint.evolveum.com/xml/ns/public/common/common-3"<http://midpoint.evolveum.com/xml/ns/public/common/common-3> xmlns:c="http://midpoint.evolveum.com/xml/ns/public/common/common-3"<http://midpoint.evolveum.com/xml/ns/public/common/common-3> xmlns:icfs="http://midpoint.evolveum.com/xml/ns/public/connector/icf-1/resource-schema-3"<http://midpoint.evolveum.com/xml/ns/public/connector/icf-1/resource-schema-3> xmlns:org="http://midpoint.evolveum.com/xml/ns/public/common/org-3"<http://midpoint.evolveum.com/xml/ns/public/common/org-3> xmlns:q="http://prism.evolveum.com/xml/ns/public/query-3"<http://prism.evolveum.com/xml/ns/public/query-3> xmlns:ri="http://midpoint.evolveum.com/xml/ns/public/resource/instance-3"<http://midpoint.evolveum.com/xml/ns/public/resource/instance-3> xmlns:t="http://prism.evolveum.com/xml/ns/public/types-3"<http://prism.evolveum.com/xml/ns/public/types-3> oid="bfb7ae2b-2473-46df-9c12-eb62103d3eca" version="5">
    <name>Extension 123</name>
    <extension xmlns:gen633="http://sample.evolveum.com/xml/ns/sample-idm/extension"<http://sample.evolveum.com/xml/ns/sample-idm/extension>>
        <gen633:rsExtensionFirstName xmlns:xsd="http://www.w3.org/2001/XMLSchema"<http://www.w3.org/2001/XMLSchema> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"<http://www.w3.org/2001/XMLSchema-instance> xsi:type="xsd:string">Foo</gen633:rsExtensionFirstName>
        <gen633:rsExtensionLastName xmlns:xsd="http://www.w3.org/2001/XMLSchema"<http://www.w3.org/2001/XMLSchema> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"<http://www.w3.org/2001/XMLSchema-instance> xsi:type="xsd:string">Bar</gen633:rsExtensionLastName>
        <gen633:rsExtensionNumber xmlns:xsd="http://www.w3.org/2001/XMLSchema"<http://www.w3.org/2001/XMLSchema> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"<http://www.w3.org/2001/XMLSchema-instance> xsi:type="xsd:string">123</gen633:rsExtensionNumber>
        <gen633:rsExtensionDecomissioned xmlns:xsd="http://www.w3.org/2001/XMLSchema"<http://www.w3.org/2001/XMLSchema> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"<http://www.w3.org/2001/XMLSchema-instance> xsi:type="xsd:boolean">false</gen633:rsExtensionDecomissioned>
    </extension>

The file used to export is the one present in https://evolveum.com/downloads/midpoint/3.7.2/midpoint-3.7.2-dist.zip

i.e.

ninja.jar, 73380648 bytes, SHA1 BA7B17AD64385ED098400A8454890316EDE413E4.

it is a bit different from ninja-3.7.2.jar from 3.8 distribution, which is 73380649 bytes long with SHA1 27320463CE95EDFB40EDFC009CEBA89FAAD505AD (but the difference seems to be only because it was built on a different time).

For me, both ninja JARs produce the correct result.

Are you really sure you are using one of these to generate the export?

Best regards,

Pavol Mederly

Software developer

evolveum.com


On 23.07.2018 16:22, Chris Woods wrote:
Hi Pavol,

no Problem.  I noticed the spelling mistake in “SericeExtensionType” -> instead of ServiceExtensionType however. As far as I know this was in the PoC as well, though.

Cheers,
Chris.

From: midPoint <midpoint-bounces at lists.evolveum.com><mailto:midpoint-bounces at lists.evolveum.com> On Behalf Of Pavol Mederly
Sent: Monday, July 23, 2018 12:06 PM
To: midpoint at lists.evolveum.com<mailto:midpoint at lists.evolveum.com>
Subject: *EXT* [Newsletter] Re: [midPoint] [Newsletter] Re: [Newsletter] Re: [Newsletter] Re: [Newsletter] Re: Migrating from 3.7.x to 3.8 with custom schema


Hello Chris,

this is strange indeed. I have tried to replicate your problem using freshly built midPoint 3.7.2:

1. Took a clean repository
2. Created an extension XSD file containing the 4 attributes (rsExtensionFirstName, rsExtensionLastName, rsExtensionNumber, rsExtensionDecommisioned)
3. Imported your service object (with xsi:type removed)
4. Stopped midPoint and exported the object using ninja from 3.7.2

Got this - exactly as it should look like:

<service xmlns="http://midpoint.evolveum.com/xml/ns/public/common/common-3"<http://midpoint.evolveum.com/xml/ns/public/common/common-3> xmlns:c="http://midpoint.evolveum.com/xml/ns/public/common/common-3"<http://midpoint.evolveum.com/xml/ns/public/common/common-3> xmlns:icfs="http://midpoint.evolveum.com/xml/ns/public/connector/icf-1/resource-schema-3"<http://midpoint.evolveum.com/xml/ns/public/connector/icf-1/resource-schema-3> xmlns:org="http://midpoint.evolveum.com/xml/ns/public/common/org-3"<http://midpoint.evolveum.com/xml/ns/public/common/org-3> xmlns:q="http://prism.evolveum.com/xml/ns/public/query-3"<http://prism.evolveum.com/xml/ns/public/query-3> xmlns:ri="http://midpoint.evolveum.com/xml/ns/public/resource/instance-3"<http://midpoint.evolveum.com/xml/ns/public/resource/instance-3> xmlns:t="http://prism.evolveum.com/xml/ns/public/types-3"<http://prism.evolveum.com/xml/ns/public/types-3> oid="bfb7ae2b-2473-46df-9c12-eb62103d3eca" version="5">
    <name>Extension 123</name>
    <extension xmlns:gen118="http://sample.evolveum.com/xml/ns/sample-idm/extension"<http://sample.evolveum.com/xml/ns/sample-idm/extension>>
        <gen118:rsExtensionFirstName xmlns:xsd="http://www.w3.org/2001/XMLSchema"<http://www.w3.org/2001/XMLSchema> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"<http://www.w3.org/2001/XMLSchema-instance> xsi:type="xsd:string">Foo</gen118:rsExtensionFirstName>
        <gen118:rsExtensionLastName xmlns:xsd="http://www.w3.org/2001/XMLSchema"<http://www.w3.org/2001/XMLSchema> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"<http://www.w3.org/2001/XMLSchema-instance> xsi:type="xsd:string">Bar</gen118:rsExtensionLastName>
        <gen118:rsExtensionNumber xmlns:xsd="http://www.w3.org/2001/XMLSchema"<http://www.w3.org/2001/XMLSchema> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"<http://www.w3.org/2001/XMLSchema-instance> xsi:type="xsd:string">123</gen118:rsExtensionNumber>
        <gen118:rsExtensionDecomissioned xmlns:xsd="http://www.w3.org/2001/XMLSchema"<http://www.w3.org/2001/XMLSchema> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"<http://www.w3.org/2001/XMLSchema-instance> xsi:type="xsd:boolean">false</gen118:rsExtensionDecomissioned>
    </extension>

Maybe the difference is in the XSD file. Please, could you send it to me (maybe privately, if needed).

Best regards,

Pavol Mederly

Software developer

evolveum.com


On 09.07.2018 7:35, Chris Woods wrote:
Hi Pavol,

same result, I’m afraid. I even tried with the ninja.jar from the 3.7.2 distribution (1 byte size difference).

the resulting output is the same (except, of course for the genXXX)

<service xmlns="http://midpoint.evolveum.com/xml/ns/public/common/common-3"<http://midpoint.evolveum.com/xml/ns/public/common/common-3> xmlns:c="http://midpoint.evolveum.com/xml/ns/public/common/common-3"<http://midpoint.evolveum.com/xml/ns/public/common/common-3> xmlns:icfs="http://midpoint.evolveum.com/xml/ns/public/connector/icf-1/resource-schema-3"<http://midpoint.evolveum.com/xml/ns/public/connector/icf-1/resource-schema-3> xmlns:org="http://midpoint.evolveum.com/xml/ns/public/common/org-3"<http://midpoint.evolveum.com/xml/ns/public/common/org-3> xmlns:q="http://prism.evolveum.com/xml/ns/public/query-3"<http://prism.evolveum.com/xml/ns/public/query-3> xmlns:ri="http://midpoint.evolveum.com/xml/ns/public/resource/instance-3"<http://midpoint.evolveum.com/xml/ns/public/resource/instance-3> xmlns:t="http://prism.evolveum.com/xml/ns/public/types-3"<http://prism.evolveum.com/xml/ns/public/types-3> oid="bfb7ae2b-2473-46df-9c12-eb62103d3eca" version="5">
    <name>Extension 123</name>
    <extension xmlns:gen802="http://sample.evolveum.com/xml/ns/sample-idm/extension"<http://sample.evolveum.com/xml/ns/sample-idm/extension> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"<http://www.w3.org/2001/XMLSchema-instance> xsi:type="c:ExtensionType">
        <gen802:rsExtensionFirstName>Foo</gen802:rsExtensionFirstName>
        <gen802:rsExtensionLastName>Bar</gen802:rsExtensionLastName>
        <gen802:rsExtensionNumber>123</gen802:rsExtensionNumber>
        <gen802:rsExtensionDecomissioned>false</gen802:rsExtensionDecomissioned>
    </extension>

Cheers,
Chris.



From: midPoint <midpoint-bounces at lists.evolveum.com><mailto:midpoint-bounces at lists.evolveum.com> On Behalf Of Pavol Mederly
Sent: Friday, July 6, 2018 2:51 PM
To: midpoint at lists.evolveum.com<mailto:midpoint at lists.evolveum.com>
Subject: *EXT* [Newsletter] Re: [midPoint] [Newsletter] Re: [Newsletter] Re: [Newsletter] Re: Migrating from 3.7.x to 3.8 with custom schema


Chris,

yes, please try that and let us know.

Pavol Mederly

Software developer

evolveum.com


On 06.07.2018 14:47, Chris Woods wrote:
Hi Pavol,

my ninja-3.7.2.jar is also 73380649 bytes with sha1sum 27320463ce95edfb40edfc009ceba89faad505ad. I still have the old database so it’s no problem to re-export and see if the new export is different.


Cheers,
Chris.

From: midPoint <midpoint-bounces at lists.evolveum.com><mailto:midpoint-bounces at lists.evolveum.com> On Behalf Of Pavol Mederly
Sent: Friday, July 6, 2018 11:34 AM
To: midpoint at lists.evolveum.com<mailto:midpoint at lists.evolveum.com>
Subject: *EXT* [Newsletter] Re: [midPoint] [Newsletter] Re: [Newsletter] Re: Migrating from 3.7.x to 3.8 with custom schema


Hello Chris,

yes, this is exactly the same as in that other case.

Which is really strange. Please, could you tell me which ninja-3.7.2.jar you used to do the export?

Mine - which produces correct xsi:type information - has 73,380,649 bytes, is dated 13.6.2018 10:22, and its sha1 is 27320463CE95EDFB40EDFC009CEBA89FAAD505AD.
Just btw, it is possible to fix the export XML simply by deleting xsi:type="c:ExtensionType" string. But if you used wrong JAR, it is fairly possible that the export of attributes in shadows will be wrong (xsi:type is obligatory for their values).

Best regards,





Pavol Mederly

Software developer

evolveum.com
On 06.07.2018 11:27, Chris Woods wrote:

Hi Pavol,



yes, no problem - there are quite a few objects that can't be imported, but here's a snippet from one of the services (Radovan should recognize it ;-): btw. the attribute is rsExtensionXXX because it is supposed to represent a telephone extension - and nothing to do with schema extension :-)



<extension xmlns:gen774="http://sample.evolveum.com/xml/ns/sample-idm/extension"<http://sample.evolveum.com/xml/ns/sample-idm/extension> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"<http://www.w3.org/2001/XMLSchema-instance> xsi:type="c:ExtensionType">

        <gen774:rsExtensionFirstName>Foo</gen774:rsExtensionFirstName>

        <gen774:rsExtensionLastName>Bar</gen774:rsExtensionLastName>

        <gen774:rsExtensionNumber>123</gen774:rsExtensionNumber>

        <gen774:rsExtensionDecomissioned>false</gen774:rsExtensionDecomissioned>

    </extension>



Regards,

Chris.



-----Original Message-----

From: midPoint <midpoint-bounces at lists.evolveum.com><mailto:midpoint-bounces at lists.evolveum.com> On Behalf Of Pavol Mederly

Sent: Friday, July 6, 2018 8:40 AM

To: midpoint at lists.evolveum.com<mailto:midpoint at lists.evolveum.com>

Subject: *EXT* [Newsletter] Re: [midPoint] [Newsletter] Re: Migrating from 3.7.x to 3.8 with custom schema



Hello Chris,



I have seen such an error recently. In that particular case it was probably caused by doing the export by a different version of ninja-3.7, resulting in wrong placement of xsi:type declarations in resulting XML.



Could you paste a snipped of <extension> element of an object that couldn't be imported in your case just to verify this?



Best regards,



Pavol Mederly

Software developer

evolveum.com



On 05.07.2018 14:05, Wojciech Staszewski wrote:

Hmmm... then I have no idea how to import this data. Maybe someone

from Evolveum should reply.

Regards!

WS



W dniu 05.07.2018 o 13:37, Chris Woods pisze:

Hi Wojciech,



I kept the installation directory the same (I used the upgrade.sh

scripts from the 3.8 dist beforehand) so the schema extensions are

still there. Originally I didn't start midpoint 3.8 before starting

the import (there is now mention in the upgrade instructions that

this is necessary). Even after starting midpoint and then retrying

the import I still received the same errors.



Cheers,

Chris.



-----Original Message-----

From: midPoint <midpoint-bounces at lists.evolveum.com><mailto:midpoint-bounces at lists.evolveum.com> On Behalf Of

Wojciech Staszewski

Sent: Thursday, July 5, 2018 1:23 PM

To: midpoint at lists.evolveum.com<mailto:midpoint at lists.evolveum.com>

Subject: *EXT* [Newsletter] Re: [midPoint] Migrating from 3.7.x to

3.8 with custom schema



Is your custom schema xml copied to the v3.8 and was MidPoint

restarted after that?



W dniu 05.07.2018 o 12:22, Chris Woods pisze:

Hi!



I am currently trying to migrate our 3.7.2 installation (PoC) to

3.8. The export worked fine (after realizing I also needed to use –U

with the jdbc url,not just the –j option for the jdbc driver). I

also have my backup.zip containing all the objects – these were

backed up from our existing MySQL 5.6 database.



I am, however, restoring to a fresh MySQL v8.0 database (used the

3.8 create script to create the tables). The only problem seems to

be with objects that have a value in an attribute from our custom

schema – ninja skips those objects with the following Exception(s):



com.evolveum.midpoint.util.exception.SchemaException: Item

'{http://sample.evolveum.com/xml/ns/sample-idm/extension}rsCompany'

without definition can't be saved.



          at

com.evolveum.midpoint.repo.sql.helpers.ObjectUpdater.createDataObjec

tF

romJAXB(ObjectUpdater.java:604) ~[repo-sql-impl-3.8.jar!/:na]



          at

com.evolveum.midpoint.repo.sql.helpers.ObjectUpdater.addObjectAttemp

t(

ObjectUpdater.java:131) ~[repo-sql-impl-3.8.jar!/:na]



          at

com.evolveum.midpoint.repo.sql.SqlRepositoryServiceImpl.addObject(Sq

lR

epositoryServiceImpl.java:431) ~[repo-sql-impl-3.8.jar!/:na]



          at

com.evolveum.midpoint.ninja.action.worker.ImportConsumerWorker.run(I

mp

ortConsumerWorker.java:65) ~[classes!/:na]



          at

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor

.j

ava:1142) [na:1.8.0_131]



          at

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.

java:617) [na:1.8.0_131]



          at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]



Caused by: java.lang.reflect.InvocationTargetException: null



          at sun.reflect.GeneratedMethodAccessor309.invoke(Unknown

Source) ~[na:na]



          at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcce

ss

orImpl.java:43) ~[na:1.8.0_131]



          at java.lang.reflect.Method.invoke(Method.java:498)

~[na:1.8.0_131]



          at

com.evolveum.midpoint.repo.sql.helpers.ObjectUpdater.createDataObjec

tF

romJAXB(ObjectUpdater.java:594) ~[repo-sql-impl-3.8.jar!/:na]



          ... 6 common frames omitted



Caused by:

com.evolveum.midpoint.repo.sql.util.DtoTranslationException: Item

'{http://sample.evolveum.com/xml/ns/sample-idm/extension}rsCompany'

without definition can't be saved.



          at

com.evolveum.midpoint.repo.sql.data.common.RObject.copyFromJAXB(RObj

ec

t.java:760) ~[repo-sql-impl-3.8.jar!/:na]



          at

com.evolveum.midpoint.repo.sql.data.common.RObject.copyFromJAXB(RObj

ec

t.java:726) ~[repo-sql-impl-3.8.jar!/:na]



          at

com.evolveum.midpoint.repo.sql.data.common.RFocus.copyFromJAXB(RFocus.

java:312) ~[repo-sql-impl-3.8.jar!/:na]



          at

com.evolveum.midpoint.repo.sql.data.common.RUser.copyFromJAXB(RUser.

ja

va:248) ~[repo-sql-impl-3.8.jar!/:na]



          ... 10 common frames omitted



Caused by: java.lang.IllegalArgumentException: Item

'{http://sample.evolveum.com/xml/ns/sample-idm/extension}rsCompany'

without definition can't be saved.



          at

org.apache.commons.lang.Validate.notNull(Validate.java:192)

~[commons-lang-2.6.jar!/:2.6]



          at

com.evolveum.midpoint.repo.sql.data.common.any.RAnyConverter.convert

To

RValue(RAnyConverter.java:168) ~[repo-sql-impl-3.8.jar!/:na]



          at

com.evolveum.midpoint.repo.sql.data.common.RObject.copyFromJAXB(RObj

ec

t.java:752) ~[repo-sql-impl-3.8.jar!/:na]



          ... 13 common frames omitted



Does anyone know what I’m doing wrong?



Regards,



Chris.







_______________________________________________

midPoint mailing list

midPoint at lists.evolveum.com<mailto:midPoint at lists.evolveum.com>

http://lists.evolveum.com/mailman/listinfo/midpoint





--

Wojciech Staszewski

Administrator Systemów Sieciowych

tel. kom: 663 680 236

www.diagnostyka.pl<http://www.diagnostyka.pl>

Diagnostyka Sp. z o. o.

ul. Prof. M. Życzkowskiego 16, 31-864 Kraków Numer KRS: 0000381559

(Sąd Rejonowy dla Krakowa-Śródmieścia w Krakowie, XI Wydział

Gospodarczy KRS)

NIP: 675-12-65-009; REGON: 356366975

Kapitał zakładowy: 33 756 500 zł.



Pomyśl o środowisku zanim wydrukujesz ten e-mail.

_______________________________________________

midPoint mailing list

midPoint at lists.evolveum.com<mailto:midPoint at lists.evolveum.com>

http://lists.evolveum.com/mailman/listinfo/midpoint

_______________________________________________

midPoint mailing list

midPoint at lists.evolveum.com<mailto:midPoint at lists.evolveum.com>

http://lists.evolveum.com/mailman/listinfo/midpoint



_______________________________________________

midPoint mailing list

midPoint at lists.evolveum.com<mailto:midPoint at lists.evolveum.com>

http://lists.evolveum.com/mailman/listinfo/midpoint



_______________________________________________

midPoint mailing list

midPoint at lists.evolveum.com<mailto:midPoint at lists.evolveum.com>

http://lists.evolveum.com/mailman/listinfo/midpoint

_______________________________________________

midPoint mailing list

midPoint at lists.evolveum.com<mailto:midPoint at lists.evolveum.com>

http://lists.evolveum.com/mailman/listinfo/midpoint







_______________________________________________

midPoint mailing list

midPoint at lists.evolveum.com<mailto:midPoint at lists.evolveum.com>

http://lists.evolveum.com/mailman/listinfo/midpoint






_______________________________________________

midPoint mailing list

midPoint at lists.evolveum.com<mailto:midPoint at lists.evolveum.com>

http://lists.evolveum.com/mailman/listinfo/midpoint





_______________________________________________

midPoint mailing list

midPoint at lists.evolveum.com<mailto:midPoint at lists.evolveum.com>

http://lists.evolveum.com/mailman/listinfo/midpoint

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20180731/62db0bcb/attachment.htm>


More information about the midPoint mailing list