[midPoint] Any type of fail safe tricks or settings?

[Jason Everling]

Thanks for the ideas!

I was able to create a couple of new processes in my talend job, so
basically instead of what I was doing, checking connection > database query
> write file , I am now checking connection > database query > write temp
file in temp location > get total enrollment count from external source >
analyze temp file to match enrollment count to row count in temp csv > if
correct > send to folder where midpoint syncs > else > error, send email
alert and do not copy file.

I feel much more confident that it will not send over an empty file now.
Awesome

JASON



JASON

On Thu, Nov 12, 2015 at 12:48 AM, Petr Gašparík <petr at gasparik.cz> wrote:

> Also, you can incorporate some logic like If there is more than 100
> changes, wait for manual confirmation of changes.
>
> P.
>
> Dne st 11. 11. 2015 15:36 uživatel Jason Everling <jeverling at bshp.edu>
> napsal:
>
>> I didn't actually think to run anything on the file itself or check total
>> count before transfer to file on source system. I am pretty sure I could
>> write something up in Talend to check the source and count before
>> processing any further.
>>
>> Thanks!
>> JASON
>>
>> On Wed, Nov 11, 2015 at 5:11 AM, <midpoint at mybtinternet.com> wrote:
>>
>>> Hi Jason,
>>>
>>>   I've had to solve these kinds of problems for various projects in the
>>> past, admittedly
>>>   not using midPoint. There are also other things that could be
>>> happening that cause
>>>   similar issues; e.g. in-flight / broken transfers etc.
>>>
>>>   Solving the problem with midPoint would imply having some sort of
>>> preview or a running
>>>   count of changes vs non-changes, or validation in the connector. Not
>>> aware of anything
>>>   like this ...
>>>
>>>   Having a system with some decent scripting can help. Some strategies I
>>> used in the past:
>>>
>>>    - Check that modification time on target is at least # mins ago;
>>>    e.g. not in-flight transfer
>>>    - Generate a checksum on the source and validate that on the target
>>>    (md5sum, sha1sum, ...)
>>>    - A status line at the end; e.g. end of feed, a count, checksum, etc
>>>
>>>
>>>   You could, if using (UNIX / Linux) do a grep -v on header - check
>>> result for zero size.
>>>   Incoming transmission write to a filename different from what your
>>> resource expects; validation
>>>   copies / renames to resource input name is validation ok.
>>>
>>> Regards,
>>>   Anton
>>>
>>> ----Original message----
>>> From : jeverling at bshp.edu
>>> Date : 05/11/2015 - 18:16 (GMT)
>>> To : midpoint at lists.evolveum.com
>>> Subject : [midPoint] Any type of fail safe tricks or settings?
>>>
>>>
>>> I was thinking about this today..
>>>
>>> What if one day my CSV file sends out a file that is blank with just the
>>> headers, like if something just goes wrong with the connection or base
>>> query. Since my CSV resource is authoritative for who should be enabled or
>>> disabled, it would disable everyone because they are missing from the file.
>>>
>>> I don't think that could happen, I wrote in as many fail safes as I
>>> could into the Talend job that creates the CSV but....
>>>
>>> Is there any type of mechanism that I can set that says if accounts to
>>> be changed is more than {n} then stop or suspend task?
>>>
>>> Just thinking!
>>>
>>> JASON
>>>
>>>
>>>
>>> CONFIDENTIALITY NOTICE:
>>> This e-mail together with any attachments is proprietary and
>>> confidential; intended for only the recipient(s) named above and may
>>> contain information that is privileged. You should not retain, copy or use
>>> this e-mail or any attachments for any purpose, or disclose all or any part
>>> of the contents to any person. Any views or opinions expressed in this
>>> e-mail are those of the author and do not represent those of the Baptist
>>> School of Health Professions. If you have received this e-mail in error, or
>>> are not the named recipient(s), you are hereby notified that any review,
>>> dissemination, distribution or copying of this communication is prohibited
>>> by the sender and to do so might constitute a violation of the Electronic
>>> Communications Privacy Act, 18 U.S.C. section 2510-2521. Please immediately
>>> notify the sender and delete this e-mail and any attachments from your
>>> computer.
>>>
>>>
>>>
>>> _______________________________________________
>>> midPoint mailing list
>>> midPoint at lists.evolveum.com
>>> http://lists.evolveum.com/mailman/listinfo/midpoint
>>>
>>>
>>
>>
>> --
>> JASON
>>
>>
>>
>> CONFIDENTIALITY NOTICE:
>> This e-mail together with any attachments is proprietary and
>> confidential; intended for only the recipient(s) named above and may
>> contain information that is privileged. You should not retain, copy or use
>> this e-mail or any attachments for any purpose, or disclose all or any part
>> of the contents to any person. Any views or opinions expressed in this
>> e-mail are those of the author and do not represent those of the Baptist
>> School of Health Professions. If you have received this e-mail in error, or
>> are not the named recipient(s), you are hereby notified that any review,
>> dissemination, distribution or copying of this communication is prohibited
>> by the sender and to do so might constitute a violation of the Electronic
>> Communications Privacy Act, 18 U.S.C. section 2510-2521. Please immediately
>> notify the sender and delete this e-mail and any attachments from your
>> computer.
>> _______________________________________________
>> midPoint mailing list
>> midPoint at lists.evolveum.com
>> http://lists.evolveum.com/mailman/listinfo/midpoint
>>
> --
> --
> Petr G.
>
> _______________________________________________
> midPoint mailing list
> midPoint at lists.evolveum.com
> http://lists.evolveum.com/mailman/listinfo/midpoint
>
>

-- 


CONFIDENTIALITY NOTICE:
This e-mail together with any attachments is proprietary and confidential; 
intended for only the recipient(s) named above and may contain information 
that is privileged. You should not retain, copy or use this e-mail or any 
attachments for any purpose, or disclose all or any part of the contents to 
any person. Any views or opinions expressed in this e-mail are those of the 
author and do not represent those of the Baptist School of Health 
Professions. If you have received this e-mail in error, or are not the 
named recipient(s), you are hereby notified that any review, dissemination, 
distribution or copying of this communication is prohibited by the sender 
and to do so might constitute a violation of the Electronic Communications 
Privacy Act, 18 U.S.C. section 2510-2521. Please immediately notify the 
sender and delete this e-mail and any attachments from your computer. 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.evolveum.com/pipermail/midpoint/attachments/20151116/9a7d6d84/attachment.htm>