Subject:Best Practice for Pipeline error handling? Author:Bruce Cantor Date:15 Oct 2019 07:06 AM Originally Posted: 15 Oct 2019 07:00 AM
Regarding the Pipeline execution in the Pipeline server:
What is the correct way to handle errors in a flow?
Our flows will integrated with our ERP system using files, we receive a file from the ERP - convert it, and return it in another folder.
My thought was that if things go wrong I would place the original file in an error folder, making it visible to the ERP what files could not be converted with success. Warnings, e.g. if files are not schema valid would be copied to a Warning folder.
However, if a file fails in conversion due to a runtime error, e.g. if a file is not valid XML, the file will just be reported in the Pipeline server as Failed and I cannot place any files in the Error folder.
So, I was thinking that this is not the correct approach and that we should be looking into the Pipeline server log for errors and warnings.
Does anybody have experience with this? And if so, how do you read the log files?
The log server-log is not in structured format, can this be changed?
In the Pipeline Server documentation I found that it should be possible to setup emails for different error levels, would it be possible to create a file instead?
Subject:Best Practice for Pipeline error handling? Author:Ivan Pedruzzi Date:16 Oct 2019 11:58 PM Originally Posted: 16 Oct 2019 10:31 PM
The example attached here shows one possible approach.
The idea is to design a pipeline that checks if a file is valid. You may perform any number of validations in program checkFile.xquery.
Such query behaves as dispatcher, it moves the file to other locations for further processing.
The pipeline also highlights the ability to feed the input URL instead of the input data which is documented in the XML Pipeline Server User’s Guide, chapter “Data Types, File Types”