Skip to main content

I would like some inputs if possible. I want to write several thousand of rows to a database (each of these rows represent a pdf file). I have created an attribute called 'status' that should have the value 1 or 0 based on whether the particular row has been written into the database. This should be a precaution if FME breaks down. Then it is possible to see which rows have been written into the database and which haven't. 

 

I don't expect a complete answer at all but perhaps just some ideas on how to achieve this. Thanks in advance.

On what table does this 'status' attribute sit? if it is on this pdf table, then there's no way you would ever end up with a status = 0. If FME writes the row it does, if it doesn't, it doesn't.

If you're worried about failures, validate the data before you write to be sure there's nothing wrong with it.

If the database itself is unreliable, then you could structure your process to read both your input files and the current output table, compare and find what needs to be written, and write to the database what needs to go. At the end of the process filecopy the pdfs to somewhere else to mark them as processed. So if the database falls over, your input files are still there and you can safely retry, and the workspace also tells you what the difference is.


On what table does this 'status' attribute sit? if it is on this pdf table, then there's no way you would ever end up with a status = 0. If FME writes the row it does, if it doesn't, it doesn't.

If you're worried about failures, validate the data before you write to be sure there's nothing wrong with it.

If the database itself is unreliable, then you could structure your process to read both your input files and the current output table, compare and find what needs to be written, and write to the database what needs to go. At the end of the process filecopy the pdfs to somewhere else to mark them as processed. So if the database falls over, your input files are still there and you can safely retry, and the workspace also tells you what the difference is.

If I have to write 10000 files to the database, I want to be able to pause at some point. I just wanted some method to secure that all the files would be written. What you write makes absolute sense. This seems like the way to go:

If the database itself is unreliable, then you could structure your process to read both your input files and the current output table, compare and find what needs to be written, and write to the database what needs to go


Reply