Skip to main content
Question

How to dynamically append attribute value to rows when writing to database


I would like some inputs if possible. I want to write several thousand of rows to a database (each of these rows represent a pdf file). I have created an attribute called 'status' that should have the value 1 or 0 based on whether the particular row has been written into the database. This should be a precaution if FME breaks down. Then it is possible to see which rows have been written into the database and which haven't. 

 

I don't expect a complete answer at all but perhaps just some ideas on how to achieve this. Thanks in advance.

2 replies

ctredinnick
Supporter
Forum|alt.badge.img+18
  • Supporter
  • July 13, 2023

On what table does this 'status' attribute sit? if it is on this pdf table, then there's no way you would ever end up with a status = 0. If FME writes the row it does, if it doesn't, it doesn't.

If you're worried about failures, validate the data before you write to be sure there's nothing wrong with it.

If the database itself is unreliable, then you could structure your process to read both your input files and the current output table, compare and find what needs to be written, and write to the database what needs to go. At the end of the process filecopy the pdfs to somewhere else to mark them as processed. So if the database falls over, your input files are still there and you can safely retry, and the workspace also tells you what the difference is.


  • Author
  • July 13, 2023
ctredinnick wrote:

On what table does this 'status' attribute sit? if it is on this pdf table, then there's no way you would ever end up with a status = 0. If FME writes the row it does, if it doesn't, it doesn't.

If you're worried about failures, validate the data before you write to be sure there's nothing wrong with it.

If the database itself is unreliable, then you could structure your process to read both your input files and the current output table, compare and find what needs to be written, and write to the database what needs to go. At the end of the process filecopy the pdfs to somewhere else to mark them as processed. So if the database falls over, your input files are still there and you can safely retry, and the workspace also tells you what the difference is.

If I have to write 10000 files to the database, I want to be able to pause at some point. I just wanted some method to secure that all the files would be written. What you write makes absolute sense. This seems like the way to go:

If the database itself is unreliable, then you could structure your process to read both your input files and the current output table, compare and find what needs to be written, and write to the database what needs to go


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings