Following this question csv flushing it has been suggested that the FME CSV writer should auto-flush the writes following a manual cessation of the workspace.
Archived
auto-flush CSV files after stopping workspace
This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
+59- Safer
I'm not sure we would auto-flush those formats. Seems to me that in most
cases if I stop a workspace it's because I *don't* want to write any
data. Or at least, if we did add this it would be optional. Maybe we add a "Transaction Interval" parameter to the CSV writer, although it would need to be in Append mode for that to make sense.
Yes I agree - it could be an option to autoflush if workspace is manually stopped.
As identified above, there's at least one situation where I *do* want to the write the data when stopping the workspace (i.e. the TweetStreamer).
I've noticed that when there are more than one CSV writers in a TweetStreamer workspace, and I stop it after half an hour, the second CSV file has been 100% buffered and *nothing* has been written (not even the empty file).
This also has further implications - I have seen colleagues sequentially process substantial datasets and everything on the second CSV file appears to be buffered. Two problems exists here:
1) massive RAM consumption results.
2) when testing a workspace it's common to run for a small amount of time and stop it to check the results. Can't do that if nothing is written and the buffer has not been flushed.