Question

Save each record to file

  • 20 October 2014
  • 3 replies
  • 9 views
J
Rank
Badge +1
  • jimo Participant
  • Participant
  • 126 replies
Do the FME text / csv writers have the ability to save each record that gets sent to them? I am running a process that will retrieve about 200,000 records. From my test results it appears that FME only writes to the file when the process completes. But since this process will take a few days (I have a decelerator in the script) I don't want the script to choke part-way through. It would be better to save each record that arrives at the writer.

 

 

Thanks

3 replies

david_r
Rank
Userlevel 4
Hi,

 

 

a couple of things come into play here:
  1. By default, FME writes out features as soon as they hit the writer. But there are some transformers that cannot let features pass until it has processed all of them, these are called blocking transformers. Examples include the Sorter. If you have blocking transformers in your workspace, then the features won't hit the writer until the workspace terminates.
  2. Windows will not always show the true file size until the file has been closed. So if you have a running translation that takes a while, the Windows Explorer might report the output file size as 0 bytes until the workspace terminates and the file is closed.
David
J
Rank
Badge +1
What you say in # 2 makes sense, David, and I suspected this was the case. However, I'm wondering what happens if FME crashes before  the workspace terminates normally. Is there some buffer process at work here?

 

 

This can happen when you add a new record to an Access table and then try and view it in another program (like a Python script opening the table); the new record is cached and does not actually get written until you close the database. If it is the same with FME then my script might process for two days, then crash, and nothing gets actually written to the file. I would feel better if I knew that FME (outside of #1 above) ALWAYS writes each feature to the file when they hit the writer.

 

 

Thanks
pratap
Rank
Badge +2
Hi,

 

 

I far as I know FME will not wait untill 200,000 records gets processed. It will write when 2000 records hits the writer (until you have some blocking transformers like feature holder or any such in between the process)...

 

 

Pratap

Reply


Community Stats

28,723
Posts
109,511
Replies
36,559
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.