Skip to main content
  • EN

Hi Team,

I have 7 million records coming out from source and i need to write to two databases simultaneously.

why simultaneous:

if i write one writer and the second writer will be waiting first one to complete. And in between memory issue comes and fme hangs or stops abruptly.

so I need to write to two databases at a time. need serial connection to the writers.

not parallel.

can anyone please help me.

If memory is an issue, writing to two databases, could you write to one database? Then use a second workspace to read from the first database and copy to the second?

I know this seems too much work, but sometimes the amount of data limits the options.

Another option, that I have used before, is writing to FFS file as an intermediate step and then write that data to the database(s).

The final option (if you want to do it in one workspace) is using two FeatureWriter transformers. They allow streaming, where actual writers will store data in memory.

@erik_jan

I like the final option,

But if the writers are two different databases, can i send all the attributes after writing to featurewriter1 thru summary port?

@erik_jan

I like the final option,

But if the writers are two different databases, can i send all the attributes after writing to featurewriter1 thru summary port?

That will be an option in FME 2018 (data output after FeatureWriter. The Summary port writes a summary like the stats in the log file. I don't think that is what you want.). But can you send all features to two different FeaureWriter transformers.

 

 

Reply


Community Stats

30,107
Posts
114,275
Replies
37,740
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.