Skip to main content
Question

Force writer or featureWriter to write data to file before end

W
Forum|alt.badge.img

Hello,

I have a workspace in which I take a lot of data from DB using postGIS reader. Then I try to save it to different files based on fanoutExpression like: output\\@Value(myAttribute).gml

 

 

In DB I have 20 GB of data so at some point I get out of memory exception.

 

My idea was to first use featureReader to read only myAttribute, then use duplicateFilter and then use FeatureReader with where statement: myAttribute = @Value(myAttribute)

 

 

but result is saved to file only when all data is read so I get the same: "out of memory exception."

 

 

Is there any way to:

First read all unique myAttribute from given table and then close the connection (because connection timeout may be also a problem)

 

 

next for each of those unique myAttribute values read data from given table and save to separate file one by one? This way I think that I won't have out of memory or connection timeout exceptions.

 

 

Thanks for any hints on that!

 

 

6 replies

david_r
Celebrity
  • david_rCelebrity
  • August 30, 2018

Are you certain that the problem lies with the writer, and not with the DuplicateFilter?

If the incoming features are already ordered by the attributes used by the DuplicateFilter, consider setting "Input is ordered" to Yes and the memory consumption should be much lower:

If the features aren't already ordered, consider using using a SQLCreator rather than the FeatureReader and specify an ORDER BY clause.

W
Forum|alt.badge.img
  • witos
  • Author
  • August 31, 2018
I think that my question is related to:

 

 

https://knowledge.safe.com/idea/25368/add-option-data-is-ordered-by-fanout-to-writers.html

 

 

david_r
Celebrity
  • david_rCelebrity
  • August 31, 2018
witos wrote:
I think that my question is related to:

 

 

https://knowledge.safe.com/idea/25368/add-option-data-is-ordered-by-fanout-to-writers.html

 

 

Yes, that could very well be the case.
W
Forum|alt.badge.img
  • witos
  • Author
  • August 31, 2018
david_r wrote:

Are you certain that the problem lies with the writer, and not with the DuplicateFilter?

If the incoming features are already ordered by the attributes used by the DuplicateFilter, consider setting "Input is ordered" to Yes and the memory consumption should be much lower:

If the features aren't already ordered, consider using using a SQLCreator rather than the FeatureReader and specify an ORDER BY clause.

Hi @david_r, it didn't help. I just want to run workspace this way that writer will be executed separately by each value of the myAttribute sql query result. I think that the only way is to execute child workspace :/

 

david_r
Celebrity
  • david_rCelebrity
  • August 31, 2018
witos wrote:
Hi @david_r, it didn't help. I just want to run workspace this way that writer will be executed separately by each value of the myAttribute sql query result. I think that the only way is to execute child workspace :/

 

I'm guessing it's the fanout that's creating the memory issues here. Consider asking Safe support if they can suggest a workaround. If so, consider sending a copy of your workspace and the log file from the last run.
W
Forum|alt.badge.img
  • witos
  • Author
  • August 31, 2018
david_r wrote:
I'm guessing it's the fanout that's creating the memory issues here. Consider asking Safe support if they can suggest a workaround. If so, consider sending a copy of your workspace and the log file from the last run.
Event when I'm not using fanout the probem exists.

 

 

Reply


Helpful Members This Week

Community Stats

31,525
Posts
119,504
Replies
39,151
Members

Latest FME

Terms & ConditionsCookie settingsAccessibility statement

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.