Skip to main content
  • EN

Hello,

I have a workspace in which I take a lot of data from DB using postGIS reader. Then I try to save it to different files based on fanoutExpression like: output\\@Value(myAttribute).gml

 

 

In DB I have 20 GB of data so at some point I get out of memory exception.

 

My idea was to first use featureReader to read only myAttribute, then use duplicateFilter and then use FeatureReader with where statement: myAttribute = @Value(myAttribute)

 

 

but result is saved to file only when all data is read so I get the same: "out of memory exception."

 

 

Is there any way to:

First read all unique myAttribute from given table and then close the connection (because connection timeout may be also a problem)

 

 

next for each of those unique myAttribute values read data from given table and save to separate file one by one? This way I think that I won't have out of memory or connection timeout exceptions.

 

 

Thanks for any hints on that!

 

 

Are you certain that the problem lies with the writer, and not with the DuplicateFilter?

If the incoming features are already ordered by the attributes used by the DuplicateFilter, consider setting "Input is ordered" to Yes and the memory consumption should be much lower:

If the features aren't already ordered, consider using using a SQLCreator rather than the FeatureReader and specify an ORDER BY clause.

I think that my question is related to:

 

 

https://knowledge.safe.com/idea/25368/add-option-data-is-ordered-by-fanout-to-writers.html

 

 

I think that my question is related to:

 

 

https://knowledge.safe.com/idea/25368/add-option-data-is-ordered-by-fanout-to-writers.html

 

 

Yes, that could very well be the case.

Are you certain that the problem lies with the writer, and not with the DuplicateFilter?

If the incoming features are already ordered by the attributes used by the DuplicateFilter, consider setting "Input is ordered" to Yes and the memory consumption should be much lower:

If the features aren't already ordered, consider using using a SQLCreator rather than the FeatureReader and specify an ORDER BY clause.

Hi @david_r, it didn't help. I just want to run workspace this way that writer will be executed separately by each value of the myAttribute sql query result. I think that the only way is to execute child workspace :/

 

Hi @david_r, it didn't help. I just want to run workspace this way that writer will be executed separately by each value of the myAttribute sql query result. I think that the only way is to execute child workspace :/

 

I'm guessing it's the fanout that's creating the memory issues here. Consider asking Safe support if they can suggest a workaround. If so, consider sending a copy of your workspace and the log file from the last run.
I'm guessing it's the fanout that's creating the memory issues here. Consider asking Safe support if they can suggest a workaround. If so, consider sending a copy of your workspace and the log file from the last run.
Event when I'm not using fanout the probem exists.

 

 

Reply


Community Stats

30,159
Posts
114,480
Replies
37,807
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.