Question

Iterative execution of an FME workflow

pcarrerabalsell
Rank
Badge +4

 

I'm new to FME. I have a process that intersects a line buffer from Oracle with a shp stored in local. In the end there is a dissolver to reduce the number of polygons and write it in a shp.

The problem is that from Oracle come 11 million records ... and in the dissolver I get an error.

My question would be if there would be a way to execute the process in the following way: that I read 500,000 records, process them, write them in the shp, then another 500,000, then others and so on until the end. It could be done?

4 replies

redgeographics
Rank
Userlevel 5
Badge +25

Yes, that could be done. You could for example use a SQLCreator to get the data out of Oracle through a SQL query and limit the number of returned records there, but it would be difficult to make sure each set of 500000 records is actually geographically contiguous.

Dissolving is a rather hefty operation I'm afraid, if you can show us your process we might be able to come up with other suggestions to get this done.

One question I have right away: are the line buffers in Oracle, or the lines from which you create buffers in FME? If it's the 2nd then joining the lines (LineCombiner) before buffering might be a way to speed things up.

pcarrerabalsell
Rank
Badge +4

Hello @redgeographics. This is an image of my workspace . As you can see, lines buffers in workspace, not in oracle. The attibute to dissolve is inherited from oracle, and all the lines have same value on this attribute. I want the buffer as clean as possible, without overlaps, but I don't know if this is the best way to do that.

 

Thanks,

 

Pol CB

pcarrerabalsell
Rank
Badge +4

@redgeographics I almost forget it! This process will surely run on a virtual machine ... I do not know if this can affect the performance of the process too much

redgeographics
Rank
Userlevel 5
Badge +25

Hello @redgeographics. This is an image of my workspace . As you can see, lines buffers in workspace, not in oracle. The attibute to dissolve is inherited from oracle, and all the lines have same value on this attribute. I want the buffer as clean as possible, without overlaps, but I don't know if this is the best way to do that.

 

Thanks,

 

Pol CB

I would try withe LineCombiner before the Bufferer, hard to say what the performance will be but there really is only one way to try it. If you can try it with a small subset of data that would be best of course.

A VM should have little effect I'd say, assuming it's properly specced.

Reply


Community Stats

28,731
Posts
109,527
Replies
36,565
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.