Skip to main content
Question

Batch Processing (ascii grid) how to process one file at a time rather all at once


Forum|alt.badge.img

So i have a workspace that

- imports all ascii grid files in a directory

 

- does a raster to point conversion (RasterCellCoercer)

 

- offsets points by half cell size - so point at cell center rather than lower left corner (OffSetter)

 

- adds XYZ attributes (AttributeCreator)

 

- Finally writes to MapInfo Tab files

by default FME seems to import all my ascii files and hold everything in memory before committing the outputs. so my model is a RAM hog.

How can i adapt the model so each ascii tile goes through the process one after another which should need significantly less RAM.

11 replies

Forum|alt.badge.img

Try making a child model. You open this child model for each tile, therefore reading it one by one and not all at the same time.


Forum|alt.badge.img

@bakkermans Thanks for the reply so would this involve using something like the WorkspaceRunner transformer?


Forum|alt.badge.img
jamiefinney wrote:

@bakkermans Thanks for the reply so would this involve using something like the WorkspaceRunner transformer?

That's right. You make the location of every ascii grid file as an attribute in the parent model. Next, you call the child model (actual processing) with a WorkspaceRunner. For each feature in the parent model (i.e. a location), the child model will run once.


erik_jan
Contributor
Forum|alt.badge.img+17
  • Contributor
  • March 21, 2016

In addition to what @bakkermans is mentioning: Use the "Directory and File Path" Reader to loop through a directory and use the attribute windows_path to fire up the WorkSpaceRunner for each file you have in your directory.

That will allow you to use this workspace each time you have new files in your directory.


Forum|alt.badge.img

@bakkermans so is that any different than running in batch mode (Run > Batch Deploy)


mark2atsafe
Safer
Forum|alt.badge.img+44
  • Safer
  • March 21, 2016
jamiefinney wrote:

@bakkermans so is that any different than running in batch mode (Run > Batch Deploy)

The big difference is that WorkspaceRunner allows several jobs to run simultaneously, whereas batch deploy won't. I used to be a huge fan of Batch Deploy, but I never use it now. WorkspaceRunner is more efficient and more predictable.


jdh
Contributor
Forum|alt.badge.img+28
  • Contributor
  • March 21, 2016
mark2atsafe wrote:

The big difference is that WorkspaceRunner allows several jobs to run simultaneously, whereas batch deploy won't. I used to be a huge fan of Batch Deploy, but I never use it now. WorkspaceRunner is more efficient and more predictable.

The other big advantage to the workspace runner scenario is you have complete control over your parameters. The batch deploy can get very interesting when your have multiple readers.


Forum|alt.badge.img
erik_jan wrote:

In addition to what @bakkermans is mentioning: Use the "Directory and File Path" Reader to loop through a directory and use the attribute windows_path to fire up the WorkSpaceRunner for each file you have in your directory.

That will allow you to use this workspace each time you have new files in your directory.

@erik_jan Thanks for your reply. so i assume you mean setting up a reader (using the "open advanced browser" option selecting a folder + sub folders)

From here how do you expose the windows_path attribute?


erik_jan
Contributor
Forum|alt.badge.img+17
  • Contributor
  • March 22, 2016
jamiefinney wrote:

@erik_jan Thanks for your reply. so i assume you mean setting up a reader (using the "open advanced browser" option selecting a folder + sub folders)

From here how do you expose the windows_path attribute?

Hi @jamiefinney. If your C;\\temp would contain the ascii files this would be the settings for the reader. The windows_path attribute will be exposed automatically.


erik_jan
Contributor
Forum|alt.badge.img+17
  • Contributor
  • March 22, 2016
jamiefinney wrote:

@erik_jan Thanks for your reply. so i assume you mean setting up a reader (using the "open advanced browser" option selecting a folder + sub folders)

From here how do you expose the windows_path attribute?

One more screenshot:


Forum|alt.badge.img
erik_jan wrote:

One more screenshot:

@erik_jan Awesome thanks just trying that now. Looks like it will process my data much quicker now.


Reply


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings