Question

Batch Processing (ascii grid) how to process one file at a time rather all at once

J
Badge

So i have a workspace that

- imports all ascii grid files in a directory

 

- does a raster to point conversion (RasterCellCoercer)

 

- offsets points by half cell size - so point at cell center rather than lower left corner (OffSetter)

 

- adds XYZ attributes (AttributeCreator)

 

- Finally writes to MapInfo Tab files

by default FME seems to import all my ascii files and hold everything in memory before committing the outputs. so my model is a RAM hog.

How can i adapt the model so each ascii tile goes through the process one after another which should need significantly less RAM.

11 replies

B
Badge

Try making a child model. You open this child model for each tile, therefore reading it one by one and not all at the same time.

J
Badge

@bakkermans Thanks for the reply so would this involve using something like the WorkspaceRunner transformer?

B
Badge

@bakkermans Thanks for the reply so would this involve using something like the WorkspaceRunner transformer?

That's right. You make the location of every ascii grid file as an attribute in the parent model. Next, you call the child model (actual processing) with a WorkspaceRunner. For each feature in the parent model (i.e. a location), the child model will run once.

erik_jan
Rank
Userlevel 2
Badge +12

In addition to what @bakkermans is mentioning: Use the "Directory and File Path" Reader to loop through a directory and use the attribute windows_path to fire up the WorkSpaceRunner for each file you have in your directory.

That will allow you to use this workspace each time you have new files in your directory.

J
Badge

@bakkermans so is that any different than running in batch mode (Run > Batch Deploy)

mark2atsafe
Rank
Userlevel 4
Badge +25

@bakkermans so is that any different than running in batch mode (Run > Batch Deploy)

The big difference is that WorkspaceRunner allows several jobs to run simultaneously, whereas batch deploy won't. I used to be a huge fan of Batch Deploy, but I never use it now. WorkspaceRunner is more efficient and more predictable.

jdh
Rank
Badge +22

The big difference is that WorkspaceRunner allows several jobs to run simultaneously, whereas batch deploy won't. I used to be a huge fan of Batch Deploy, but I never use it now. WorkspaceRunner is more efficient and more predictable.

The other big advantage to the workspace runner scenario is you have complete control over your parameters. The batch deploy can get very interesting when your have multiple readers.

J
Badge

In addition to what @bakkermans is mentioning: Use the "Directory and File Path" Reader to loop through a directory and use the attribute windows_path to fire up the WorkSpaceRunner for each file you have in your directory.

That will allow you to use this workspace each time you have new files in your directory.

@erik_jan Thanks for your reply. so i assume you mean setting up a reader (using the "open advanced browser" option selecting a folder + sub folders)

From here how do you expose the windows_path attribute?

erik_jan
Rank
Userlevel 2
Badge +12

@erik_jan Thanks for your reply. so i assume you mean setting up a reader (using the "open advanced browser" option selecting a folder + sub folders)

From here how do you expose the windows_path attribute?

Hi @jamiefinney. If your C;\\temp would contain the ascii files this would be the settings for the reader. The windows_path attribute will be exposed automatically.

erik_jan
Rank
Userlevel 2
Badge +12

@erik_jan Thanks for your reply. so i assume you mean setting up a reader (using the "open advanced browser" option selecting a folder + sub folders)

From here how do you expose the windows_path attribute?

One more screenshot:

J
Badge

One more screenshot:

@erik_jan Awesome thanks just trying that now. Looks like it will process my data much quicker now.

Reply


Community Stats

28,670
Posts
109,369
Replies
36,531
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.