Skip to main content
Question

Batch Processing (ascii grid) how to process one file at a time rather all at once

  • March 21, 2016
  • 11 replies
  • 164 views

Forum|alt.badge.img

So i have a workspace that

- imports all ascii grid files in a directory

 

- does a raster to point conversion (RasterCellCoercer)

 

- offsets points by half cell size - so point at cell center rather than lower left corner (OffSetter)

 

- adds XYZ attributes (AttributeCreator)

 

- Finally writes to MapInfo Tab files

by default FME seems to import all my ascii files and hold everything in memory before committing the outputs. so my model is a RAM hog.

How can i adapt the model so each ascii tile goes through the process one after another which should need significantly less RAM.

This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

11 replies

Forum|alt.badge.img

Try making a child model. You open this child model for each tile, therefore reading it one by one and not all at the same time.


Forum|alt.badge.img
  • Author
  • March 21, 2016

@bakkermans Thanks for the reply so would this involve using something like the WorkspaceRunner transformer?


Forum|alt.badge.img

@bakkermans Thanks for the reply so would this involve using something like the WorkspaceRunner transformer?

That's right. You make the location of every ascii grid file as an attribute in the parent model. Next, you call the child model (actual processing) with a WorkspaceRunner. For each feature in the parent model (i.e. a location), the child model will run once.


erik_jan
Contributor
Forum|alt.badge.img+26
  • Contributor
  • March 21, 2016

In addition to what @bakkermans is mentioning: Use the "Directory and File Path" Reader to loop through a directory and use the attribute windows_path to fire up the WorkSpaceRunner for each file you have in your directory.

That will allow you to use this workspace each time you have new files in your directory.


Forum|alt.badge.img
  • Author
  • March 21, 2016

@bakkermans so is that any different than running in batch mode (Run > Batch Deploy)


mark2atsafe
Safer
Forum|alt.badge.img+59
  • Safer
  • March 21, 2016

@bakkermans so is that any different than running in batch mode (Run > Batch Deploy)

The big difference is that WorkspaceRunner allows several jobs to run simultaneously, whereas batch deploy won't. I used to be a huge fan of Batch Deploy, but I never use it now. WorkspaceRunner is more efficient and more predictable.


jdh
Contributor
Forum|alt.badge.img+40
  • Contributor
  • March 21, 2016

The big difference is that WorkspaceRunner allows several jobs to run simultaneously, whereas batch deploy won't. I used to be a huge fan of Batch Deploy, but I never use it now. WorkspaceRunner is more efficient and more predictable.

The other big advantage to the workspace runner scenario is you have complete control over your parameters. The batch deploy can get very interesting when your have multiple readers.


Forum|alt.badge.img
  • Author
  • March 22, 2016

In addition to what @bakkermans is mentioning: Use the "Directory and File Path" Reader to loop through a directory and use the attribute windows_path to fire up the WorkSpaceRunner for each file you have in your directory.

That will allow you to use this workspace each time you have new files in your directory.

@erik_jan Thanks for your reply. so i assume you mean setting up a reader (using the "open advanced browser" option selecting a folder + sub folders)

From here how do you expose the windows_path attribute?


erik_jan
Contributor
Forum|alt.badge.img+26
  • Contributor
  • March 22, 2016

@erik_jan Thanks for your reply. so i assume you mean setting up a reader (using the "open advanced browser" option selecting a folder + sub folders)

From here how do you expose the windows_path attribute?

Hi @jamiefinney. If your C;\\temp would contain the ascii files this would be the settings for the reader. The windows_path attribute will be exposed automatically.


erik_jan
Contributor
Forum|alt.badge.img+26
  • Contributor
  • March 22, 2016

@erik_jan Thanks for your reply. so i assume you mean setting up a reader (using the "open advanced browser" option selecting a folder + sub folders)

From here how do you expose the windows_path attribute?

One more screenshot:


Forum|alt.badge.img
  • Author
  • March 23, 2016

One more screenshot:

@erik_jan Awesome thanks just trying that now. Looks like it will process my data much quicker now.