Skip to main content
  • EN

I have a bunch of rasterfiles (Geotiff's) which shows flooding-depths under different scenarios.

All in all I have 35 rasterfiles with a resolution of 0.4x0.4 which equals ~12 GB each. I also have the same data split in smaller files (approx. 700 files).

 

What I'd like is some advise on how to optimize my workflow. eg. how do i choose a reasonable tile size? Does the float to intergerconversion pay off? Is it faster to call an external rastertopolygon coercer?

Right now I: Tile input -> remove redundant tiles -> convert to integer representation -> reclassify tiles -> rasterToPolygon -> simplify -> clip -> collect.

Unfortunately the client is reluctant towards resampling the input.

 

I added a workspace and some exampledata at https://drive.google.com/drive/folders/1AHXnj8nDOZwjqDsam9KCBgMYoKQzAc6i?usp=sharing

Let me know if you need more input.

Best, Mathias.

 

It mostly comes down to processing power. RasterToPolygonCoercer is the easiest way to convert to a polygon. You'll want to load to a File Geodatabase so you don't spit out 2GB shapefile limits and crash the process. From there you can do all your calculations on polygons vs. the imagery.

 

For processing power, you might give Safe a call to discuss running an on-demand FME cloud/server or getting an on-demand AWS EC2 with a temp license. Cloud is $6 an hour and/or you can get a windows server on AWS with 16 cores and 64GB of ram for ~$4 an hour but it's not configured. (takes about 20 minutes to setup and get going)

 

If you go the route of a AWS you can run 5-6 of the images at a time on a 16 core machine using a parent workbench with a list of file names and a WorkspaceRunner with a variable sent to a child workspace. Think you can do the same on cloud, but not sure.

It mostly comes down to processing power. RasterToPolygonCoercer is the easiest way to convert to a polygon. You'll want to load to a File Geodatabase so you don't spit out 2GB shapefile limits and crash the process. From there you can do all your calculations on polygons vs. the imagery.

 

For processing power, you might give Safe a call to discuss running an on-demand FME cloud/server or getting an on-demand AWS EC2 with a temp license. Cloud is $6 an hour and/or you can get a windows server on AWS with 16 cores and 64GB of ram for ~$4 an hour but it's not configured. (takes about 20 minutes to setup and get going)

 

If you go the route of a AWS you can run 5-6 of the images at a time on a 16 core machine using a parent workbench with a list of file names and a WorkspaceRunner with a variable sent to a child workspace. Think you can do the same on cloud, but not sure.

Thanks a lot. It set up a WorkSpaceRunner to run in parallel, then I'll see if I need to call Safe.

Reply


Community Stats

30,185
Posts
114,569
Replies
37,847
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.