• Home
  • Forums
  • FME Form
  • Transformers
  • I am encountering an error where my clipped raster data is not being written to the designated output folder . Please see details and images of my workspace
Question

I am encountering an error where my clipped raster data is not being written to the designated output folder . Please see details and images of my workspace

  • 30 September 2021
  • 6 replies
  • 29 views
M

I am encountering an error where my clipped raster data is not being written to the designated output folder . I am using FME Workbench 2021.1.

 

I was following the Raster Clipping workflow here. However, I must be making a mistake when adding the writer and working with the settings. Perhaps I was a bit confused as to how to set the writer and parameters. There error I received was something like FME file not accessible. Unfortunately, my FME crashed and closed before I could copy the error I received. I'm hoping someone can guide me through adding a writer.

 

I am trying to clip ~3000 geotifs to a shapefile file of Ottawa, On. and fan out by Ward name.

 

Thank you

 

 

workspace

6 replies

virtualcitymatt
Rank
Userlevel 4
Badge +26

Ahh, when working with raster data you should switch off feature caching. When using feature caching (which is on by default) you will use waaaay more resources on your computer.

 

But clipping 3000 geotiffs is also a big job anyway. You should consider processing these data in batch mode if you still get crashes. If you use a workspace runner you will be also to also process in parallel. Here's an example on how you can use it: https://community.safe.com/s/article/batch-processing-using-the-workspacerunner-1

 

M

Hi and thanks for the tip to turn off feature caching. I will definitely do that! The crash was a one off thing. It hasn't happened before when I've run processes on large datasets. I only mentioned it because I wasn't able to share the error I was receiving during the writing phase. I also made an FME_TEMP environment on my external drive (14TB) where all cache files go to.

 

More importantly, I think I'm not getting my writer settings right. Everything runs smoothly (read in Geotiff dataset | RasterBandAdder Transformer | Read in Wards_project | Clipper Transformer...) but the clipped rasters are not showing up in the output folder I designated. I want the rasters clipped to the Wards_polygon using the ward number (there are 23 wards) attribute as fanout (I think I'm using the correct terminology here).

 

  1. When I add the writer (see image Add_Writer) do I put the folder I want for the output in Dataset?
  2. Select Feature Type - which one do I choose - GEOTIFF [Ottawa2..] or Wards_project SHAPE? See image Select_ Feature_Type. I thought if I chose Wards then I have the option to use the wards_en attribute as fanout?
  3. In the Feature Type parameters I chose WARD_EN for Raster File Name and I can see the Writer output folder is correct...I think. See Feature Type image.

Just not sure why the clipped rasters are not going to the output folder FMETrialON2dataset [GEOTIFF]

 

Select_Feature_TypeAdd_writer 

Feature_Type 

Any guidance is most appreciated. Thanks!

 

 

 

 

M

Edit: I want the fanout by WARD_EN (my mistake :) )

Here is an image of my workspace again with the general parameters box showing:

workspace

caracadrian
Rank
Badge +20

You should check the Merge Attributes box in Clipper.

I think you will need to use RasterMosaicker with GroupBy set to WARD_EN to get a single file for each Ward. I b4elieve you have a geotiff tiles dataset at hand.

To Fanout to different Folders you have to put the attribute in the Dataset parameter of the writer, if you want to Fanout to different files your settings are ok.

Be careful of the Outside output of the clipper. You will not be able to use WARD as Fanout as the features will not have it. Use a different writer or drop them out.

M

You should check the Merge Attributes box in Clipper.

I think you will need to use RasterMosaicker with GroupBy set to WARD_EN to get a single file for each Ward. I b4elieve you have a geotiff tiles dataset at hand.

To Fanout to different Folders you have to put the attribute in the Dataset parameter of the writer, if you want to Fanout to different files your settings are ok.

Be careful of the Outside output of the clipper. You will not be able to use WARD as Fanout as the features will not have it. Use a different writer or drop them out.

NAILED IT! I bow to thee. I thought I had the correct settings to fanout to folders but I did not. I double checked the dataset parameter and BAM! I have Ward folders and all the tiles that clipped to the different wards in the folders. Thanks again!

M

Ahh, when working with raster data you should switch off feature caching. When using feature caching (which is on by default) you will use waaaay more resources on your computer.

 

But clipping 3000 geotiffs is also a big job anyway. You should consider processing these data in batch mode if you still get crashes. If you use a workspace runner you will be also to also process in parallel. Here's an example on how you can use it: https://community.safe.com/s/article/batch-processing-using-the-workspacerunner-1

 

A big thank you to you for suggesting I turn off feature caching. This went from a two day job to a few hours. YES!

Reply


Community Stats

28,727
Posts
109,522
Replies
36,566
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.