Ahh, when working with raster data you should switch off feature caching. When using feature caching (which is on by default) you will use waaaay more resources on your computer.
But clipping 3000 geotiffs is also a big job anyway. You should consider processing these data in batch mode if you still get crashes. If you use a workspace runner you will be also to also process in parallel. Here's an example on how you can use it: https://community.safe.com/s/article/batch-processing-using-the-workspacerunner-1
Hi and thanks for the tip to turn off feature caching. I will definitely do that! The crash was a one off thing. It hasn't happened before when I've run processes on large datasets. I only mentioned it because I wasn't able to share the error I was receiving during the writing phase. I also made an FME_TEMP environment on my external drive (14TB) where all cache files go to.
More importantly, I think I'm not getting my writer settings right. Everything runs smoothly (read in Geotiff dataset | RasterBandAdder Transformer | Read in Wards_project | Clipper Transformer...) but the clipped rasters are not showing up in the output folder I designated. I want the rasters clipped to the Wards_polygon using the ward number (there are 23 wards) attribute as fanout (I think I'm using the correct terminology here).
- When I add the writer (see image Add_Writer) do I put the folder I want for the output in Dataset?
- Select Feature Type - which one do I choose - GEOTIFF FOttawa2..] or Wards_project SHAPE? See image Select_ Feature_Type. I thought if I chose Wards then I have the option to use the wards_en attribute as fanout?
- In the Feature Type parameters I chose WARD_EN for Raster File Name and I can see the Writer output folder is correct...I think. See Feature Type image.
Just not sure why the clipped rasters are not going to the output folder FMETrialON2dataset tGEOTIFF]
Any guidance is most appreciated. Thanks!
Edit: I want the fanout by WARD_EN (my mistake 
)
Here is an image of my workspace again with the general parameters box showing:
You should check the Merge Attributes box in Clipper.
I think you will need to use RasterMosaicker with GroupBy set to WARD_EN to get a single file for each Ward. I b4elieve you have a geotiff tiles dataset at hand.
To Fanout to different Folders you have to put the attribute in the Dataset parameter of the writer, if you want to Fanout to different files your settings are ok.
Be careful of the Outside output of the clipper. You will not be able to use WARD as Fanout as the features will not have it. Use a different writer or drop them out.
You should check the Merge Attributes box in Clipper.
I think you will need to use RasterMosaicker with GroupBy set to WARD_EN to get a single file for each Ward. I b4elieve you have a geotiff tiles dataset at hand.
To Fanout to different Folders you have to put the attribute in the Dataset parameter of the writer, if you want to Fanout to different files your settings are ok.
Be careful of the Outside output of the clipper. You will not be able to use WARD as Fanout as the features will not have it. Use a different writer or drop them out.
NAILED IT! I bow to thee. I thought I had the correct settings to fanout to folders but I did not. I double checked the dataset parameter and BAM! I have Ward folders and all the tiles that clipped to the different wards in the folders. Thanks again!
Ahh, when working with raster data you should switch off feature caching. When using feature caching (which is on by default) you will use waaaay more resources on your computer.
But clipping 3000 geotiffs is also a big job anyway. You should consider processing these data in batch mode if you still get crashes. If you use a workspace runner you will be also to also process in parallel. Here's an example on how you can use it: https://community.safe.com/s/article/batch-processing-using-the-workspacerunner-1
A big thank you to you for suggesting I turn off feature caching. This went from a two day job to a few hours. YES!
