having issues getting the parallel processing to work properly when using raster mosaic tool. Trying mosaic a large volume of images via this tool and improve efficiency.
+11- Safer
Hi @clindsey,
What results are you getting from the transformer when running the RasterMosaicker with parallel processing so far? I noticed a few resources that talk about using parallel processing that might be useful to take a look at. Check out the Usage Notes section from About Parallel Processing as it explains that for parallel processing to improve performance, it can depend on whether the processing (CPU) time for each group is expected to be significantly more than the overhead of launching a new process and sending features back and forth between processes. A good way to test for that is to experiment with a small subset of your data to see if it will really help on your dataset.
Another resource is this article on Parallel Processing which leads to a few other useful pages like this one on How to Use Parallel Processing in FME. Hope these help a bit!
+3- Contributor
jovitaatsafe wrote:
Hi @clindsey,
What results are you getting from the transformer when running the RasterMosaicker with parallel processing so far? I noticed a few resources that talk about using parallel processing that might be useful to take a look at. Check out the Usage Notes section from About Parallel Processing as it explains that for parallel processing to improve performance, it can depend on whether the processing (CPU) time for each group is expected to be significantly more than the overhead of launching a new process and sending features back and forth between processes. A good way to test for that is to experiment with a small subset of your data to see if it will really help on your dataset.
Another resource is this article on Parallel Processing which leads to a few other useful pages like this one on How to Use Parallel Processing in FME. Hope these help a bit!
good morning thank you for your help. My one major remaining question is i'm not sure that parallel processing is actually running. how do i determine if this is something running. Can i upload my template to get feedback on how its setup to see if it's parameterized properly.
+11- Safer
clindsey wrote:
good morning thank you for your help. My one major remaining question is i'm not sure that parallel processing is actually running. how do i determine if this is something running. Can i upload my template to get feedback on how its setup to see if it's parameterized properly.
You should be able to check it in the logfile of the translation. For example if I don't set a Group By parameter, it will give me a warning that I hadn't set it so parallel processing did not run. In the Workbench Log, information messages show license limit (if applicable), the request, process memory usage for each "worker", and identifying information about each.
There's a bit more information under Workbench Log in the documentation that shows an example. And yes, please feel free to share your template workspace if the logging still isn't clear and I can take a look at it. It might just be a matter of experimenting with the settings and seeing if it is running faster than if you were to run the workspace without parallel processing for your dataset.
+3- Contributor
jovitaatsafe wrote:
You should be able to check it in the logfile of the translation. For example if I don't set a Group By parameter, it will give me a warning that I hadn't set it so parallel processing did not run. In the Workbench Log, information messages show license limit (if applicable), the request, process memory usage for each "worker", and identifying information about each.
There's a bit more information under Workbench Log in the documentation that shows an example. And yes, please feel free to share your template workspace if the logging still isn't clear and I can take a look at it. It might just be a matter of experimenting with the settings and seeing if it is running faster than if you were to run the workspace without parallel processing for your dataset.
Parallel Processing Raster Mosaicker No Data Setter.fmw
@jovitaatsafe i would GREATLY appreciate it. I'm actually seeing that it's taking longer to run in parallel than stand alone. my test is on 1100 GeoTIFFs which is about 25 Gb in size total. My thought process is to mosaic the images into longitudinal "strips" parallel then follow up and mosaic the 28 "strips" into a single image using a single CPU. i thought this might be quicker than rastering 1100 geotiffs into one image on a single CPU. I have 128 Gb Ram on the machine using 16 CPUs which it's only using 8 which i think is a licensing issue.
+11- Safer
clindsey wrote:
Parallel Processing Raster Mosaicker No Data Setter.fmw
@jovitaatsafe i would GREATLY appreciate it. I'm actually seeing that it's taking longer to run in parallel than stand alone. my test is on 1100 GeoTIFFs which is about 25 Gb in size total. My thought process is to mosaic the images into longitudinal "strips" parallel then follow up and mosaic the 28 "strips" into a single image using a single CPU. i thought this might be quicker than rastering 1100 geotiffs into one image on a single CPU. I have 128 Gb Ram on the machine using 16 CPUs which it's only using 8 which i think is a licensing issue.
Hi @clindsey,
Sorry for the delay in response! I took a look at your workspace, and it looks like you need to set the Parallel Processing parameter in the RasterMosaicker to a level of parallelism. Currently you have it set to "No Parallelism" which is the same as saying "No parallel processing".
The level of parallelism controls how many processes can be executed at a single time. Try selecting one of the levels (such as Moderate or Aggressive) to try it out and remember to look for the process in the log file to check if it's working.
Reply
Most helpful members this week
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.