Skip to main content
  • EN

Hello people. I am having an issue that i cannot resolve. I am using the FME for 2 months now. Basically i have a workspace that performs pointOnRaster value extraction, does some calculations based on the result and writes to table/excel file/csv or whatever. Because i want to run this process over 10 000 files i created a workspace runner that loops trough a folder of *.nc files and compares them to one single shapefile containing the points. Regardless of the output format or the different approaches i've tried i am not getting the whole batch of results written in the output. For example i ran the workflow over the first 326 files and the WorkspaceRunner gives me 326 successfully processed entries. But the output file contains only 87 entries. If i run that workspace again it gives me between 67 to 90 entries but nothing close to 326. The log file doesn't indicate any kind of a problem. I've tried to do that with vector files as well, but still i am having the same issue with big amount of data.

I've tried with different values on the maximum number of concurrent processes and the number of workspaces. The only solution working is if the number of workspaces per FME process is set to 1 but this takes more than 12 hours to get trough the files which i can accomplish faster with qgis or other tool but the idea here is to automate the process and upload it on the fme server when we have a license for that.

 

Here is a screenshot of the workflow itself:

Because you are running workspaces concurrently, there are conflicts when writing out to the excel file which means you are not seeing all the results. My approach for this sort of scenario is to have each workspace write to a separate csv file and then combine these at the end of the process to a single excel file (using fme of course!)

Hi, I am Also facing similar kind of issue where workspace runner succeeded port shows the correct count but in actual Processed records are less....any  help would be really appreciated

Reply


Community Stats

30,351
Posts
115,341
Replies
38,081
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.