Skip to main content
  • EN

I have a workspace that reads an excel spreadsheet that contains 20,000+ rows with file names and paths to DWG files which works 99.9% of the time.  The simplified process is as follows:

  1. Filter the list of files from the spreadsheet to remove certain filename patterns
  2. Read the dwg model space, but only keep lines that are on layers that starts with BND
  3. Test to see that they fit in a polygon using spatialfilter
  4. clean up attributers and write the results to shp file

For some reason when it hits a certain file, the process stops when it hits an error (even it is set to continue on rejected feature and ignore failed readers is set to Yes.  

I have attached a zip with the log file as well as a FFS file it create which contains a single line that doesn't appear to be anything unusual or unexpected.  

My work-around for now is to skip the offending file by using the “features to read” settings on the reader.  Given that there are 20,000+ files, I tend to run 5000 at a time.

The FFS file is from a feature which FME Failed to reproject and got rejected. I don’t think that this is the feature which is causing the problem. 

The thing which jumps out at me is the line:

|ERROR |NOT_THESE_FILES (TestFactory): Internal error: Failed to process bulk features.

Are you able to share the cad file at all?


 

It would be good to start to investigate which transformer gives the fatal error. Without opening your zip (for security reasons) I have little clue what is going wrong.

You mention that the workbench crashes when it hits a certain file. One way to prevent everything else to stop is to build a workspacerunner and run the process that crashes in a separate workbench triggering it for every file. This way the process only needs to be run for that one file that fails.

At one time we had a workbench running for several hours and crashing without a clue on a rasterdemgenerator. Then we cut the complete area in 25 smaller area’s and run 25 workbenches. This way we could figure out what area’s where the problem.

Skip to the end unless you want the story of how I figured it out….

 

​@virtualcitymatt Ha - you caught me being lazy.  |NOT_THESE_FILES (TestFactory) results from my test filter which I had the logic reversed in my head.  The files that I directed to port “Not these file” were in fact the ones I wanted to keep.  I never did change the name of the output port.

I tend to agree that it may not be the specific cad file that the FFS pointed to.  I just did a simple test where I read that DWG and then wrote it to SHP format and it worked fine.

​@jkr_wrk makes a valid point, and in a way I am breaking down the process into several smaller runs.  Rather than having multiple workbenches running in parallel, I am simply re-running the same one over again after changing the input and outputs.  

I am no expert on FME so don’t know how to figure out which transformer is causing the problem.  I have run it with caching on and manually ran one transformer at a time, and it crashes when it writes the final output file.  When I look at the section of the log file pasted below, I see a reference to a reprojection error.  

The dwg files I am loading are not true autocad drawings but created with a product called MicroSurvey which is built on the Intellicad engine.  When I load the project in MicroSurvey, it shows that the coordinate system is set to CSRS.UTM17 but if I load the same drawing in Autocad Map3d, it comes up as being in UTM83-17. 

Perhaps that is a clue, but then again I can read this DWG and save it to SHP format in a simple test.  The file also converts if I restrict my input to that single file.

If I include two files from the input (set “Features to Read: Start Feature” to 5068 and “Max Feature to Read” to 2 (instead of 1), I get the error.  So I think it has to do with the two different drawings having different coordinate system definitions embedded within the DWG file.  

I am using the FeatureReader to read the DWG files and have the coordinate system set to Unknown.

The FeatureWriter is set to use “same as source”.

Actually, WAS.  I just tried the two file test again and set the coordinate system explicitly in the Reader to CSRS.UTM-17N and it ran without crashing.😀 So it must have been a problem with the source coordinate system of that file not being compatible with the previous files.

 

2025-07-30 12:38:26|   3.3|  0.0|INFORM|CS-MAP Reprojector: Transformation will be automatically selected for 'UTM83-17' -> 'CSRS.UTM-17N', and is not guaranteed to remain the same in future releases of FME
2025-07-30 12:38:26|   3.4|  0.1|WARN  |CS-MAP: CSRS :: Referenced grid file does not exist or cannot be opened for reading.
2025-07-30 12:38:26|   3.4|  0.0|STATS |Storing feature(s) to FME feature store file `C:\DataSSD1\FME\MScadFileLog_log.ffs'
2025-07-30 12:38:26|   3.4|  0.0|ERROR |+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

 

Community Stats

31,790
Posts
120,751
Replies
39,405
Members

Latest FME

Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.