Question

Need help with inconsistent results from a workspace


My team is relatively new to using FME Desktop, so our technical knowledge is limited for troubleshooting issues. We've created a basic workspace with a Reader Feature type, to read Microstation V7 files (IGDS), connected to a Writer feature type to output shapefiles. We've also create a "Master" workspace with a series of Workspace Runner Transformers to pass a few thousand files from multiple directories automatically through the original workspace. The problem that we're having is inconsistent results and outputs from the core workspace.

The expectation of the core workspace is to read each layer of the Microstation file, and output it as a shape file. The output of each layer should also be fanned out based on the "IGDS_Basename" and create separate shape files for each layer based on the different geometries within the layer. An example would be layer 19, which contains text, line, point, and polygon as different geometries. In an ideal run, the output will include 4 shape files for layer 19, labeled as 19_text, 19_line, 19_point, 19_polygon.

The problem that we're experiencing is an inconsistency on running the workspace on a single file, vs a batch. When we run the workspace for a single file, the output appears to be complete, with all of the layers extracted and their full geometries fanned out. When we run the same workspace as a batch, we find that some of the layers are not being fully extracted. In the example above, only 19_text and 19_point are output. The 19_line and 19_polygon shape files are missing.

I'm reaching out to the community to find out if this is a known issue, or if there is a workaround for this type of problem.


5 replies

Userlevel 5
Badge +25

It could be a case of the 19_line and 19_polygon feature types not being represented in the workspace because those were not present in the specific dgn file you used to generate this workspace.

 

Any chance you can share it with us?

It could be a case of the 19_line and 19_polygon feature types not being represented in the workspace because those were not present in the specific dgn file you used to generate this workspace.

 

Any chance you can share it with us?

Thanks for the response. I've attached a screenshot of the workbook. The most perplexing part of this is, we're getting inconsistent results from the exact same file. In the example I gave, we ran the file through the workbook alone and got one output. When we ran it as a batch with multiple other files, the output for that specific file had different results.

Thanks for the response. I've attached a screenshot of the workbook. The most perplexing part of this is, we're getting inconsistent results from the exact same file. In the example I gave, we ran the file through the workbook alone and got one output. When we ran it as a batch with multiple other files, the output for that specific file had different results.

Adding a few of the additional parameter information

Userlevel 1
Badge +10

Hi @joek86​. I'm curious to know if your dynamic writer feature type is set up to handle different incoming geometries. Can you share what the Geometry parameter is set to in the Writer Feature Type?

image

Hi @joek86​. I'm curious to know if your dynamic writer feature type is set up to handle different incoming geometries. Can you share what the Geometry parameter is set to in the Writer Feature Type?

image

Hi @nampreetatsafe​,

 

Our geometry is set for "From Schema Definition". We've been experimenting a little bit more with the workspace, and we think we determined a general cause for the inconsistencies. We don't know the reason behind it, but we have a cause.

 

In my original post, I described the two processes as a single run, verse a "batch" run. I was incorrect in my definition of a batch run. In our original workspace, we established the source design files as a wildcard followed by the required file extension; ie "*.dgn" (our files are .DGN type, but the extensions are unique for different map types). Our goal was to have the reader go into a specific folder, and read all files with only specific file extensions. Running the workspace this way did work, but we found inconsistencies against running a single file; ie "filename.dgn".

 

FME Workbook Parameters 3 

While experimenting, we used the "Batch Deploy" wizard and selected all of the files individually. As a result, instead of the workspace opening the folder and selecting all files of a specific file type, the batch wizard is now selecting each individual file and processing it. We found that using the "Batch Deploy" wizard, we were able to get consistent results which also matched the results of an individual file run. We also found that performance improved dramatically with the tool. Using the other methodology, the tool would take multiple hours to run. Using the "Batch Deploy" wizard, the process took no more than 15 minutes.

 

We're still analyzing our findings and refining our final outputs, but those are the results we found. We are now trying to figure out why our original methodology output inconsistent results.

Reply