Question

Workspace inconsistently freezing on featurereader

  • 17 October 2023
  • 2 replies
  • 16 views
M

Hello,

 

I have a workspace that is attempting to look for substantial changes to the schema/attributes of client data by comparing a past dataset to a new dataset. It does its job well, except for when it freezes. Irregularly, when processing one of its featurereader transformers, FME freezes completely. When it does freeze, the FME process is hung and will not stop/free the license until the machine is rebooted. 

 

I am running this workspace via the CLI, but the freezes have happened running from the GUI as well. When it freezes, the log file appears to be attempting to either find a plugin to use for the file, or merging the schemas (based on the last thing written/what would be written next). The freezes sometimes happen on the first featurereader, sometimes on the second. I have grabbed the last lines of the log file below from a recent freeze. This workspace normally takes ~90 seconds to complete, and is still running 16 hours later. I've waited over 48 hours before with no change.

 

I have tried:

Recreating the workspace from scratch

Using data from multiple other clients

Running on a different Windows machine

Running on a Linux machine

Running with non-zipped .gdb files

Adding in a FileGDB reader for a dummy file to load the FileGDB.dll before reaching the featurereaders

 

I'm out of ideas of what to try to get around this freezing. I'd welcome any troubleshooting suggestions or wisdom. Thanks in advance!

-Michael

 

FME Form Version: FME(R) 2023.0.0.0 (20230424 - Build 23283 - WIN64) 

2023-10-16 17:38:01|  18.8| 17.1|INFORM|Creating reader for format: Esri Geodatabase (File Geodb Open API)
2023-10-16 17:38:01|  18.8|  0.0|INFORM|Trying to find a DYNAMIC plugin for reader named `FILEGDB'
2023-10-16 17:38:01|  18.8|  0.0|INFORM|Performing query against FILEGDB dataset <past_client_data.zip>
2023-10-16 17:38:02|  18.9|  0.1|INFORM|Creating reader for format: Esri Geodatabase (File Geodb Open API)
2023-10-16 17:38:02|  18.9|  0.0|INFORM|Trying to find a DYNAMIC plugin for reader named `FILEGDB'
2023-10-16 17:38:05|  20.6|  1.7|INFORM|Merged 47 schema features read from 1 datasets into 47 resulting feature types
2023-10-16 17:38:34|  37.7| 17.1|INFORM|Creating reader for format: Esri Geodatabase (File Geodb Open API)
2023-10-16 17:38:34|  37.7|  0.0|INFORM|Trying to find a DYNAMIC plugin for reader named `FILEGDB'
2023-10-16 17:38:34|  37.7|  0.0|INFORM|Performing query against FILEGDB dataset <current_client_data.zip>
2023-10-16 17:38:34|  37.8|  0.1|INFORM|Creating reader for format: Esri Geodatabase (File Geodb Open API)
2023-10-16 17:38:34|  37.8|  0.0|INFORM|Trying to find a DYNAMIC plugin for reader named `FILEGDB'

 

2 replies

virtualcitymatt
Rank
Userlevel 5
Badge +32

You can try the windows process monitor (procmon.exe) - this can be a helpful tool for figuring out what a program is doing/what files it's trying to access. This might give you some clues.

Have you contacted Safe Support yet - if you're able to reproduce it (even intermittently) with the same input datasets then hopefully they are able to help.

You say you tested on Linux - did you have issues there too?

In your FeatureReaders can you try changing the reader mode so that it reads data features only?

How often is intermittently? Are you able to run it a few times in a row to randomly force the freeze?

You say that the license gets stuck? Even if you kill the process in the task manager?

Ideally you want to be in a position where you're able to reliably reproduce the freeze so you can change things to see if anything helps.

 

M

You can try the windows process monitor (procmon.exe) - this can be a helpful tool for figuring out what a program is doing/what files it's trying to access. This might give you some clues.

Have you contacted Safe Support yet - if you're able to reproduce it (even intermittently) with the same input datasets then hopefully they are able to help.

You say you tested on Linux - did you have issues there too?

In your FeatureReaders can you try changing the reader mode so that it reads data features only?

How often is intermittently? Are you able to run it a few times in a row to randomly force the freeze?

You say that the license gets stuck? Even if you kill the process in the task manager?

Ideally you want to be in a position where you're able to reliably reproduce the freeze so you can change things to see if anything helps.

 

Procmon is a good idea, I'll see if that can clue me in to anything. I have not tried Safe Support yet, I came to the forums first.

 

When I tested on Linux, I had the same issues with freezing. I was able to force the process to end on Linux, unlike Windows, but the freeze happened still.

 

The FeatureReaders are reading Schema features only, as that's the information I'm attempting to compare between the two datasets.

 

The freeze is inconsistent. After my most recent change of loading a dummy file load to prep the FileGDB.dll, I've had a freeze after 206 runs, and a freeze after 78 runs, which is much better than the ~1/20 that it was prior. I'm not sure if this is truly improved from my change, or just a small sample size after the change.

 

The license gets stuck because the process is stuck. I'm unable to terminate the process via task manager or with taskkill. Linux has the ability to kill the process, but it remains hung on Windows until I reboot.

Reply


Community Stats

29,355
Posts
111,610
Replies
37,019
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.