Question

Releasing file handles after run ?

  • 28 September 2021
  • 9 replies
  • 0 views

Userlevel 1
Badge +22

Hi list.

We have a job on an FME server job, that reads from a ZIP file. It's initiated by an outside broker, and the broker continues to perform other tasks after the FME server job has finished.

We've encountered an error, where the broker tries to delete the input ZIP file 12 seconds after the FME server job ends successfully. The error message says something about other processes using the file.

It doesn't happen consistently, but happens fairly often, so is there a problem with releasing file handles from ended FME server jobs ?

Cheers.


9 replies

Userlevel 4
Badge +25

If the workspace runs properly it should release the file handles when it's done. However, what you may be seeing is FME Server passing on features to a ZIP process, which takes some time to create the zip file. Once the last feature has been sent FME Server considers itself "done", and report it as such, while that zip process continues to run (and keep the file open) for a bit longer.

Userlevel 1
Badge +22

If the workspace runs properly it should release the file handles when it's done. However, what you may be seeing is FME Server passing on features to a ZIP process, which takes some time to create the zip file. Once the last feature has been sent FME Server considers itself "done", and report it as such, while that zip process continues to run (and keep the file open) for a bit longer.

Hi Red.

That might be true, if I were writing to a ZIP. However, in this case I'm just reading from a ZIP, and closing it after reading shouldn't take that long?

Cheers

Userlevel 4
Badge +25

Hi Red.

That might be true, if I were writing to a ZIP. However, in this case I'm just reading from a ZIP, and closing it after reading shouldn't take that long?

Cheers

Oops, I misread your post, sorry about that. Yes, that is pretty weird that it keeps the zip file open that long. Maybe somebody from Safe can shed some light on this @mark2atsafe​ ?

Userlevel 2
Badge +9

Hi @Lars I Nielsen​,

Sorry to hear you're running into this issue.

Are you able to provide a copy of your log file or an example here of the exact error you're getting? This will help me to pinpoint the issue and see if we've had similar problems in the past.

Userlevel 1
Badge +22

Hi @Lars I Nielsen​,

Sorry to hear you're running into this issue.

Are you able to provide a copy of your log file or an example here of the exact error you're getting? This will help me to pinpoint the issue and see if we've had similar problems in the past.

This is what I have: a screen dump of the log of our broker, and the corresponding FME log fil.

Userlevel 1
Badge +22

This is what I have: a screen dump of the log of our broker, and the corresponding FME log fil.

The broker log

Userlevel 1
Badge +22

This is what I have: a screen dump of the log of our broker, and the corresponding FME log fil.

The FME log

Userlevel 1
Badge +22

Hi @Lars I Nielsen​,

Sorry to hear you're running into this issue.

Are you able to provide a copy of your log file or an example here of the exact error you're getting? This will help me to pinpoint the issue and see if we've had similar problems in the past.

Any progress ?

Badge +2

Any progress ?

Hi @Lars I Nielsen​ ,

 

Apologies for the delay, Dan has been out of the office the last few days. I've taken a look at the information provided and also done some testing on my end and I am unable to reproduce the behaviour you are observing intermittently.

 

When the source dataset points to a zip file, FME firsts extract the zip contents before it can read in the data, you can see this in your log file at line 90, where it is opening the MapInfo file from C:\\Users\\SVC-FME\\AppData\\Local\\Temp\\fmeengines\\FME-SERVER002_Engine2\\TempFS_1632680563821_11200 instead of the original path //collect_test_db/DATA...etc

 

Therefore from the timestamp where the data is being read in, I would expect the zip extraction process to be complete and the file to be released. In my testing I placed a System Caller after the Reader with the command 'del $(FILENAME)' and through repeated runs was always able to delete the zip.

 

What method is used by the broker to delete the file?

Are there any other jobs/processes running at the same time that would make use of that same file?

Reply