Skip to main content
  • EN
  • Home
  • Forums
  • FME Form
  • Data
  • File Copy Writer Error attempting to copy from when writing to Biztalk monitored fileshare


Hi, I have
this error using FME FileCopyWriter (runs on FME Server) to copy files processed
by FME to a fileshare that is monitored by Biztalk. Biztalk is taking care of
transferring files to a different (and isolated) location.  We are currently testing a set of about 100 files
(140Mb, mostly shapefile, jpg and wld), the final set would take about 4500
files and 40Gb.


I use a
workspace that reads from Excel a list of source and target locations, and a FeatureReader
(PATH Reader) to read all files at the source location. Finally a File Copy
Writer to write each file to the Biztalk target location.


Source
files are ready and static, no read/write actions on them. When running the
workspace with Biztalk services turned off things are all running fine.
However, when Biztalk services are running, the process crashes with the
following error: 

2017-08-08 09:33:07|  0.5|  0.0|INFORM|File Copy Writer: Copying file '2016_01w_O10_1_2.jpg'
2017-08-08 09:33:07|  0.5|  0.0|ERROR |File Copy Writer: Error attempting to copy from 'd:\Data\FME\FME Server\data\AGM\LUFO\2016_01w_O10_1_2.jpg' to '\\<someserver>\fme_data\Uit\AGM\FME01\2016_01w_O10_1_2.jpg'.  Please check for sufficient permissions on the source and destination.  Alternatively, if Overwrite Existing File is set to 'No' please ensure output file does not already exist
2017-08-08 09:33:07|  0.5|  0.0|ERROR |A fatal error has occurred. Check the logfile above for details
2017-08-08 09:33:07|  0.5|  0.0|ERROR |... Last line repeated 2 times ...


The file
generating the error is different each time, so it’s not like a lock on some source
file is blocking the copy process. Some files do get copied, so it’s not like the writing process has no authorization for writing
that location. 


We suspect
that, when running, Biztalk is picking up the file and locking it or moving it
elsewhere even before FME finishes writing. The guys at Biztalk tell me that
Biztalk would never touch a file that is locked for writing. I assume that FME
File Copy Writer locks the file when writing until its done, yet cannot find
any switch to ensure it keeps the file locked until really done (and not
writing blocks to the file and freeing the file after writing each block). 


Anybody
having any idea on how to solve this or any experience transferring data this
way? We are using:


  • FME
    Server 2016.1.0.1 - Build 16494 - win64 
  • FME
    Desktop 2015.1.3.1 (20151117 - Build 15573 - WIN32)

I've seen something similar and the errors were due to the monitoring software picking up the destination files before they'd been fully transferred (large files and slow network between the servers).

We solved this by configuring the monitoring software to exclude files with a certain extension or prefix. After the file copy had terminated, we renamed the destination file to remove the extension/prefix to allow the monitoring software to pick it up.

This works well since renaming a file is instantaneous.

Are you using a writer or a FeatureWriter transformer? Whichever you are using, try the other to see if it works. I'm 99.999% sure it ought not to make a difference, but maybe worth a try (although I'd go with David's excellent suggestion first of all)

Hi @KenAtSafe, @Mark2AtSafe, @david_r, We just succeeded transferring the files to biztalk, using the arrangement that Biztalk only transmits files with names prefixed by “ready_”.

I modified the workspace so that it keeps track of the last processed file (using VariableSetter).
When processing the next file, I read the variable and transfer it to a PythonCaller which does an os.rename ()
The Pythonscript also stores the current file in an environment variable, which is read by a shutdown Python script that renames the last processed file.

The workspace was using a PATH Reader from a FeatureReader transformer and a CopyFile Writer to copy single files to their destination. Originating folders are read from an Excel Reader file.
Hi @KenAtSafe, @Mark2AtSafe, @david_r, We just succeeded transferring the files to biztalk, using the arrangement that Biztalk only transmits files with names prefixed by “ready_”.

I modified the workspace so that it keeps track of the last processed file (using VariableSetter).
When processing the next file, I read the variable and transfer it to a PythonCaller which does an os.rename ()
The Pythonscript also stores the current file in an environment variable, which is read by a shutdown Python script that renames the last processed file.

The workspace was using a PATH Reader from a FeatureReader transformer and a CopyFile Writer to copy single files to their destination. Originating folders are read from an Excel Reader file.
Seing that biztalk already expects a prefix to signal that the file is ready, I'd say it probably isn't the first time their developers experience this type of isse.

 

Good to hear my suggestion could help you along.
Seing that biztalk already expects a prefix to signal that the file is ready, I'd say it probably isn't the first time their developers experience this type of isse.

 

Good to hear my suggestion could help you along.
I did vote you up, not shure it got there?

 

 

Reply


Community Stats

30,096
Posts
114,255
Replies
37,728
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.