Skip to main content
  • EN

Hi there,

I have a translation between Microstation DGN files and AutoCAD DWG files where there can be as many as 6000 files at a time, but I'm encountering memory resource issues. I want to avoid splitting this translation into a child process with a WorkspaceRunner or JobSubmitter where it will process 1 file at a time.

  • I'm using FeatureReader-DGN and tried with FeatureWriter-DWG / DWG Writer.
  • FME_TEMP points to a 2 TB drive and the translation accumulated 350 GB of temp files like these before crashing with another TB free:

     

    FME_OBJC_MultiWriter_1518629821107_13172_1

     

    ...

     

    FME_OBJC_MultiWriter_1518629821107_13172_235
  • Virtual memory is set to automatic paging on the same drive
  • I have 16 GB RAM
  • DGN file sizes range from 13 KB to 29 MB
  • FME 2016.1.3.2 #16717 WIN64

FME seems to accumulate all the features from all the files before it writes out a single feature to a physical file. Is there a way to maybe (1) batch up the list files so that n-number-of-files are processed at a time, or (2) get the writers to output a full file before the reader reads the next file?

Attaching a sample workspace.

Thank you.

Hi,

My approach is differnet than yours.

1st workspace: I will start with DGN reader (single merged feature type option) and I will use published parameter to pass the file name. I will make necessary modification to the features as required and write to DWG writer.

2nd workspace: I will start with excel reader to pass the parameters with workspace runner of above workspace.

This will make 1 dgn / execution and can get rid of memory issues even when dgn files contains more number of features

Hope this helps.

Pratap

Hi,

My approach is differnet than yours.

1st workspace: I will start with DGN reader (single merged feature type option) and I will use published parameter to pass the file name. I will make necessary modification to the features as required and write to DWG writer.

2nd workspace: I will start with excel reader to pass the parameters with workspace runner of above workspace.

This will make 1 dgn / execution and can get rid of memory issues even when dgn files contains more number of features

Hope this helps.

Pratap

With your method, have you had job failure 1000+ or 2000+ files into the translation on FME Server? I've had it happen for another translation and it is why I want to avoid WorkspaceRunner or JobSubmitter in this case.

 

With your method, have you had job failure 1000+ or 2000+ files into the translation on FME Server? I've had it happen for another translation and it is why I want to avoid WorkspaceRunner or JobSubmitter in this case.

 

You mean to say that we will get memory issues, if input is 1000+ files...

 

 

As pratap already advised, I think you already mentioned your own solution. Try using a parent/child structure. It will always be a better way to process so many files/data.

You mean to say that we will get memory issues, if input is 1000+ files...

 

 

When there's 2000+ files, I'd get this error:

 

ERROR |Terminator_deliver_asc_xyz: Failed to
open FME feature store file `\\storage\05232112\FME\resources\logs\engine\current\jobs\9000\job_9253_0_log.ffs' for write -- ensure that the folder exists
My understanding is, this job_####_0_log.ffs
file gets overwritten again and again, based on how many times the child
workspace opens.

 

 

Did you have this behaviour?

 

When there's 2000+ files, I'd get this error:

 

ERROR |Terminator_deliver_asc_xyz: Failed to
open FME feature store file `\\storage\05232112\FME\resources\logs\engine\current\jobs\9000\job_9253_0_log.ffs' for write -- ensure that the folder exists
My understanding is, this job_####_0_log.ffs
file gets overwritten again and again, based on how many times the child
workspace opens.

 

 

Did you have this behaviour?

 

I have not tried in same translation i.e. dgn to dwg but I have tried to dgn to oracle db some time ago for 2000+ files. It didnt gave any error.

 

 

As far as I understand, FME wont overwrite the file as many times the child opens until we tell to FME to perform.

 

 

Reply


Community Stats

30,159
Posts
114,480
Replies
37,806
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.