Skip to main content
  • EN

Hi, 

 

 

Have a few queries that don't appear to be listed on the forums (suprisingly) nor google; if there are any whitepapers or the like out there please provide me with a url.

 

 

I am involved in a large data migraiton (10gb data held in oracle spatial (single table)) that needs to be migrated as both a single & continual (i.e. change only updates).  Minimal transformation required between source & target (oracle to oracle).

 

 

I'm wanting to use FME as opposed to direct sql for a number reasons, including geometry validation as well as there will be further changes in the future that FME will enable me to take advantage of.  My query however is this:

 

1. What is the reccomended max. GML file-size for such transfers & how can one implement this in FME?  I'm assuming on its import, i want to keep GML to a management size for memory conservation?    Assume GML is the best format?

 

2. Can this simlarly be done for zip files (i.e max zip file size)?

 

3.  What if any does server offer as an advantage to just having a workbench ran off a scheduled tasks; does it offer greater flexiblty in terms of scalabilty (engines); i assume my approach to wanting to keep file sizes contained may complicate this approach?.

 

 

Appreciate any advice as just getting to grips with FME however expirenced in no-GI ETLs.

 

 

Luke

Hi Luke,

 

I am not aware of a maximal size although memory issues can arise with very large files, but also for such a scenario there are alternatives.

 

Scalability is only one aspect of FME server flexibility and distribution are some other aspects to consider.

 

Beside that 10gb can be easily translated, a lot depends on the strategy and trouble shooting along the way.

Hi Itay, 

 

 

Thanks for the info.  You indicate possible alternatives, can you elaborate or suggest the best approaches?  

 

 

I've still not managed to find any info or reccomendations / best practice for large file transfer designs which surprises me somewhat given the nature of spatial data & also the maturity of FME as a spatial ETL.  

 

 

I'm thinking twice whether to use FME for the extract / load given i am porting between two oracle DB's as well as minimal transformation.

Reply


Community Stats

30,108
Posts
114,284
Replies
37,742
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.