Skip to main content
  • EN
Hello everybody!

 

 

I'm asking, if it is possible to use a Hadoop Distribution on FME Desktop's work.

 

On one side, for storing input and output files but more importantly for sharing the cpu-usage of a translation into multiple Hadoop tasks on multiple Computers.

 

I'm working on a project right now where the tranlation with 100.000 lines takes about 45 minutes.

 

I can only imagine the time it will need on about 1.200.000.000 lines if I'm computing it on a single computer.

 

 

Thanks for reading!

 

 

PS

 

So far I'm only working on FME Desktop, I watched some Youtube Videos about FME Server and could imagine that it could work with Hadoop quiet well.
Hi,

 

 

I have never heard of anybody using Hadoop with FME, but it would certainly be cool to hear a success story with that kind of technology! That being said, I suspect it'd be easier to just use regular FME Server functionality to accomplish the same thing (splitting a process over several enginges, distributed over several servers)

 

 

For the idea about Hadoop, just be aware that you'd probably need a separate FME Desktop license for each server...

 

 

David

If you want to test the beta you can certainly use Hadoop for storing input and output files with HDFSConnector, but as far as parallel processing goes David makes a good point: FME Server will allow parallel processing with multiple engines on multiple machines.

https://knowledge.safe.com/idea/54272/working-with-hadoop-hdfs-preview-fme-2018.html

If you want to test the beta you can certainly use Hadoop for storing input and output files with HDFSConnector, but as far as parallel processing goes David makes a good point: FME Server will allow parallel processing with multiple engines on multiple machines.

https://knowledge.safe.com/idea/54272/working-with-hadoop-hdfs-preview-fme-2018.html

@RichardAtSafe this is very interesting news!

 

Reply


Community Stats

30,632
Posts
116,065
Replies
38,263
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.