Skip to main content
Solved

Can I let FME Desktop use Resources over Hadoop?

  • June 28, 2014
  • 3 replies
  • 28 views
H
Hello everybody!

 

 

I'm asking, if it is possible to use a Hadoop Distribution on FME Desktop's work.

 

On one side, for storing input and output files but more importantly for sharing the cpu-usage of a translation into multiple Hadoop tasks on multiple Computers.

 

I'm working on a project right now where the tranlation with 100.000 lines takes about 45 minutes.

 

I can only imagine the time it will need on about 1.200.000.000 lines if I'm computing it on a single computer.

 

 

Thanks for reading!

 

 

PS

 

So far I'm only working on FME Desktop, I watched some Youtube Videos about FME Server and could imagine that it could work with Hadoop quiet well.

Best answer by david_r

Hi,

 

 

I have never heard of anybody using Hadoop with FME, but it would certainly be cool to hear a success story with that kind of technology! That being said, I suspect it'd be easier to just use regular FME Server functionality to accomplish the same thing (splitting a process over several enginges, distributed over several servers)

 

 

For the idea about Hadoop, just be aware that you'd probably need a separate FME Desktop license for each server...

 

 

David
This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

3 replies

david_r
Celebrity
  • david_rCelebrity
  • Best Answer
  • June 30, 2014
Hi,

 

 

I have never heard of anybody using Hadoop with FME, but it would certainly be cool to hear a success story with that kind of technology! That being said, I suspect it'd be easier to just use regular FME Server functionality to accomplish the same thing (splitting a process over several enginges, distributed over several servers)

 

 

For the idea about Hadoop, just be aware that you'd probably need a separate FME Desktop license for each server...

 

 

David
richardatsafe
Safer
Forum|alt.badge.img+10

If you want to test the beta you can certainly use Hadoop for storing input and output files with HDFSConnector, but as far as parallel processing goes David makes a good point: FME Server will allow parallel processing with multiple engines on multiple machines.

https://knowledge.safe.com/idea/54272/working-with-hadoop-hdfs-preview-fme-2018.html

danilo_fme
Celebrity
Forum|alt.badge.img+51
  • danilo_fmeCelebrity
  • Celebrity
  • October 4, 2017

If you want to test the beta you can certainly use Hadoop for storing input and output files with HDFSConnector, but as far as parallel processing goes David makes a good point: FME Server will allow parallel processing with multiple engines on multiple machines.

https://knowledge.safe.com/idea/54272/working-with-hadoop-hdfs-preview-fme-2018.html

@RichardAtSafe this is very interesting news!

 

Partner Solutial Brazil - www.solutial.com.br

Community Stats

32,120
Posts
121,954
Replies
39,772
Members

Latest FME

Terms & ConditionsCookie settingsAccessibility statement
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.