Skip to main content
Solved

Can I let FME Desktop use Resources over Hadoop?

  • Jun 28, 2014 4:42 PM UTC
  • 3 replies
  • 74 views

Hello everybody!

 

 

I'm asking, if it is possible to use a Hadoop Distribution on FME Desktop's work.

 

On one side, for storing input and output files but more importantly for sharing the cpu-usage of a translation into multiple Hadoop tasks on multiple Computers.

 

I'm working on a project right now where the tranlation with 100.000 lines takes about 45 minutes.

 

I can only imagine the time it will need on about 1.200.000.000 lines if I'm computing it on a single computer.

 

 

Thanks for reading!

 

 

PS

 

So far I'm only working on FME Desktop, I watched some Youtube Videos about FME Server and could imagine that it could work with Hadoop quiet well.

Best answer by david_r

Hi,

 

 

I have never heard of anybody using Hadoop with FME, but it would certainly be cool to hear a success story with that kind of technology! That being said, I suspect it'd be easier to just use regular FME Server functionality to accomplish the same thing (splitting a process over several enginges, distributed over several servers)

 

 

For the idea about Hadoop, just be aware that you'd probably need a separate FME Desktop license for each server...

 

 

David
View original
This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

3 replies

david_r
Celebrity
  • Best Answer
  • Jun 30, 2014 8:56 AM UTC
Hi,

 

 

I have never heard of anybody using Hadoop with FME, but it would certainly be cool to hear a success story with that kind of technology! That being said, I suspect it'd be easier to just use regular FME Server functionality to accomplish the same thing (splitting a process over several enginges, distributed over several servers)

 

 

For the idea about Hadoop, just be aware that you'd probably need a separate FME Desktop license for each server...

 

 

David

richardatsafe
Safer
Forum|alt.badge.img+11

If you want to test the beta you can certainly use Hadoop for storing input and output files with HDFSConnector, but as far as parallel processing goes David makes a good point: FME Server will allow parallel processing with multiple engines on multiple machines.

https://knowledge.safe.com/idea/54272/working-with-hadoop-hdfs-preview-fme-2018.html


danilo_fme
Celebrity
Forum|alt.badge.img+58
  • Celebrity
  • Oct 4, 2017 3:20 AM UTC

If you want to test the beta you can certainly use Hadoop for storing input and output files with HDFSConnector, but as far as parallel processing goes David makes a good point: FME Server will allow parallel processing with multiple engines on multiple machines.

https://knowledge.safe.com/idea/54272/working-with-hadoop-hdfs-preview-fme-2018.html

@RichardAtSafe this is very interesting news!