Hello,
We are using HDFS as our file system for some of our files. Can we integrate FME solutions inside HADOOP in the Map-Reduce process? Or there is any other practice to use FME in HADOOP context? Thank you for your time,
Kitty
Solved
FME and HADOOP - best practice
Best answer by mark2atsafe
Hi Kitty,
There's nothing official within FME that will connect with Hadoop at the moment. The best I can say is that you might be able to connect to Hadoop databases (Cloudera) with the JDBC reader/writer.
I see there are already enhancement requests with our developers to support HDFS as a file system, but nothing much more than that, so any integration looks like it will be a ways into the future.
I'd suggest you file a case with our support team (safe.com/support) and let them know you are interested in Hadoop integrations with FME. If you can provide them with some specific use cases then they will be able to log your interest and maybe increase the priority of the development work.
Sorry I can't help much more at this point.
Mark
Mark Ireland
Product Evangelist
Safe Software Inc
+43- Safer
- Best Answer
Hi Kitty,
There's nothing official within FME that will connect with Hadoop at the moment. The best I can say is that you might be able to connect to Hadoop databases (Cloudera) with the JDBC reader/writer.
I see there are already enhancement requests with our developers to support HDFS as a file system, but nothing much more than that, so any integration looks like it will be a ways into the future.
I'd suggest you file a case with our support team (safe.com/support) and let them know you are interested in Hadoop integrations with FME. If you can provide them with some specific use cases then they will be able to log your interest and maybe increase the priority of the development work.
Sorry I can't help much more at this point.
Mark
Mark Ireland
Product Evangelist
Safe Software Inc
mark2atsafe wrote:
Hi Kitty,
There's nothing official within FME that will connect with Hadoop at the moment. The best I can say is that you might be able to connect to Hadoop databases (Cloudera) with the JDBC reader/writer.
I see there are already enhancement requests with our developers to support HDFS as a file system, but nothing much more than that, so any integration looks like it will be a ways into the future.
I'd suggest you file a case with our support team (safe.com/support) and let them know you are interested in Hadoop integrations with FME. If you can provide them with some specific use cases then they will be able to log your interest and maybe increase the priority of the development work.
Sorry I can't help much more at this point.
Mark
Mark Ireland
Product Evangelist
Safe Software Inc
Thank you Mark, we will start with the claudera with JDBC, hoping in the future there with be support also for HDFS.
+17- Contributor
kitty wrote:
Thank you Mark, we will start with the claudera with JDBC, hoping in the future there with be support also for HDFS.
I see you did vote up the "idea" related to this as well. We think we could do something like an "HDFSUploader" type transformer ala the S3Uploader that could be used after a FeatureWriter eventually.
HDFSConnector transformer was added to FME 2018. HDFSConnector transformer allows you to move files in and out of HDFS - see also: Working with Hadoop HDFS
Reply
Helpful Members This Week
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.