Skip to main content

Open ideas have been reviewed by our Product Management and are open for commenting and voting.

Filter by idea status

Filter by product area

4628 Ideas

Working with Hadoop HDFS: Preview FME 2018Released

Connecting to HDFS is now in FME Desktop 2018As of the FME 2018.0 release, we have added the HDFSConnector transformer as a technology preview. This means that while it’s not quite ready for prime time yet, it is available to take for a test run.We’d like to call on anyone interested in this new transformer and especially anyone who voted on any HDFS, Hadoop, Hive enhancements for this to try this transformer and let us know what you think.Give us your feedbackWe’re actively working on developing more integration with Hadoop systems and we want to hear from you!How would you like to use FME in a Hadoop setting? How are you using Hadoop and what formats are you using to store your data? Does the HDFS connector meet your needs or do you need some more functionality like a Hive, Hbase, Oozie, or Spark connector? Would you like to be able to trigger a Hadoop command or execute a query once an upload was complete? If so, what would that look like?Please post your feedback as a comment back to this idea.How does it work?The HDFSConnector uses your HDFS account credentials (either via a previously defined FME web connection, or by setting up new FME web connection right from the transformer) to access the file storage service.Depending on your choice of actions, it will upload or download files, folders, and attributes; list information from the service; or delete items from the service. On uploads, path attributes are added to the output features. On List actions, file/folder information is added as attributes.https://www.youtube.com/embed/85RxCTjdquo

stevenjh
Contributor
stevenjhContributor

Emulation of interactive workspace running with named run instancesArchived

Since the introduction of caching and the tools on bookmarks to "Run Just Contained" and "Between Selected" I find myself almost writing 'multiple workbenches in one' and only working with them interactively.Which has made me think it would be neat to have a way to actually Run them in a traditional FME sense.Original IdeaI've only thought about this a little bit, but maybe it's a mix of:a property that we can set on the bookmark to state its 'Run-able' a section in bookmarks where we can order run-able bookmarksShow a hierarchy of linked bookmarks that would run after one anotherInclude an option to set bookmarks to run between to handle forking in a data flow.A step that allows disabling bookmarks to prevent running un-wanted upstream sections when running betweenFrom the traditional Run prompt, add an option that allows you to set specific bookmark(s?) to run or 'All'Example"Bookmark One" runs some transformers to generate an AOI to a database"Bookmark Two" reads this AOI to download and generate more data saved locallylinked "Bookmark Processing" runs some initial processing that generates results and errors that require manual tweaks on the downloaded data"Bookmark Three" reads the downloaded in "Bookmark Two"Linked "bookmark Processing" runs again against the corrected data, generates results and errors ...Evolved ideaAs I've typed this out I'm thinking maybe rather than run-able bookmarks we create "named run-instances", which has a similar concept where bookmarks are ordered within each instance but de-couples the bookmarks themselves.Now you could create a separate tab view for each named instance which shows just that data flow and links between bookmarks that don't have feature-flows connecting them etc..'Named run-instances" is a much better idea and I've changed my draft title from "Run a workspace by bookmark"ps I've now ended up thinking about this more than just a little and I left this as a stream of conscious because I think there's value in understanding how people get to ideas (and I didn't want to rewrite everything!)