Open ideas have been reviewed by our Product Management and are open for commenting and voting.
The Active periods in the FME Flow Engine Management currently (FME 2024.1) allows for a daily and weekly recurrence. It would be useful if there would be more fine-grained control so we could also specify, for example, day of week (so we can set an active period during regular weekday office hours), monhtly or day-of-month
This was posed as a question a few years ago so I thought I’d turn it into an idea as it doesn’t seem to exist as yet. The single merged feature type is a powerful feature of reader feature types and the behaviour should be replicated easily in the Feature Reader - an example being where one wants to scan all feature classes in a file geodatabase and read only the features that intersect the initiator’s geometry.This means not specifying particular feature types to read, as the existing reader functionality goes. The Feature Reader has the option of a single output port but the existing reader behaviour takes care both of the feature types to read and the output port and that’s what I’m talking about here.If I’ve missed an idea then feel free to close this and to redirect.
I find the SchemaScanner essential in discovering schemas of source data and feeding dynamic writers. (FME 2024.1,1 is the version I’m using)I also find the SchemaScanner very useful for merging schemas by ignoring attribute names or grouping by attribute values.This leads me to want more Schema Analysis functionality.I think it would be very helpful to have the ability to extract a common schema from a set of features.for example if all features have the same set of attributes:featureName, featureDesc, featureID. It would be great to extract the common schema to support writing to what would look like a base class (table) and then have the ability to write the full schema and features in another table to extend the base class. Or have an option to extract the common schema and at the same time have the option to pack the remaining attributes of the passed through features into a json array (like the AttributeJsonPacker) thus merging the features into a single common schema.
Hi FME community,From FME 2021.1 onwards the Autodesk ReCaP writer will be deprecated (https://community.safe.com/s/article/FME-Deprecations). Please vote when you currently use the Autodesk ReCaP writer and don't want it to deprecate. BackgroundEngineers of our infrastructural projects use very often the Autodesk ReCap data format for viewing their point clouds in applications like Revit, AutoCAD, Autodesk ReCaP and Autodesk 3ds max. With this efficient format they can compare their designs with the current - often complex - situation. I would suspect that the whole engineering industry benefits from the efficient way of FME indexing raw point cloud data that's ready to use in BIM programs like Revit. Several alternatives we’ve considered for our point cloud workflow:Importing txt, xyz, pts files in Revit, however the indexing happens on PC’s of the engineers where they’re not equipped for.Importing LAZ files into Autodesk ReCap and save the RCP/RCS files to view them in Revit, however this is an error prone manual process that we want to automate. Different versions of FME paralleled to be still capable of using the ReCap writer, however this is not possible due to our use compability issues with FME Server.Viewing the point clouds in Point Cloud Scene Layers, but comparison with the designs in BIM applications is more difficult.If you use other (maybe even better) formats, please let me know. Else, vote for getting the format high on the backlog of FME's Product Managers.Illustration in ReCap of Rijksmuseum, The Netherlands (source: open elevation dataset AHN3)Kind regards,Jan Droesen FME specialist Movares Netherlands
I think it would be nice to have a checkbox “Retry on canceled” in the “run workspace” element in an automation that would manage resubmitting the job when being canceled (for instance, when it exceeds the max. queue time).
I would like to see FeatureWriter or another transformer updated to allow data being written to a temporary table and an SQL statement (or several) to be run afterwards to manipulate that data, most likely to move it to its permanent location and/or create partitions for the range to be inserted.Related to a community question I opened last year, and to a similar idea I submitted for temporary files.One method I commonly use (at least in Postgres) to efficiently write large amounts of data to large tables is to create a temporary table that’s identical to my target table (minus indexes and constrainsts), use some bulk loading method like COPY to fill that table with the data I’m inserting, and then running whatever statement I need to insert/upsert the final table with that data. This allows the DBMS to use whatever method works best for filling that data and does the actual data insertion as a single transaction. For FME specifically, this allows rows to be upserted without opting out of bulk-mode insertion.As you can see from my community question, this turns out to be very complicated with FME because transactions/sessions aren’t scoped across transformer nodes in a way that makes it possible to use temporary tables, so you need to:Create a schema for fake temporary tables Figure out some way to make a table name that is unique per run, which works reliably for both FME Form and FME Flow and fits within the table name length limits CREATE UNLOGGED TABLE “fme_fake_temp”.temp_table_name LIKE source_table_name EXCLUDING CONSTRAINTS EXCLUDING INDEXES In a subsequent SQLExecutor, run your insert statement and drop the tableDropping the table is often unreliable, insertion errors will often cause the temporary table to remain there and slowly inflate the fake temporary schema, requiring preriodical cleanup. Creating a unique table name is also inconveinient; $(FME_UUID) (which is what TempPathnameCreator uses) would work reasonnably well, since it’s an underscore-separated UUID prefixed with the transformer name, but it doesn’t work with FeatureWriter and requires transformer names to be valid table identifiers and shorter than 25 characters long to fit under Postgres’ 63 character limit. Besides that, FME offers no identifier that is unique per run and available on both Form and Flow.This is a lot of gymnastics for what I believe is a fairly common data insertion pattern, and FME would benefit from supporting it “natively”, so to speak, possibly using actual temporary tables. Other DBMS support similar private/temporary tables, like Oracle’s Private Temporary Tables, MySQL’s temporary tables, MS-SQL’s #TemporaryTables, or the (somewhat more convoluted) SQLite approach of mounting a separate database (possibly one that only exists in :memory:) as a schema, and creating temporary tables there. For FeatureWriter proper, however, these mechanisms wouldn’t work, so having FME track temporary “real” tables and drop them at the end of the flow run would likely be the least disruptive option. Using the actual mechanisms for session-local tables would probably require something more akin to InlineQuerier.
Migration Note: Idea originally posted on 2015-10-29
Hello all,Working for Dutch local governmental organisations, I more and more encounter webservices which require a certificate to be send as part of the authentication / authorization process. My guts feeling is, that this is especially the case with API's providing 'personal information', and that it is a hobby of Java developers...Searching the Safe forums, I found this post most prominent, which suggests to use the python requests library (also the most FME compatible solution The Internet provides).I can't see how to implement this in a connection-services-template...So, please Safe Software, make this possible...Martin
Right now only file.path, file.rootName and file.event are visible: It feels like it shouldn’t be difficult to expose a new output attribute for the file.type, to be able to tell whether the updated item is a folder or a file, or the file.extension, rather than hoping for the best that nobody included the sequence “.shp” in the name of their excel files.Yes, you could easily do this within the workspace, but it doesn’t feel right.
It would be beneficial to have a transformer that displays all attribute values simultaneously, rather than using the TestFilter Transformer. In some cases, I need to process each value, which can be time-consuming when using TestFilter to have each attribute value on its own tab.
It would be very helpful to have a Rejected-port on the PythonCaller.Outputting a failed feature could e.g. look like this:self.pyoutput_failed(feature)This would simplify the pattern where you need to set a temporary attribute to indicate success/failure, then a Tester and then removing the temporary attribute.
Instead of creating features with no attributes, enhance the creator to optionally allow a table of attributes and values to be used to populate a series of features, one per row of the table.
Hi, renaming an attribute in large workspaces oftentimes takes a lot of time because of the related manual adjustments needed. I would appreciate a function that is able to change a given attribute to another given attribute within a whole workspace in every Transformer and Writer.Best regardsJudith
For comments and workspace description etc a spell check would be a nice to have. My splling is terribkle. Currently I have to copy all the text into another tool.
Basically add a CoordinateSystemSetter inside the VertexCreator to avoid having to use a CoordinateSystemSetter up front. Just 1 extra 'optional' parameter required.
When you have created geometry you are usually done with attributes supplying the coordinates, GeometryReplacer has an option to drop them, let's do the same for VertexCreator.
I often use the DuplicateFilter to get a quick list of unique attribute values for a specific attribute. For this use case all other attributes that are still present after the DuplicateFilter are not needed and to see the desired result, you either have to remove the superfluous attributes with another transformer like the AttributeKeeper or hide them in the visual preview. That’s why I suggest to add the option to remove unselected attributes to the DuplicateFilter.Remark: I’m aware, that you can already get a similar result with the Aggregator, but if a feature geometry is present all input geometries would have to be processed allthough they are not needed for the above use case.
As FME users deal with increasingly complex data transformations, the ability to efficiently compare workspaces becomes essential. Currently, the "Compare Workspaces" feature is confined to a single window, which can be limiting for those using multiple monitors. Current State: FME allows side-by-side comparisons within a single application window. Proposed Feature: Independent windows that can be placed on different monitors,I propose a Dual-Window Workspace Comparison feature that would allow users to open and view two separate FME windows on different screens. Key Benefits:Boosts productivity by leveraging multiple monitors. Simplifies workspace comparison and troubleshooting. Enhances user experience in managing complex data transformations.
BACnet (Building automation and control network), is a protocol that building utility sensors use. If FME was able to read the sensors directly the information that could be read in would be extremely valuable for aggregating data on building performance/status. I see this working fantastic with FME Data Streaming and pushing this data to ArcGIS services, so we can visual in dashboards which rooms are currently being cooled/the temperature/which rooms are occupied/etc, and so many other things.
Version control is only as good as the business processes surrounding it - being able to commit to a git repo/FME version control is great, but it’s easy to miss in a publishing workflow and is not currently possible to enforce.In order to support more robust SDLC & change control processes, it would be extremely useful to be able to either automate a git commit when publishing/republishing a workspace, or to configure a workspace/repo/flow env to make commits compulsory.My blue-sly vision for this would be that the configuration is set at the repository level on FME Flow, and in repos where compulsory commit is enabled, the publish action in FME Form cannot be used until a commit has been performed with the current version of the workspace.
Would just make workspaces a little cleaner.
When using the Multipart / Form Data option in a HTTPCaller, the MIME Type of the upload is quite hidden. It took me a while to figure out the option even existed. Right now you have to click on the ...-button to change the type while it would be a lot more clear to add an extra column to the Multipart Upload option table which is already there.
I like that in FME Flow it's possible to configure cleanup tasks.However, to me it wasn't so intuitive how the cleanup frequency actually works.For example, if I configure a cleanup task to cleanup files that are older than 1 hour, I was tempted to think that files that would be in the folder would be cleaned up after 1 hour (and surely after multiple hours).However, I learned that that isn't actually the case. I learned that there is actually a relation/dependency between;a) The 'Remove Files Older Than' parameter (at tab 'Tasks' of 'System Cleanup'), andb) The 'Task Interval' parameter for 'Normal Conditions' (that can be configured at the 'Configuration' tab of 'System Cleanup')Specifically, how often the cleanup tasks take place, are configured by the 'Task Interval’ parameter, which I think by default is 1 day.So, for my newly configured cleanup task, I would need to wait at least a day for files to be removed, and not 1 hour.I think it may avoid some confusion, when a warning would show up, in case a user would configure a cleanup task to be scheduled more frequent than the general 'task interval’ in which the cleanup tasks are performed. Ps. This is an idea following ‘item 2’ of my question on the FME Community: How to periodically Cleanup the FME Flow Temp Resource (i.e. '$(FME_SHAREDRESOURCE_TEMP)') | Community (safe.com)
TL;DR: Allow me to set a list of pfererred file types to read in a Generic Reader and FeatureReader.Background: When feeding files into a Generic Reader or FeatureReader, it appears that FME (Form/Flow) selects one format automatically from its available formats in alphabetical order, for example:.gml: AIXML5 .gpkg: GEOPACKAGE_RASTER .json: CITYJSONI’m working with geospatial vector data, so instead I’d prefer:.gml: GML .gpkg: OGCGEOPACKAGE .json: GEOJSONThere’s currently no way set a preference, an allow- or blocklist to set which formats to use or not.Some workarounds I found on these forums:Rename files (not viable for user-provided data) Build my own mapping with a FileNameExtractor to get the extension → AttributeValueMapper and feed it into a FeatureReader (may be error-prone) Create a custom format that starts with AAAAA… to place it at the start of the alphabetical listRelated forum posts and somewhat similar ideas:
I regularly run routines writing to multiple tables in multiple databases. When writing out several hundred thousand records, it is troubling to get a warning in the translation log that merely states "Microsoft SQL Server Non-Spatial Writer: 2 attribute value(s) were truncated"Since FME knows that it truncated a value, could the warning be expanded to include additional information like the table, field, and data value so that we don't have to spend time with AttributeValidators and StringLengthCalculators to try and figure out which of the myriad of data entries was the offending value?
In-depth courses for all FME skill levels.
Explore our extensive collection of how-to guides and tutorials.
Log an issue with us. Our support team is ready to help.
