Shape the future of FME with your ideas
Open ideas have been reviewed by our Product Management and are open for commenting and voting.
When using the Multipart / Form Data option in a HTTPCaller, the MIME Type of the upload is quite hidden. It took me a while to figure out the option even existed. Right now you have to click on the ...-button to change the type while it would be a lot more clear to add an extra column to the Multipart Upload option table which is already there.
I like that in FME Flow it's possible to configure cleanup tasks.However, to me it wasn't so intuitive how the cleanup frequency actually works.For example, if I configure a cleanup task to cleanup files that are older than 1 hour, I was tempted to think that files that would be in the folder would be cleaned up after 1 hour (and surely after multiple hours).However, I learned that that isn't actually the case. I learned that there is actually a relation/dependency between;a) The 'Remove Files Older Than' parameter (at tab 'Tasks' of 'System Cleanup'), andb) The 'Task Interval' parameter for 'Normal Conditions' (that can be configured at the 'Configuration' tab of 'System Cleanup')Specifically, how often the cleanup tasks take place, are configured by the 'Task Interval’ parameter, which I think by default is 1 day.So, for my newly configured cleanup task, I would need to wait at least a day for files to be removed, and not 1 hour.I think it may avoid some confusion, when a warning would show up, in case a user would configure a cleanup task to be scheduled more frequent than the general 'task interval’ in which the cleanup tasks are performed. Ps. This is an idea following ‘item 2’ of my question on the FME Community: How to periodically Cleanup the FME Flow Temp Resource (i.e. '$(FME_SHAREDRESOURCE_TEMP)') | Community (safe.com)
TL;DR: Allow me to set a list of pfererred file types to read in a Generic Reader and FeatureReader.Background: When feeding files into a Generic Reader or FeatureReader, it appears that FME (Form/Flow) selects one format automatically from its available formats in alphabetical order, for example:.gml: AIXML5 .gpkg: GEOPACKAGE_RASTER .json: CITYJSONI’m working with geospatial vector data, so instead I’d prefer:.gml: GML .gpkg: OGCGEOPACKAGE .json: GEOJSONThere’s currently no way set a preference, an allow- or blocklist to set which formats to use or not.Some workarounds I found on these forums:Rename files (not viable for user-provided data) Build my own mapping with a FileNameExtractor to get the extension → AttributeValueMapper and feed it into a FeatureReader (may be error-prone) Create a custom format that starts with AAAAA… to place it at the start of the alphabetical listRelated forum posts and somewhat similar ideas:
I regularly run routines writing to multiple tables in multiple databases. When writing out several hundred thousand records, it is troubling to get a warning in the translation log that merely states "Microsoft SQL Server Non-Spatial Writer: 2 attribute value(s) were truncated"Since FME knows that it truncated a value, could the warning be expanded to include additional information like the table, field, and data value so that we don't have to spend time with AttributeValidators and StringLengthCalculators to try and figure out which of the myriad of data entries was the offending value?
The RC of IFC 4.3 is available at:https://technical.buildingsmart.org/standards/ifc/ifc-schema-specifications/ Would be great to try out the RC already if possible with read and write with FME. Not sure how IFC is implemented vs XML. However with XML you can fairly easy point to an xsd and get a new xml standard within FME. I guess its not as easy to do the same for IFC + XSD ?Mostly using this towards roads, railway and bridgesChanges here: https://technical.buildingsmart.org/standards/ifc/ifc-schema-specifications/ifc-release-notes/
Currently there is only the option to filter on the subject of an incoming email using the Email-IMAP trigger in automations. It would be very handy to also have a filter based on the sender. We have an automation running that gets triggered by a certain text in the subject of an email, independent of who the sender is and downloads the attachments for later use in the automation.Since we have several projects using this email box we have to build a seperate workspace that filteris on the sender. If the sender is of an unexpected email the downloaded attachments get deleted to not clutter the recources on FME Flow and the automations doesn't continue. It would be very handy to have this filter option already in the Email-IMAP trigger.
Context: Schema featuresWhen dealing with dynamic workflows, FME introduces the need to work with schema features, which are used by some transformers to recognize all the “columns” (attribute types, list attributes, struct fields with their types, geometry type, etc.) to be found in incoming features. SchemaScanner is able to generate those from data of an unknown type by (as the name suggest) scanning the first N lines of data and assuming it is representative of the rest. Dynamic input nodes for Custom Transformers also can’t know the schema of incoming features unless a schema feature is supplied.The Problem: Making/getting schema features (when you know your schema)The thing is, even in dynamic workspaces, users tend to have some notion of what their data format is at any given point (as can be previewed by looking at cached features in FME Form, and especially if AttributeManager is used to set attribute types), yet getting a schema feature representative of that known format is fairly difficult. Official tutorials reccomend making one by hand, using an AttributeCreator, which basically achieves the same result as the “User Attributes” tab of a FeatureWriter, only much more clumsily and difficult to keep up to date when attributes are added or removed. FME deliberately avoids exposing enough information to bridge that gap in Python or something similar.The Solution: SchemaCreator?FME should probably have a transformer that’s a cross between a SchemaScanner and a FeatureReader: a transformer that recieves features and outputs a single schema feature, before reoutputting the input features. Unlike SchemaScanner, though, the schema definition would be static, defined by the user (or detected automatically) using a Feature Type menu mostly identical to what is used by FeatureWriter, with the cached input feature values used to provide type suggestions.A mock-up of the basic idea. The interface only needs to reproduce the Feature Type menu used by writers and FeatureWriter, and for the transformer to output it as-is on the <Schema> port.
It would be nice that the R interpreter would be able to look at a certain folder on the C disk for example and when there is different version of R, then the R interpreter would automatically change it.For now, I manually changed the interpreter, however ideally I would like during an update of R, that the R interpreter would automatically change the R interepreter.Maybe something like ${R_HOME} or ${RHOME}?
My workbenches always start with the name None → None. Even though I save the workbench with a unique name.I think it gets the name from the “none template” I use, but it will be nice if it would show the name of my saved file instead of None → None.
Feature transmission is something I’ve noticed to be a common cause for slowdowns in my flows (especially with Python), and one important way to keep execution time down is to make sure that features are transmitted in bulk mode.The only list of what does and doesn’t support Bulk mode is this one, which is long outdated.That leave log messages like “Splitting bulk features into individual features” as the main way through which users are meant to figure out what does and doesn’t use Bulk mode, which are easily missed and don’t really give a complete view of data flow. Worse, this is often conditionnal, as a Postges FeatureWriter is able to insert features in bulk mode, but upserting falls back to individual features.Adding a visual cue (whether only while running the flow, or also in edit mode if that information is known) would help. Some possibilities on top of my head:A crow’s foot, like they use in UML for many-to-many/many-to-one relations, in this case indicating that both sides support bulk mode or that one supports it and the other will have to split it.Colored lines while running the workspace to indicate which connections are actually sending data as feature tables, with colors near the recieving node if it ends up splitting the features recieved. Some kind of marching ants animation when funning the flow to show features being sent, vs a moving gradient similar to Windows indeterminate progress bars for bulk mode if using colors isn’t an option, perhaps?Giving some form of visual indication would go a long way in demistifying the feature for most users, I believe.
When you add columns to, let’s say FME Jobs Completed page, the following occurs:Scrolling Bar Menu is displayed at the bottom of the page which is rather inconvenient. Edge display the page content better than Chrome as in re-adjusting the columns when adding / removing columnsFound it at the bottom of the page whilst looking for it
Support for apache arrow in memory data exchange format as input to Pythn script caller and allow an apache arrow object to be returned by the python caller as well would be great. There would be a number of advantages to this.Remove the need to install fme objects into my python environment. Make the python interface cleaner and easier to use. Just convert the apacje arrow object into your data frame, Reducing the barrier to using python in workflows. Scripts could more easily be developed and tested in IDEs like vscode, and pycharm making the development of scripts far nicer. Supported by many libraries. e.g. tensorflow, pandas, polars, duckdb, pyspark etc. Would make it easier to embed machine learning models into workflows. Potentially opens up the integration of other languages into FME due to the arrow format being language an gnostic.
As mentioned in other posts, HttpCaller can be used to check for the existence of an API resource, in which case a 404 response is not an error.I often use this approach in upload workflows where I need to check if a layer already exists in Geoserver and needs to be refreshed or if it needs to be created. The alternative of making a call to list out all of the layers and then parsing that list to see if the one I’m interested in is there is much slower.It would be great to have an option to output 404s (and potentially other response codes) to custom output ports, rather than rejecting and raising an error. 132 2024-6-4 01:01:38 | upload_checker_2_HTTPCaller_2 (HTTPFactory): [51]: HTTP transfer summary (*.*.*): status code: 404, download size: 91 bytes, DNS lookup time: 4 ms, total transfer time: 68 ms 133 2024-6-4 01:01:38 | upload_checker_2_HTTPCaller_2 (HTTPFactory): An error occurred while accessing the URL Httpcaller Error | Community (safe.com)Ability to change transformer, reader and writer logging level | Community (safe.com)
A seemingly simple idea: Add formatting options into the JSONTemplator.It’s not really needed, but I would love to not have to add a JSONFormatter after the JSONTemplator if I want a nicely formatted document. That is all.
As the title suggest it would be nice wth a glob pattern when listing items fom a sharepoint location. For example: /*.pdfOr on input field for the glob pattern: *.pdfinclude subfoler “yes”would retrieve only items ending with .pdf
Enable the selection of Deployment Parameters in every System Event Notification within the text editor. In that way you can even better customize the System Event Notification for every environment and deployment.Best regards
FME's Python interface for feature attribute manipulation seems to be mainly oriented towards small datasets, as attributes are always only accessed field by field and feature by feature. Python has a very large ecosystem around data science and data processing of very large tables, so it's often practical to load data in a dataframe and run all your computations on the dataset as a whole, to benefit from various performance optimizations around vectorization, not to mention use of fammiliar tools for people in the data science field.Right now, loading features into a dataframe (I'm talking hundreds of columns and hundreds of thousands of rows, millions of cells) is both slow and very error-prone for a few reasons:Loss of schema information: getAttributeType() in the Python API can't return the full set of FME feature types, so dynamic workbenches can't properly handle things like dates unless a lot of work is done around schema detection and handling.Difficult to parse timestamp format: Python's standard library and most dataframe libraries can only parse fractionnal seconds up to 6 decimals, while FME does up to 9 and DatetimeNow() notably provides 7 decimals. This is a difficult problem on its own, but time series are common in the data science field, so it's especially noticeable here.Attribute access is per-row and field-by-field (slow): With no way to do bulk access on features, the data is accessed and converted value by value (save for lists, which can be accessed as a whole). If you have several million fields to read, that's a lot of slow python code running in very tight loops.Null and missing values are distinct and need to be checked separately: getAttribute() only returns None on missing values. When encountering a null value, it returns an empty string, which can throw off dataframe libraries and need to be checked for each value suceptible of returning null, adding to the overhead.Feature output is also per row and per feature: It's also very slow, and I belive it also blocks on waiting for downstream.Using files to pass data between nodes is a lot to ask: You can use feature readers and writers to create temporary files so that Python could load and dump these features in one shot, but even if I could get it to work (I haven't), it wouldn't be worth the dozen of extra nodes needed to make this work and all the visual clutter that creates.The easy way this could be resolved, in my eyes, is to simply provide a new Python transformer (the titular DataframeTransformer) that already does the conversion of features to and from dataframes and only provides the user's code with said dataframe.There have been efforts lately to create a standard dataframe interchange protocol throughout the Python ecosystem to encourage interoperability across libraries and avoid locking-in users into using a specific library, which Pandas supports converting to and reading from. To avoid depending on Pandas and/or its APIs, fmeobjects could expose some hypothetical "FeatureDataframe" object and leave it up to the users to load it into their dataframe library of choice.Related questions[...] my results are in a Pandas DataFrame. There&#039;s any way to convert that table to features to use in FME?Bulk feature marshalling/unmarshalling from FME to Python
HiI have a travelling salesman use case, where a fleet of perhaps 2-4 vehicles need to reach a shared set of locations in a municipality. The locations are patients that are to be visited during the day by one of the vehicles. The list of locations (patients) is shared between the vehicles and only one of the vehicles needs to visit each patient. Also, all of the vehicles always start and end the day at the same location (the health clinic). In short, I would like FME to calculate the shortest paths for a shared set of locations for a few vehicles.
Hi everyone,I'm proposing to enhance the Text User Parameter by an option for on the fly input checking with RegEX syntax. The check shall be appied whenever any insert happens in the text field and notify the user immediately upon not allowed characters (e.g. background colour turns red and Excecute button becomes invalid)Regex checking could be triggerd by a checkbox in Manage User Parameters section.
FME can currently read in a single layer file, however, FME doesn't support reading layer files that contain group or nested layers.If you are interested, please comment below on how this would be important for your use case.
I have a process that is fully automated except for the last step which is creating a transmittal or eTransmit, a special ZIP with references which can be created from AutoCad (AutoCAD 2024 Help | Package Drawings for Transmittal | Autodesk)Could creating a transmittal be added to FME?
Working with teams all using FME Flow in various time zones, I think it would be beneficial if there was a setting to control the timestamping of log files. My idea would be to add an option under the Log Settings to toggle Logged Timestamp Information between Local Computer Time and UTC.
Title pretty much says it all, it would be a good thing to also have a Running Job Expiry time setting for Flow Apps.
Idea for a new simple transformer (or extension to the AttributeFilter):Like the AttributFilter but for ParametersWhile it's easy enough to set up a TestFilter it does take a little bit of time to configure sometimes. Having a ParameterFilter which can automatically check for allowed values (e.g., if using the 'Choice' type) and just create the output ports would be nice.Simply pick your parameter form a drop down list then enter values in the same way as an AttributeFilter OR click a button to import values (if the ParameterType supports it)
In-depth courses for all FME skill levels.
Explore our extensive collection of how-to guides and tutorials.
Log an issue with us. Our support team is ready to help.
