Shape the future of FME with your ideas
New and Open ideas are awaiting review or have been reviewed by our Product Management team, and are open for commenting and voting.
I propose adding DuckDB as the underlying database engine in the Inline Querier transformer for the following reasons:1. Performance Benefits:Columnar Storage: DuckDB’s columnar storage model allows for much faster query execution, particularly on large datasets, as it optimises for analytical queries. This would result in quicker workflows, especially for FME users dealing with substantial data volumes. Vectorised Execution: DuckDB uses a vectorised execution model that can process chunks of data more efficiently than the row-wise approach used in SQLite. This would significantly enhance performance in complex querying scenarios.2. Better Handling of Analytical Workloads:DuckDB is designed with analytical workloads in mind, making it more suitable for data processing tasks typically performed in FME. This switch would provide users with more efficient query processing when handling large geospatial datasets.3. Increased Flexibility with Data Types:DuckDB offers more advanced data types and functions, which would extend the flexibility of the Inline Querier transformer. This would enable more sophisticated transformations and analytics directly within FME, reducing the need for external tools.4. Seamless Integration:SQL Syntax Compatibility: DuckDB supports standard SQL queries similar to SQLite, ensuring a relatively smooth transition with minimal changes to existing workflows. Users will benefit from enhanced performance without needing to extensively rework their SQL queries.5. Broader Ecosystem Support:DuckDB has strong integrations with modern data ecosystems (e.g., Python, R, Parquet), which can open doors for future FME integrations and expand the functionality of the Inline Querier.6. Open-Source, Actively Developed:Like SQLite, DuckDB is open-source but has a much more active development roadmap focused on big data analytics. This would future-proof the Inline Querier transformer by providing new features and optimisations more regularly.Finally the ability to create macros in duckdb sql and refer to these inline could be very useful indeed, also the native spatial support would really help!
(1) Show the assumed date and time of the NEXT run (I know it may be delayed because of lack of free engines) ; (2) show if the job has not started at defined time (because of engine unavailability); (3) if possible, show the running time of latest job run. All these would greatly improve the usability of the scheduler when using FME Servers as an ETL machine. Thank you !
Allow generation of tokens via a REST endpoint by supplying username/password/expiry as parameters. This is similar to the now deprecated fme token service. When trying to automate some workflows that include FME Server jobs through scripting (e.g. using Powershell), currently the only recommended way to generate a token for REST API calls is to manually create an API token via the web interface. This doesn't fit well with automation and also involves storing/using a long-lived token, which is potentially insecure. The only other documented way to generate a token is to first use Basic Authentication, before creating a token, which is definitely insecure as user/password are sent in plain text. If there was a "generateToken" endpoint with user/password/expiry parameters that supported HTTPS and POST only (similar to the ArcGIS Server approach) this would be a secure way of generating a short-lived token for subsequent API calls.
It would be useful to have a list of dependencies for objects within FME Server. The example below focuses on workspace relationships and finds all the Repositories, Automations, and Schedules that are associated with a workspace and displays the report in the dashboards. This workspace works on all versions 2019.2.1.0 (19813) and higher and uses the REST API calls to query FME Server.What dependencies would you like to see? How do you want them displayed?How to use:1) Update the FME Server Web Connection parameter to use a connection to your server.2) Publish to FME Server3) Choose if you want a report on a specific workspace, or if you want all the workspace run with the parameter value "All"4) Run workspace5) View report from Dashboards
When experimenting with spatial data and comparing the effects of various processes we often end up with numerous (say 5 plus) Views in the Data Inspector. This is because we want to see the difference in geometries and attributes associated with the processes and analyse the results.Viewing the results at full extent is not always acceptable and there is a need to zoom into the same extent in each View to carry out the comparisons.An improvement within the Data Inspector would be the ability to nominate one View as the 'lead View' and have all other Views automatically adjust to the same spatial extent. Obviously with the ability to turn this function on and off.
I often put Terminator nodes on joins that are not expected to fail and TestFilter nodes meant to be exhaustive as kind of assertions to make sure my data makes sense. However, if these assumptions start breaking, they usually break for more than one feature, and if that happens, debugging this once it’s already been pushed to FME Flow becomes a game of whack-a-mole (with the reported error actually hiding a dozen more), unless we modify the flow in order to debug the error.I do have custom solutions to buffer errors in some cases, but getting this to work is complex and requires a lot of carefully arranged transformers for each potential termnator.(ProritySwitch is used to ensure no features can come through until all errors have been handled. It’s a custom transformer, more info here) It would be conveinient if Terminators could be configured to buffer all errors instead of terminating on the first one, to allow outputting and formatting all errors at the same time.
Just a new tool in Data Inspector "View" Menu : "close all views"
If a query in SQLCreator does not return any records it is not possible to terminate the workspace because there is no output.In SQLExecutor there is an output port 'Initiator' with the number of records in '_matched_records' which can be tested.Is it possible to add an 'initiator' output port in SQLCreator to make it more usefull,
FME Workspace Scheduler allows you to skip a job if it's already in progress. However, if you have a workbench that is triggered by both a schedule and an automation then the same workbench could potentially run at the same time (one on an engine and one on a dynamic engine). In general, I'd like to be able to have a setting at the workbench level that ensures that the workbench itself will never be run simultaneously. Right now we have an automation that redirects workbenches to dynamic engines but it's resulting in db locks and other issues.
The RabbitMQConnector works great and can be hosted as a Stream from Flow/Server. While we do not have any technical issues with that deployment, it does require a dedicated engine. So, whenever we need to build a new RabbitMQ listener (and this would be true for other messaging platforms as well) we need to license a new engine even if we don’t really need a dedicated one.Launching through an automation trigger we could just process the message on any available engine but not tie it up indefinitely.Thanks for the consideration.
While working for short time in Informatica PC I notice a neat feature there. From each transformer one could track back and forward (or booth) every attribute and therefore know from where it came and where it goes. It was really helpful on mappings done by somebody else.
I use the Revit reader quite a bit now, it's a big help, much appreciated.But some minor hurdles (from 2020.0):I don't see a way how to read the "angle to true north" for survey and origin points:I would like to query all available coordinate systems in a Revit file, not just the active one:FME's "SurveyPoint" feature is a bit confusing: In most of my cases, FME's "SurveyPoint" is actually Revit's "Origin Point", just multiplied with -1?Since Revit files are always in feet, a parameter to scale everything into non feet units would be nice. I doubt most people are aware of this...
GLTF has extensions which allows for the definition of attributes/properties. The EXT_mesh_features extension allows that geometries in a gltf can be given a specific ID.https://github.com/CesiumGS/glTF/tree/3d-tiles-next/extensions/2.0/Vendor/EXT_mesh_featuresTaking if further the ID can be linked to defined schemas with the EXT_structureal_metadata extension.The structural metadata extensionhttps://github.com/CesiumGS/glTF/tree/3d-tiles-next/extensions/2.0/Vendor/EXT_structural_metadataBoth of these are used by the new 3DTiles 1.1 spec - adding these extensions would be required for 3DTiles 1.1 support. In the case the 3DTiles 1.1 support does get added it only makes sense the the GLTF writer be improved to also support this.
To integrate FME, and make it able to communicate with Microsoft services it would help if FME goes trough the Publisher Verification process for there integrations.Our IT department is seeing risks in accepting App registrations for Apps that are not verified by Microsoft.The risk they point out is described here: https://pushsecurity.com/blog/how-attackers-compromise-azure-organizations-through-saas-apps/At this moment we would like FME to be able to e-mail using Microsoft Email Service: Currently the following warning is given:
On FME Flow, it can be very difficult to track down what Web Connection or Database connection is being used in a workflow without downloading it. When a web and database connection is used, we should log its name.
Please update GTrans in FME. I frequently transform coordinates between systems and have recently learned that there is a new version of GTrans. It would really help my colleagues and I if the new version was added as a transformer to FME (or the old version would be updated). As of now I have to utilize a pythonCaller and have a bunch of dependencies to databases and trasformation libraries which makes sharing and versioning my work difficult. GTrans is a fantastic resource for coordinate transformation and the new version supports transformations to MGRS directly from Sweref99TM as well as GeoRef which we had to calculate by hand earlier. This would be an amazing addition to an otherwise great software.
Expose sde version for Named ESRI SDE connections in Readers or FeatureReader so can be a parameter at runtime.Unless I am missing it, the only way to parameterize sde version at runtime is to use embedded connections. It would be nice to have this option for named SDE ESRI connections, as making a named sde connection every possible sde version is not practical.
Or ‘list slicing’ of course. For example this list with element positions also as values for illustration:List[0,1,2,3,4]A following AttributeManager wants to extract:Attribute1 = List{0} so would retrieve the value of ‘0’Attribute2 = List{1} would retrieve ‘1’Attribute3 = List{2:} or {List{2:5} or List{2:-1} would retrieve ‘2,3,4’Doing this without python would save time and also those without python skills would benefit. I am aware of the ListSlicer and @takashi could correct me if I’m wrong but it doesn’t quite seem to achieve this if the element range is open-ended or goes from e.g. element 2 to the end of the list. Maybe it’s just an enhancement to that transformer?A ListIndexRetriever type transformer might be a good way to do this, or bake it into attribute transformers in addition to the current single-element functionality.
As I understand it, the Lenient option states that two polygons are equal if Within(A, B) and Within(B, A). That is, A and B occupy the same space. However, I think that an option is missing that, in addition to checking that they occupy the same space, also checks that they have the same number of vertices even if their order may be different.In my case it is important to perform topological quality control of layers such as land covers. The addition of a new vertex to a polygon, even if it is collinear, is a new version of the polygon geometry.
Context:The FME Flow Gallery App is an excellent tool for presenting multiple workflows to non-FME users, but managing numerous applications can become cumbersome. Enhancements to interface and navigation could greatly improve the user experience. Proposed Improvements:Grouping of Applications: Introduce the ability to group related applications within the Gallery App. This would reduce clutter and make it easier to navigate when dealing with a large number of applications, eliminating the need to create multiple Gallery Apps. Search Bar Functionality: Add a search bar to quickly locate specific applications within the Gallery App, improving user efficiency and experience. Custom Map Interface: Allow the map parameter (e.g., GeometryReplacer) to display directly within the application interface (e.g., on the side of the screen). This would eliminate the need to click and open a separate map, making the interface more intuitive. Data Downloads in Automation Applications: Provide a mechanism for users to download data directly from Automation Applications, such as outputs generated by writers. This would extend the functionality of Automation Applications for data delivery workflows. Improved Text Formatting in Application Titles: Fix issues with letters breaking off from words in application titles, ensuring proper text alignment and readability.
As a workspace owner, I’d like to share the workspace with other users. However, I’d also like to see anytime another user runs the workspace I created. When sharing the workspace, I’d like to have the option to view all logs / automations / workspace apps associated with the workspace that users may create.
Add precision setting to all transformers that output dimensions or metric coordinates.To negate the need to add AttributeRounders after (e.g. LengthCalculator) include a precision setting option in the initial transformer.A default setting of 3 (mm) should suffice for most operations.This setting could then be defined as a user parameter for use throughout a workspace. A search shows discussions around precision dating back a decade ago, so I may be late to the party here.
Been having the issue of having to make a larger amount of requests covering smaller areas, instead of the entire extent I’m interested in, from the OGC WMS reader because the HTTP request times out. It would be very helpful to be able to specify the timeout length.
As 3D BIM become more popular in use i would like a way to check the ifc export against a ids file. I am guessing a similar method to Using XML to filter IFC could be used, but maybe a standardized way of doing this quality control could be achieved. Maybe the control could be integrated to the IFC reader, or as a separate transformer.
This is a XML-based system. More info can be found here: https://transmodel-cen.eu/index.php/netex/
In-depth courses for all FME skill levels.
Explore our extensive collection of how-to guides and tutorials.
Log an issue with us. Our support team is ready to help.
