Skip to main content

Open ideas have been reviewed by our Product Management and are open for commenting and voting.

Filter by idea status

Filter by product area

1092 Ideas

davisblack
Contributor
davisblackContributor

FME Flow User Role Security EnhancementsOpen

The following improvements to Role Based Security would be useful:1.Increased Granularity for Job Viewing and Job Management PermissionsI need the ability to allow users to see specific other user’s jobs and logs rather than all jobs and logs. We use different service accounts for different enterprise projects and currently I am having to share all jobs/logs with users so they can monitor their processes. Since not all groups need to see each other’s jobs this is not the best solution. (Thank you Safe for new search options in 2019 as they slightly help with this issue.)Likewise, in the current Role system being able to view all jobs requires job management permissions which let users cancel jobs in the queue or terminate running jobs. This puts me in a difficult position regarding how we implement SOX compliance. It is preferred users be locked out from managing running production processes. If the ability to view all logs but not manage jobs was made available that would be a move in a positive direction. If the option to say kill queued jobs but not running jobs was an option that may also be useful.2. Additional Options on Database ConnectionsHaving a Read option alongside the Full Control option for database connections would be excellent. I am seeing issues when I grant users access to connections without giving them full control over a connection. Since we have many processes using the same named database connection it is not ideal to grant users management access since one incorrect change by any user with access could disrupt jobs for all users using the connection. Something similar to how the Notification items are broken out would be amazing.3. Automatic Content Sharing with Administrator AccountsIt would be helpful if FME Server automatically shared user created content with members of the Super User role. If this could be an option for the FME Admin role that would also be appreciated. I have a plethora of users creating content and when they ask for help it is cumbersome to have to share their content with myself and my fellow Admins before I can help troubleshoot.

nic_ran
Contributor
nic_ranContributor

Add fixed-schema format attributes to Generic output port of FeatureReader transformerOpen

Hi,some formats (such as the Text File reader) have fixed schemas that will never change. In the case of the Text File reader the only attribute in the schema is the text_line_data attribute.For these fixed-schema formats, I would like to request an enhancement to the FeatureReader to have the schema automatically exposed, even when the Generic output port is configured. Currently, the schema attributes are only exposed for named output ports.For context, I refer to the following Community question: And to my own support question (to which James Cheng provided a very comprehensive reply):https://support.safe.com/hc/en-us/requests/64419I have reproduced James’ reply below, as it so clearly explains the situation:Thank you for raising this. I understand the frustration as I share the same thoughts and can see why you'd expect the text_line_data attribute to just be available when reading a Text File through the FeatureReader. The behavior you're seeing comes down to how the FeatureReader's generic output port works. This port is a catch-all that merges everything into a single stream, designed to handle any format and feature types with varying schemas. Because of this, it doesn't commit to any schema at authoring time, it doesn't make assumptions about which feature types or attributes will be present, so all attributes come through unexposed by default, even for predictable formats. By contrast, when you use a named output port (or a standalone Text File Reader), the schema is known at authoring time, so the attributes like text_line_data are automatically exposed. In essence, by using the Generic port, the trade-off is flexibility over schema awareness, it can handle anything but at the cost of not pre-committing to any specific attributes. However, as you've rightly pointed out though, the Text File Reader always produces a predictable fixed schema with a single attribute, text_line_data, no matter which files/feature types are being read. So there's a reasonable argument that the FeatureReader could recognize this and auto-expose it on the Generic Port since there's no ambiguity what the output will be. Currently, the Generic port doesn't differentiate between fixed schema and variable schema formats as it applies the same "unexposed by default" behavior uniformly across the board. That said, I think this is a genuinely good candidate for a product enhancement for the generic port. I'd encourage you to post this as an Idea on the FME Community, specifically requesting that the FeatureReader auto expose known attributes for fixed schema formats like Text Files on the Generic port and describing your use case. Community support helps the development team gauge interest and prioritize ideas for future releases, and we can link this ticket to the Idea to keep it on their radar. In the meantime, it's worth noting for this specific format, the feature types are all fixed to text_line, so using the default One per Feature Type output port setting will also route all text file features through a single text_line named port, similarly to the Generic Port, which will expose the text_line_data attribute automatically. If your workflow requires the Generic port, then the only available options are manually setting the Attributes to Expose parameter on the FeatureReader or placing an AttributeExposer downstream. regards,Nic

vlroyrenn
Enthusiast
vlroyrennEnthusiast

Fence and/or Semaphore transformer to control the flow of input dataOpen

The problem The documentation for FeatureMerger mentions a "Suppliers First" mode that can reportedly be very beneficial to performance (and, I would imagine, crucial for Streams), but comes with the constraint that all suppliers must have arrived before the first requestor comes in. To my knowledge, there is currently no way in FME of upholding that guarantee in a reliable manner. The timing between Readers, Creators, FeatureReaders and SQLExecutors is not something I can claim to understand, and the completion order can change based on whether caching is turned on or not. This is already troublesome when editing workbenches, but it can be expecially problematic inside custom transformers, where you don't have control over the delivery order of features in your input ports. This isn't merely an issue with FeatureMerger; it's going to be a problem at any time where you rely on input ordering for a transformer to work properly, with specific attention given to Streams and Feature tables. If I have a PythonCaller that configures itself based on data coming from TransformerA before it's ready to accept data from TransformerB, there isn't a lot of options for reliably dealing with out-of-order input. Holding onto features is illegal when bulk mode support is advertized, so the PythonCaller must either opt out of it (which hurts downstream performance) to accumulate any features it's not ready to process until the configuration features have come in. Even then, it can't know when TransformerA has closed, so unless it only expects one configuration feature, it's dangerous to start processing before close() has been called. This might be clearer when considering the attached screenshot: Python_MapAttributes needs the output line from JsonTemplater in order to work with the data coming from MappedInputLines. If MappedInputLines starts sending features first, the PythonCaller can't do anything yet, and can only crash (undesirable) or start buffering features (illegal in Bulk mode). The solution The idea would be to have some sort of Semaphore transformer, something like a FeatureHolder, but with (at least?) two input ports: a "Priority" port, which lets features through normally, and a "Held" port which buffers features until the other port has closed and no more features can go through it. This would ensure that no feature from the "Priority" side can ever arrive before a "Held" feature, thus allowing workflow and transformer designers to guarantee feature ordering downstream without breaking bulk mode. Other relevent use cases One might also consider having a Terminator node which should stop the translation when an assertion or a join fail in some unexpected way, but should wait until every faulty feature has arrived to give proper context instead of immediately stopping at the first one. Bad features could be sent through the priority port and then the priority port routed to a terminator, so that no feature can be passed to the next step until it has been verified that none exist that would trip the Terminator. This would also allow the Terminator to be changed to wait for all features to have arrived before stopping the translation, instead of aborting at the first one (which currently makes sense, as the more you wait, the more you risk that downstream writers will have already started to write incomplete data).