FME Transformer most in need of some TLC?

Somehow I can't get my head around the parameters of the Counter. So I think it could use some TLC, or even a revamp.

have you seen the BetterSystemCaller from ​@alexbiz 

But I 100% agree with you on the SystemCaller and also the Counter too ​@geomancer. 

 

I have not, thanks for the tip! I would still like to see something that could stream output back to the FME log while the subprocesses is running :-)

Interesting topic, I had some thoughts about the AttributeFilter. Most often I only need 5 output ports for example becasue of 5 unique attribute values in the column Im interested in.

Would be cool to include an option to switch off the output ports Empty, Missing, Null and Unfiltered

SchemaMapper for me. I have to consult the help every time i use it to set it up, then nine times out of ten, I find out that it won’t actually do one key thing and i give up and write something in python instead.

Very relatable! I haven’t used it for many years, I believe. I tend to bump into its limitations too quickly.

I like this question.

Yes, I would like the AttributeFilter to remove the <Emtpy>, <Missing>, <Null> (with option to activate). Keep the <Unfiltered> because that's what it is. All features that did not match the other ‘tests’.

And move the <Unfiltered> to the bottom of the list (below the Attribute values).

The Counter should have Count Scope: Local as default.
The Sampler should have Sampling Type: First N Features as default
 

Add AttributeCreator functions to the Creator.

Make a GeometryCreator with the Geometry Source, Geometry Object, Coordinate List and Coordinate System parameters.

My brain does not like that a Creator can make geometry but no attributes and no other transformers can make geometry like the Creator can.

Give the PythonCaller basic script error feedback so I don’t need to run an entire workbench for the error invalid syntax.

Remove milliseconds from the %S option in the datetimeconverter or add a seconds only option.

Change the Type parameter to Output Format in the DateTimeStamper. Maybe add a Group Processing on the DateTimeStamper, this would improve run time enormous when stamping thousands of features with just the current run time. Now I use a VariableSetter and VariableRetriever to prevent FME to give every feature their own millisecond datetime.

Give the HTTPCaller: Query String Parameters, Multipart Upload and Headers box some TLC so it’s easy to add multiple headers or move upload body parameters from/to headers. Maybe add the option to import a PostMan script?

Just some things that come to mind.

You can set Attributes in a Creator

👆 this !

It’s easy enough to change your Default Set, my Sampler opens with First N Features as default.

I use the Presets option a lot to create custom defaults for various transformers.

 

My brain does not like that a Creator can make geometry but no attributes and no other transformers can make geometry like the Creator can.

 

You can set Attributes in a Creator

 

Oh, thanks. I did find this option and deleted it from memory. But I don’t think my colleagues would like me using this option. Wow. I can even hide CoordinateExtractors, DateTimeStampers etc. in AttributeRemovers never to be found:

I guess the purple arrow gives a hint.

Oh, thanks. I did find this option and deleted it from memory. But I don’t think my colleagues would like me using this option. Wow. I can even hide CoordinateExtractors, DateTimeStampers etc. in AttributeRemovers never to be found:

I guess the purple arrow gives a hint.

Devious renaming, I’m loving it :-D

Adding an option to the XMLFragmenter to keep the fragment in UTF-8, or respect the original encoding.

At the moment, the encoding in fragments from the XMLFragmenter is always set to UTF-16, causing the XMLXQueryExtractor to fail. This can be solved with a XMLFormatter upstream of the XMLXQueryExtractor tho, but it has led to some frustrating moments in the past.

I didn’t know about this one.

My wish in that regard is that both the JSON and XMLFragmenters would have an option to not preserve a fully copy of the original document on each fragment, which can really impact memory consumption when large documents are fragmented into many features.

Just remembered another one.

I’d love to see an option for Return Format in the JSON Extractor to allow return of a List Attribute, matching the option in the HTMLExtractor

Yeah I would vote for this one as well.

 

My brain does not like that a Creator can make geometry but no attributes and no other transformers can make geometry like the Creator can.

 

You can set Attributes in a Creator

 

 

Oh, thanks. I did find this option and deleted it from memory. But I don’t think my colleagues would like me using this option. Wow. I can even hide CoordinateExtractors, DateTimeStampers etc. in AttributeRemovers never to be found:

I guess the purple arrow gives a hint.

And this is why I’m working very hard on the WorkspaceFixer3000 😄

I hope it’s not able to detect these shenanigans:

​@geomancer ​@virtualcitymatt What is it about the counter that you can’t get your head around? I’m just curious as whilst some of the configurations aren’t often used (as they are quite advanced) it makes sense to me 🤔🤔

I hope it’s not able to detect these shenanigans:

 

Unfortunately not (yet? I don’t think I can get the layout of the workspace out of the Workspace reader and I don’t really want to dive in to the underlying XML), but this violates so many Best Practices…

As I'm not exactly sure what ​@geomancer is encountering, I suspect…

Adding 2 Counters in your workbench without changing settings gives unexpected results. When you know about the default Global settings and in many cases have no need to use that it get’s better.

I guess it’s not too difficult, but I think most use cases are using it to create an simple unique temp ID. Often I just want to count an attribute. Changing the default settings from Global to local makes the counter name disappear which was not the case in earlier versions of FME, why can’t I just set a count domain for local counts - I often get confused as to why it gets greyed out and it takes me a while to remember the group-by here is actually what I want - but should I make it blocking then? i don’t want that. I think most of the confusion came when trying to upgrade transformer versions a few versions back. 
I think usually leaving it with the default settings it usually fine but I’ve been burned in the past with reusing the same counter name. So now I always switch to local to just be on the safe side, but then I have to use group-by. So maybe I just need to go back to global.

Local counts where you can pick and attribute name to count I think just makes the most sense for most cases, this should just be the default setting but it’s not even available anymore.    




 

Oh yea, true. Setting it to global and then use the domain Counter_1_@Value(Color) so it counts all of a certain color is a very common usecase and makes it not blocking. But you have to remember to combine the transformer unique name and the attributename to prevent unwanted counts somewhere else in your workbench. While it could just be a local counter with a Group By that would not be blocking.

​@fmelizard Think this is becoming a tread with much low hanging fruit. Next release could be: FME is not running 20x faster. You are building the workbench 10x faster, because we tweaked the transformers.

Or… We removed the AI Assistant, FME is self-explaining.

That reminds me: Change the Regex Editor. Look at regexr.com for inspiration. Add a library of Regex favorites. Maybe even community favorites? Show recently used regex.

Sometimes I want Group Processing in these transformers - i.e. would like to choose “When Group Changes (Advanced)” in order to improve performance.

• LineCombiner
• Matcher

Remove milliseconds from the %S option in the datetimeconverter or add a seconds only option.

There is a datetime rounding function to help there, but I agree it would be nice to have a flag that was integer seconds only.

Somehow I can't get my head around the parameters of the Counter

Is that because of the local/global flag? I agree, the terms don’t make a lot of sense (what does “local” mean in FME). Can you think of better ones to represent this transformer/all Counter transformers?

it takes me a while to remember the group-by here is actually what I want - but should I make it blocking then? 

I don’t think that setting a group-by makes it more blocking than a local domain name. After all, a domain still would need all of the data to work properly. Wouldn’t it?

My wish in that regard is that both the JSON and XMLFragmenters would have an option to not preserve a fully copy of the original document on each fragment

Ways for authors to optimize memory are a lot less clear than in the “old days”, because of things like Feature Tables, but it makes sense to me that any transformer (or reader) that can process an attribute should have an option to consume it completely.

All of these are great comments though. We’ll make sure that the product managers see them. Personally, I love these smaller ideas that can have an impact much greater than their size.