Is there a way to expose attributes based on type (DateTime)

Could you create an idea from this topic:

I would like a BulkAttributeExposer with options to use a wildcard or expose by data type.

You can type-cast your attributes to string inside the JSONTemplater, if that helps you:

{ 
    "updated_at" : xs:string( fme:get-attribute("my_date_attribute") ) 
} 

Otherwise you could use a PythonCaller to loop over all the incoming attributes and re-create them as strings.

David

You could use a schemareader , select the date_type attributes by tester.

Then use a merger on the data to select them to be reformatted.

A attributeexploder could also be used to to this.

I don't see how you could do this without a script of some sort. The AttributeExploder and a ListConcatenator might give you the list of attribute names. Then copy/paste your DateFormatter into a text editor and you'll see the underlying scripting which you could recreate in a TCLCaller and adapt to your needs. That's how I would start to go about this. Hope it helps!

Mark

I will look into this thank you,

@marko did you get this resolved? If not I might have done something similar this week that could help.

It doesn't expose the fields but creates a parameter that I can use in a DateFormatter transformer. Issue for you might be if the input files are different formats it could get unwieldy. Also in my example I could detect the Date attribute as it was known data type. If you're reading a CSV for example this wouldn't help as the data type is unknown.

See https://knowledge.safe.com/questions/24743/format-...

HI @mark_1spatial, thanks of the reply and tip. Looks like a great solution from first glimpse, although are you using a python start up script or using a caller? I'm not sure if a start up will work whilst using a feature reader in generic mode?

Thanks for the reply I will look into it. Presently I am just exposing the values that I know to be SQL date fields, a quick sql query of the sys tables showed me which ones, and then once exposed in the featurereader I can select them in the dateformatter. Its not very elegant but works!

Using it in Python Scripted Private Parameter and a standard Reader not a FeatureReader but should work if you configure it to read the same data. If it iscoming from a database should be doable.

Very interesting problem.

At the root of it is...what is known when we design the workspace, and what is known when we run it.

When we design, if it is meant to be a dynamic workspace that knows not of schema, then by definition, we know not the schema. So attribute exposing isn't really possible. The best we could do is give you an "import" option on the AttributeExposer where you'd feed it all the possible input you'd ever imagine you'd encounter, and then from that you'd pick the ones you wanted "exposed" so you could work with them. But alas you'd now nolonger have a dynamic workspace, because any previously unseen data would potentially not have exposed what you wanted.

So let's explore how to solve this nicely with FME 2016.

I'll attach a workspace -- feed any data that has schema that has DateTimes by configuring the FeatureReader appropriately. And we'll "mangle" those date time fields, very efficiently, by the end of the workflow. Leaving all other attributes alone. And never knowing anything about the input data "statically" in the workspace.

The trick is that a) we know the Schema always comes out first from the FeatureReader (this was by design, for things like this). b) we have "GlobalVariables" that we can set and reset that can communicate from one "stream" to another. c) We have an AttributeExploder to blows out features into one-feature-per-attribute, so we can work with Attributes generically then. d) we can reassemble things with the Aggregator.

I included a Decelerator at the end only to prove that no blocking is going on.

Enjoy. I look forward to feedback. @Mark2AtSafe -- we may want a knowledge type article on this technique. @mark_1spatial @marko @takashi @erik_jan @david_r Your comments particularly welcomed.dynamicdatereformat.fmw

Very interesting problem.

At the root of it is...what is known when we design the workspace, and what is known when we run it.

When we design, if it is meant to be a dynamic workspace that knows not of schema, then by definition, we know not the schema. So attribute exposing isn't really possible. The best we could do is give you an "import" option on the AttributeExposer where you'd feed it all the possible input you'd ever imagine you'd encounter, and then from that you'd pick the ones you wanted "exposed" so you could work with them. But alas you'd now nolonger have a dynamic workspace, because any previously unseen data would potentially not have exposed what you wanted.

So let's explore how to solve this nicely with FME 2016.

I'll attach a workspace -- feed any data that has schema that has DateTimes by configuring the FeatureReader appropriately. And we'll "mangle" those date time fields, very efficiently, by the end of the workflow. Leaving all other attributes alone. And never knowing anything about the input data "statically" in the workspace.

The trick is that a) we know the Schema always comes out first from the FeatureReader (this was by design, for things like this). b) we have "GlobalVariables" that we can set and reset that can communicate from one "stream" to another. c) We have an AttributeExploder to blows out features into one-feature-per-attribute, so we can work with Attributes generically then. d) we can reassemble things with the Aggregator.

I included a Decelerator at the end only to prove that no blocking is going on.

Enjoy. I look forward to feedback. @Mark2AtSafe -- we may want a knowledge type article on this technique. @mark_1spatial @marko @takashi @erik_jan @david_r Your comments particularly welcomed.dynamicdatereformat.fmw

Very interesting problem.

At the root of it is...what is known when we design the workspace, and what is known when we run it.

When we design, if it is meant to be a dynamic workspace that knows not of schema, then by definition, we know not the schema. So attribute exposing isn't really possible. The best we could do is give you an "import" option on the AttributeExposer where you'd feed it all the possible input you'd ever imagine you'd encounter, and then from that you'd pick the ones you wanted "exposed" so you could work with them. But alas you'd now nolonger have a dynamic workspace, because any previously unseen data would potentially not have exposed what you wanted.

So let's explore how to solve this nicely with FME 2016.

I'll attach a workspace -- feed any data that has schema that has DateTimes by configuring the FeatureReader appropriately. And we'll "mangle" those date time fields, very efficiently, by the end of the workflow. Leaving all other attributes alone. And never knowing anything about the input data "statically" in the workspace.

The trick is that a) we know the Schema always comes out first from the FeatureReader (this was by design, for things like this). b) we have "GlobalVariables" that we can set and reset that can communicate from one "stream" to another. c) We have an AttributeExploder to blows out features into one-feature-per-attribute, so we can work with Attributes generically then. d) we can reassemble things with the Aggregator.

I included a Decelerator at the end only to prove that no blocking is going on.

Enjoy. I look forward to feedback. @Mark2AtSafe -- we may want a knowledge type article on this technique. @mark_1spatial @marko @takashi @erik_jan @david_r Your comments particularly welcomed.dynamicdatereformat.fmw

Hi @daleatsafe, thanks for sharing the technique. I think it is commonly applicable to manipulate attributes depending on their native data type not only for date. However, seems the AttributeExploder could cause decreasing the performance especially if the number of feature attributes was very large.

I was thinking of a solution with Tcl for @mark_1spatial's question, like this: b15575-dynamicdatereformat-tcl.zip

The concept is the same as Dale's workspace. I didn't post it because I found that Mark's Python solution was more generic and flexible.

I think it would become easier to implement this concept if these abilities would be added to FME.

• A transformer that extracts attribute names which match user specified condition (data type, regex, prefix etc.) from the schema feature, as a comma-separated string or list.
• A parameter type that accepts an attribute storing attribute names (csv or list). If transformers such as DateFormatter could configure target attributes dynamically via this type parameter, the implementation would be much simpler.

@marko, this one: b15575-dynamicdatereformat.zip works with 2015.1.3+. It's different from Dale's but the concept is the same. FYI