Skip to main content

Hi all,

I've been given a bunch of GDBs where in a few cases the date field is held as a string. I need to loop over the columns and look over the columns to see if they contain the word DT or DATE and if so, format that date into an Oracle Readable date, also automatically updating the writer to match the date type.

Does anyone have any ideas how i can do this. So far i've got the regex. .*[d][a][t][e].*
.*[d][t].* and am thinking of using some sort of tester to test whether the column contains this in its name. If so i need to pass it to a DateFormatter merge back in with the rest of the dataset and pass to a writer which has automatically been configured to detect the change in datatype. This needs to be reasonably generic and work when looped over multiple GDBs,

I'm running FME desktop 2014

Any help would be much appreciated.

 

Thanks!

Hi meischris39,

I would use Python to fix this. Using a PythonCaller you can loop through all attributes and check if they contain a specific set of characters. If so, you can use some string manipulations or the datetime-library to convert to the correct date format. Below you can find a script to start with. If you can provide me some of the formats, I can try to implement them for you.

I do not know how you could automatically change the writer. You could of course use the 'automatic'-option for the attributes.

Good luck!

import fme
import fmeobjects

# Template Function interface:
def convertDate(feature):
    
    # Retrieve all attributenames of this feature
    attributes = feature.getAllAttributeNames()
   
    # loop through all attributes
    for cAttribute in attributes:
        
        # Make sure nothing is changed on the standard (hidden) fme variables
        if cAttributea0:4] == 'fme_':
            continue
        
        # Get the value of this attribute
        value = feature.getAttribute(cAttribute)

# Check if this attribute is a date
if 'DT' in value or 'DATE' in value:
# CHANGE THIS CODE TO CHANGE THE DATE TO THE CORRECT FORMAT
newValue = value
feature.setAttribute(cAttribute, newValue)  


     # Output this feature towards the Workbench
     self.pyoutput(feature)

Probably you need to write a script to find "*DATE*" and "*DT*" columns and change the date format, but I'm not sure why you need to do that.

FME readers translate a date value stored in a source dataset to a character string formatted in FME standard date/datetime format ("yyyymmdd" etc.). FME writers can convert the date string to an appropriate date value and write it into a destination dataset. Usually the translation will be performed automatically.

If the "*DATE*", "*DT*" columns are defined as date type field, the GDB reader translates the date value to "yyyymmdd". You don't need to change the format of date strings before writing into date type fields defined in the Oracle writer feature type.

In addition, there is no way to update writer setting automatically, but Dynamic Workflow may be what you are looking for. See this page to learn more: Tutorial: Dynamic Workflows


Try the AttributeExploder transformer. It will explode your attributes into individual features each with the name and value of a specific attribute. Then run the Tester on this data to look for "date" attributes (eg _attr_name contains "date").

Then turn that back into an attribute using an FMEFunctionCaller with

@SupplyAttributes(@Value(_attr_name),@Value(_attr_value))

.... then I'm not sure. You would somehow turn that into the correct date format (preferably inside that function caller), then use a FeatureMerger to get it back onto an unexploded set of data.

The whole thing would be dynamic too. But that's OK I think, because we're dealling with generic attribute names like _attr_name and _attr_value.

Hope this helps.

 


Try the AttributeExploder transformer. It will explode your attributes into individual features each with the name and value of a specific attribute. Then run the Tester on this data to look for "date" attributes (eg _attr_name contains "date").

Then turn that back into an attribute using an FMEFunctionCaller with

@SupplyAttributes(@Value(_attr_name),@Value(_attr_value))

.... then I'm not sure. You would somehow turn that into the correct date format (preferably inside that function caller), then use a FeatureMerger to get it back onto an unexploded set of data.

The whole thing would be dynamic too. But that's OK I think, because we're dealling with generic attribute names like _attr_name and _attr_value.

Hope this helps.

 

Ah great this looks great trying it now. In terms of the format it will just be the standard oracle date format that can be wrote.

The string is yyyymmdd ideally if we could validate the date at this point too to see whether it is actually a legitimate gregorian date that would be awesome.


Ah great this looks great trying it now. In terms of the format it will just be the standard oracle date format that can be wrote.

The string is yyyymmdd ideally if we could validate the date at this point too to see whether it is actually a legitimate gregorian date that would be awesome.

If you pass your features through the DateFormatter, you can specify a particular value for invalid dates. This makes it easy to filter them out later using e.g. a Tester.


If you pass your features through the DateFormatter, you can specify a particular value for invalid dates. This makes it easy to filter them out later using e.g. a Tester.

Yeah thats all working great now, problem is i can't workout what the FMEFunctionCaller is doing, if anything at all? The output following it from an inspector just shows a blank field.

After this, i still have the problem that the Inspector is picking up the value as a String and not a date.

Thanks for all the help so far guys!


Probably you need to write a script to find "*DATE*" and "*DT*" columns and change the date format, but I'm not sure why you need to do that.

FME readers translate a date value stored in a source dataset to a character string formatted in FME standard date/datetime format ("yyyymmdd" etc.). FME writers can convert the date string to an appropriate date value and write it into a destination dataset. Usually the translation will be performed automatically.

If the "*DATE*", "*DT*" columns are defined as date type field, the GDB reader translates the date value to "yyyymmdd". You don't need to change the format of date strings before writing into date type fields defined in the Oracle writer feature type.

In addition, there is no way to update writer setting automatically, but Dynamic Workflow may be what you are looking for. See this page to learn more: Tutorial: Dynamic Workflows

It does sound like the issue is less about the actual data (if it is YYYYMMDD stored as a string in the original Geodatabase, then it will write out to Oracle date fields just fine). Instead, it sounds like the issue is the Oracle table definitions themselves -- you need the fields that have *DATE* or *DT* in them to be defined as Date columns and not string columns. As Takashi points out, this can be done via Dynamic Workflows in modern FMEs. In FME 2014 I'd be more tempted to do some kind of post-process to alter the tables by using the SchemaReader as the source, pointing to Oracle, then looking at the attributes on the schema and ALTERing any tables that had these. Maybe I'd ust Python ala the post above to create teh SQL Statement, and then execute it via SQLExecutor.


Reply