Skip to main content
Question

Format GML Date For Dynamic Writer

M
Forum|alt.badge.img+2

I have a Dynamic Writer using Schema From Table referenced as Workspace Resource. Reading GML there is a date attribute with format as YYYY-MM-DD which is the Reader schema sees as xml_date type. There is a SchemaMapper process happening and the schema from table defines the target data type as fme_date. Problem is the target format wants the date formatted as YYYYMMDD (no dashes). If I format the dates using a DateFormatter transformer all is good. If not the features are dropped.

 

 

Rather than using a DateFormatter (where I need to specify all the data attributes) I'd prefer something a bit more generic. Was thinking a Python script to look at the data types of the attributes but there didn't seem any method within FMEObjects to do this. Next possibility is Python to check each attribute value to see if they are dates formatted YYYY-MM-DD and reformat.

 

 

All seems a bit clunky so thinking there is something obvious I've missed! Thanks.

11 replies

erik_jan
Contributor
Forum|alt.badge.img+17
  • erik_janContributor
  • Contributor
  • March 17, 2016

How about using a StringReplacer (that can be used on multiple attributes at once) and replacing the '-' by nothing?

M
Forum|alt.badge.img+2
  • mark_f
  • Author
  • March 17, 2016
erik_jan wrote:

How about using a StringReplacer (that can be used on multiple attributes at once) and replacing the '-' by nothing?

@erik_jan same as DateFormatter - I'd have to specify the attribute values I want to work with. I'd rather detect the date attributes and change where necessary.

erik_jan
Contributor
Forum|alt.badge.img+17
  • erik_janContributor
  • Contributor
  • March 17, 2016
mark_f wrote:

@erik_jan same as DateFormatter - I'd have to specify the attribute values I want to work with. I'd rather detect the date attributes and change where necessary.

@mark_1spatial Could you select all attributes and use regular expressions to change only those that match '\\d\\d\\d\\d-\\d\\d-\\d\\d' ?

M
Forum|alt.badge.img+2
  • mark_f
  • Author
  • March 17, 2016

Hello,

since updating to 2016.1 I have a problem with my sql statements to execute after translation. I need to update sequence values, because I am manually inserting data into serial column. However I am getting following error:

Error executing SQL command ('select setval('$(Schema).unip_property_attributes_id_seq', (select max(id) from $(Schema).unip_property_attributes))'): 'ERROR:  syntax error at or near "$"

LINE 1: ...property_attributes_id_seq', (select max(id) from $(Schema)....
                                                             ^

'
PostgreSQL Writer: SQL statement To Execute After Translation failed; see above for details

where unip_property_attributes_id_seq is the name of the sequence I am trying to modify (set value) and $(Schema) is a published parameter where the name of the schema is stored. Seems like after updating to 2016.1 there is some problem with addressing published parameters in sql commands after translation, because $(Schema) parameter used in sql executor seems to be working just fine (error points to $ character, which makes no sense to me). However what is mind boggling is that I have several very similar sql commands to execute after translation that were created in 2016.0 and work now even when I run the workspace in 2016.1, only this single freshly created sql does not work.

Here are the sql commands I am using, first one is giving errors, second one is working just fine:

FME_SQL_DELIMITER ;
select setval('$(Schema).unip_property_attributes_id_seq', (select max(id) from $(Schema).unip_property_attributes));



FME_SQL_DELIMITER ;
select setval('$(Schema).unip_object_id_seq', (select max(id) from $(Schema).unip_object));

Thanks for any help,

Radek

erik_jan
Contributor
Forum|alt.badge.img+17
  • erik_janContributor
  • Contributor
  • March 17, 2016

Mark, this issue got me thinking and resulted in posting a new idea:

add-option-to-act-on-all-attributes-to-transformer

mark2atsafe
Safer
Forum|alt.badge.img+43
  • mark2atsafeSafer
  • Safer
  • March 18, 2016
erik_jan wrote:

Mark, this issue got me thinking and resulted in posting a new idea:

add-option-to-act-on-all-attributes-to-transformer

Yes, I like that idea and was thinking the same. Not sure how feasible it will be, but I like it. I suspect if we added that it would only be on, say, the AttributeManager. I can't imagine us putting it on every transformer (though I could be wrong).

mark2atsafe
Safer
Forum|alt.badge.img+43
  • mark2atsafeSafer
  • Safer
  • March 18, 2016

The only non-Python way I can think of is to use the AttributeSplitter, then an AttributeCreator that creates a new attribute with the name @Value(_attr_name) and with the correct date format. I don't think you can use the same @Value trick inside the DateFormatter unfortunately.

There is the @CurrentAttribute function right now, that seems like it ought to be the right thing, but I don't quite think it is. Unless you can use it inside Python or an FMEFunctionCaller

M
Forum|alt.badge.img+2
  • mark_f
  • Author
  • March 22, 2016

I went down the Python route (see below) using the FMEObjects class FMEUniversalReader to read the schema, detect attributes with a native data type of XML_DATE, to create a Python Script Private Parameter which is used inside the DateFormatter. 

Credit to resident Python Jedi @takashi and @david_r for information in other posts!

import fmeobjects

# fetch parameters
master_xsd = FME_MacroValues['MASTER_XSD']
gml_file = FME_MacroValues['IN_GML']

# setup schema reader
directives = []
directives.append('RUNTIME_MACROS')
directives.append('XSD_DOC,"' + master_xsd + '"')
reader = fmeobjects.FMEUniversalReader('GML', False, directives)
 
# open dataset
# required parameters other than "REAL_FORMAT" are different
# depending on the real format.
# read "Mapping File Directives" section in the help documentation
# on the real format to learn more about the format specific parameters.

parameters = ['REAL_FORMAT', 'GML']
reader.open(gml_file, parameters)

dateattrlist = []
# read and process schema attributes into list
while True:
    schema = reader.readSchema()
    if schema == None:
        break
    for i, name in enumerate(schema.getSequencedAttributeNames()):
        nativeDataType = schema.getAttribute('*%s' % name)
        # look for date type
        if nativeDataType == 'xml_date':
          dateattrlist.append(name)
reader.close()

#use set to deduplicate list into parameter variable
dateattr = ''
for val in set(dateattrlist):
    dateattr = val + ',' + dateattr

# set the parameter
return dateattr
david_r
Evangelist
  • david_rEvangelist
  • March 23, 2016
mark_f wrote:

I went down the Python route (see below) using the FMEObjects class FMEUniversalReader to read the schema, detect attributes with a native data type of XML_DATE, to create a Python Script Private Parameter which is used inside the DateFormatter. 

Credit to resident Python Jedi @takashi and @david_r for information in other posts!

import fmeobjects

# fetch parameters
master_xsd = FME_MacroValues['MASTER_XSD']
gml_file = FME_MacroValues['IN_GML']

# setup schema reader
directives = []
directives.append('RUNTIME_MACROS')
directives.append('XSD_DOC,"' + master_xsd + '"')
reader = fmeobjects.FMEUniversalReader('GML', False, directives)
 
# open dataset
# required parameters other than "REAL_FORMAT" are different
# depending on the real format.
# read "Mapping File Directives" section in the help documentation
# on the real format to learn more about the format specific parameters.

parameters = ['REAL_FORMAT', 'GML']
reader.open(gml_file, parameters)

dateattrlist = []
# read and process schema attributes into list
while True:
    schema = reader.readSchema()
    if schema == None:
        break
    for i, name in enumerate(schema.getSequencedAttributeNames()):
        nativeDataType = schema.getAttribute('*%s' % name)
        # look for date type
        if nativeDataType == 'xml_date':
          dateattrlist.append(name)
reader.close()

#use set to deduplicate list into parameter variable
dateattr = ''
for val in set(dateattrlist):
    dateattr = val + ',' + dateattr

# set the parameter
return dateattr

That's an elegant solution, thanks for sharing!

takashi
Contributor
Forum|alt.badge.img+19
  • takashiContributor
  • Contributor
  • March 23, 2016
mark_f wrote:

I went down the Python route (see below) using the FMEObjects class FMEUniversalReader to read the schema, detect attributes with a native data type of XML_DATE, to create a Python Script Private Parameter which is used inside the DateFormatter. 

Credit to resident Python Jedi @takashi and @david_r for information in other posts!

import fmeobjects

# fetch parameters
master_xsd = FME_MacroValues['MASTER_XSD']
gml_file = FME_MacroValues['IN_GML']

# setup schema reader
directives = []
directives.append('RUNTIME_MACROS')
directives.append('XSD_DOC,"' + master_xsd + '"')
reader = fmeobjects.FMEUniversalReader('GML', False, directives)
 
# open dataset
# required parameters other than "REAL_FORMAT" are different
# depending on the real format.
# read "Mapping File Directives" section in the help documentation
# on the real format to learn more about the format specific parameters.

parameters = ['REAL_FORMAT', 'GML']
reader.open(gml_file, parameters)

dateattrlist = []
# read and process schema attributes into list
while True:
    schema = reader.readSchema()
    if schema == None:
        break
    for i, name in enumerate(schema.getSequencedAttributeNames()):
        nativeDataType = schema.getAttribute('*%s' % name)
        # look for date type
        if nativeDataType == 'xml_date':
          dateattrlist.append(name)
reader.close()

#use set to deduplicate list into parameter variable
dateattr = ''
for val in set(dateattrlist):
    dateattr = val + ',' + dateattr

# set the parameter
return dateattr

Beautiful! It's a smart solution to accomplish the goal. Thanks for sharing.

M
Forum|alt.badge.img+2
  • mark_f
  • Author
  • March 23, 2016
mark_f wrote:

I went down the Python route (see below) using the FMEObjects class FMEUniversalReader to read the schema, detect attributes with a native data type of XML_DATE, to create a Python Script Private Parameter which is used inside the DateFormatter. 

Credit to resident Python Jedi @takashi and @david_r for information in other posts!

import fmeobjects

# fetch parameters
master_xsd = FME_MacroValues['MASTER_XSD']
gml_file = FME_MacroValues['IN_GML']

# setup schema reader
directives = []
directives.append('RUNTIME_MACROS')
directives.append('XSD_DOC,"' + master_xsd + '"')
reader = fmeobjects.FMEUniversalReader('GML', False, directives)
 
# open dataset
# required parameters other than "REAL_FORMAT" are different
# depending on the real format.
# read "Mapping File Directives" section in the help documentation
# on the real format to learn more about the format specific parameters.

parameters = ['REAL_FORMAT', 'GML']
reader.open(gml_file, parameters)

dateattrlist = []
# read and process schema attributes into list
while True:
    schema = reader.readSchema()
    if schema == None:
        break
    for i, name in enumerate(schema.getSequencedAttributeNames()):
        nativeDataType = schema.getAttribute('*%s' % name)
        # look for date type
        if nativeDataType == 'xml_date':
          dateattrlist.append(name)
reader.close()

#use set to deduplicate list into parameter variable
dateattr = ''
for val in set(dateattrlist):
    dateattr = val + ',' + dateattr

# set the parameter
return dateattr

Praise from the Jedi - (insert blushes emoticon)

Reply


Helpful Members This Week

Community Stats

30,937
Posts
117,247
Replies
38,739
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.