Skip to main content
  • EN

I am writing a custom writer that has a special date format. I would like to detect when a string has the native fme_date, fme_time, or fme_datetime format and convert the internal YYYYMMDD etc. format to my internal format. I am using the C++ SDK, and am unable to find any function that tells me what native format is being used. The best I can find is getAttributeType() and getAttributeEncoding(), but these functions only return FME_ATTR_ENCODED_STRING (an FME_AttributeType enumeration), and 'utf-8'. For a regular string attribute that does not have the 'date' format in the reader (I'm using the oracle reader and connecting it to my custom writer), the attribute type is FME_ATTR_STRING rather than encoded string. Is there any way to detect on the custom writer side what native FME format was used by the reader? And if so, are there convenient format conversion functions available in the SDK?

 

Using the DateTimeConverter transformer in the workbench is not an ideal solution for our use case since the tables can be huge and checking each and every attribute by hand would be painstaking. I would very much like to be able to detect the formats programmatically and convert formats as necessary. Thanks in advance for any guidance anybody can provide!

 

 

The following people have been involved in conversations of similar topics, and I'm hoping you guys might be able to help me. Thanks!

 

@mark_1spatial @Mark2AtSafe @takashi @david_r

Timestamps are still, as far as I know, treated as strings by FME, meaning that you'd need the reader schema feature to tell the native format (see e.g. the FeatureReader's "schema" output port).

Depending on your use case, I'd consider using the following algorithm:

  1. Check that attribute is of type string
  2. Check that attribute value has a length that could indicate a timestamp
  3. Try to parse the attribute value to a C++ time object (e.g. using strptime), catching any exceptions

If you get all the way to 3 without any exceptions, you could perhaps assume a timestamp?

Not perfect, but it might be enough for a lot of use cases.

Timestamps are still, as far as I know, treated as strings by FME, meaning that you'd need the reader schema feature to tell the native format (see e.g. the FeatureReader's "schema" output port).

Depending on your use case, I'd consider using the following algorithm:

  1. Check that attribute is of type string
  2. Check that attribute value has a length that could indicate a timestamp
  3. Try to parse the attribute value to a C++ time object (e.g. using strptime), catching any exceptions

If you get all the way to 3 without any exceptions, you could perhaps assume a timestamp?

Not perfect, but it might be enough for a lot of use cases.

Thanks for the prompt feedback! Your suggestion is a good back-up plan; but the unfortunate side effect is that it slows things down (when you have billions of rows to insert).

 

The reader schema feature that you mention, is there any way to access that from the writer? I have a local FMEFeature object created from the FMESession, and that class does not seem to have any function related to native formats. Am I confusing two completely different things?

 

 

 

Thanks for the prompt feedback! Your suggestion is a good back-up plan; but the unfortunate side effect is that it slows things down (when you have billions of rows to insert).

 

The reader schema feature that you mention, is there any way to access that from the writer? I have a local FMEFeature object created from the FMESession, and that class does not seem to have any function related to native formats. Am I confusing two completely different things?

 

 

 

I agree that the workaround isn't ideal, in particular because it's not very fast, as you've noticed.

 

There's a possibility that your writer can access the reader schema IF your writer is set to dynamic, but I've got not experience with that strategy. You may want to contact Safe support about this.

I'd suggest that your writer should already "know" what columns it expects will be dates. Its "DEF lines" (to use old terminology) should say what the types of each column are going to be, and then your writer should be able to know which ones are going to be dates. And so those columns would get the conversion treatment that @david_r suggests.

If you're reading from Oracle, the dates will be formatted in the FME internal format, and Oracle will have told the FME infrastructure that the column was a date, so if your format also says it supports dates in its metafile, then automatically it should be a date type when a workspace is generated.

Thanks, @daleatsafe! I was worried that not all readers would necessarily convert the date type values to the fme_date etc. formats. In one test workspace, I had a CSV reader and my custom writer. While my writer's attribute was set to date, the CSV reader itself was not. And I was able to connect the two without FME complaining. If I only expected/allowed the fme_date format, then the only way the test case would work was if I stuck a DateTimeConverter transformer in between. But, if I want the user to have the flexibility of using fme_date for their reader (either with a transformer or automatically, like you said Oracle does), as well as using our custom writer's special format (like the CSV file content did), then I need some logic in the writer that can catch both cases.

I'm trying to be as user friendly as possible, perhaps a little at the cost of the data conversion speed. I will have to weigh the pros and cons of each choice.

Thanks again for your feedback!

Reply


Community Stats

30,463
Posts
115,690
Replies
38,289
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.