Skip to main content
Released

Date Formatter to handle milliseconds

Related products:FME Form
  • November 30, 2015
  • 9 replies
  • 50 views
rylanatsafe
siennaatsafe
david_r
takashi
revesz
+6
  • rylanatsafe
    rylanatsafe
  • siennaatsafe
    siennaatsafe
  • david_r
    david_r
  • takashi
    takashi
  • revesz
    revesz
  • dr_ysg
  • t_hohenstrater
    t_hohenstrater
  • endest
    endest
  • roland.martin
    roland.martin
  • kim
  • martinludwig

I work on Internet of Things projects. We have lots of sensors that provide data that needs to be timestamped with milisecond accuracy so that we can see which combination of sensors recognize events and objects. I want to do correlation between data at a milisecond accurracy.

The issue is that I cannot directly convert the time information (milliseconds from epoc) to date with the DateFormatter, since the %s directive only takes integer values;

http://docs.safe.com/fme/html/FME_Transformers/FME_Transformers.htm#Transformers/dateformatter.htm

<strong>This post is closed to further activity.</strong><br /> It may be a question with a best answer, an implemented idea, or just a post needing no comment.<br /> If you have a follow-up or related question, please <a href="https://community.safe.com/topic/new">post a new question or idea</a>.<br /> If there is a genuine update to be made, please contact us and request that the post is reopened.

9 replies

david_r
Celebrity
  • December 2, 2015

I agree, I've had the same need.

For what it's worth, I ended up solving it using Python.


I agree too and would 'enhance' the Topic.

a Date/Time Format with fractional second is needed and should applicable to all transformers which are using Date-Formats (TimeStamper, DateFormatter ...).

Oracle's TIME and TIMESTAMP Datatype is defined by

TIME [(fractional_second_precision)]

I've got to Import CSV-Data with Informations like "17Jul1990:11:22:33.4567", driven trough the DateFormatter with 'Source Date Format' = "%d%b%Y:%T" and

'Destination Date Format' = "%Y%m%d%H%M%S" Ends up with, suprise suprise,

"19900717112233". The fractional seconds are lost!

You've got extract them before going through DateFormatter, and You have to do this for each Time-attribut of the Feature. The DateFormatter can convert all of them in one transformer, the fractional-handling has to be done for each Attribut on it's own ..


t_hohenstrater
Contributor
Forum|alt.badge.img+2

I support this idea. I am current facing the same issue downloading the data from arcgisonline which stores its time/date in Epoch milliseconds.


fmelizard
Safer
Forum|alt.badge.img+19
  • Safer
  • March 8, 2017

Here is a workspace to deal with epoch milliseconds until this idea is implemented.

Actually, this is not quite the answer as the sub second precision will be lost if I understand the comments here correctly.

 

epoch-to-date.fmw

fmelizard
Safer
Forum|alt.badge.img+19
  • Safer
  • March 8, 2017
Could you share your code?

 

 


david_r
Celebrity
  • March 9, 2017
I would, if I could remember where I used it, way back in 2015 :-)

Forum|alt.badge.img

Hi @dr_ysg

I am very happy to announce that FME 2017 supports fractional seconds (up to 9 decimal places). The new DateTimeFormatter transformer will be implemented in FME 2017.1. In 2017.0, please use datetime functions (see Text Editor > Date/Time Functions). For example, to convert seconds from epoch into human friendly temporal value:

_datetime = @DateTimeParse(_secondsFromEpoch, %s) _datetime = @DateTimeFormat(_datetime, %Y%m%d %T)

or

_datetime = @DateTimeFormat(@DateTimeParse(_secondsFromEpoch, %s), %Y%m%d %T)

Fractional seconds will be preserved in all datetime operations in FME 2017. We are now testing Readers and Writers to make sure they have no issues with reading/writing.

Please don't use the old DateFormatter though, as it will still drop the fractional part. DateFormatter will be deprecated as soon as we have DateTimeFormatter ready.


  • March 21, 2018
I couldn't get the text workarounds above to work in FME 2017.0

 

 

I had the data in the correct format for SQL Server datetime2 "2016-02-16 22:11:15.8990000", but FME refused to write this to the database as it didn't acknowledge it as a FME date type.

 

 

Work around was as follows:

 

1)

 

Atrtibute manager: new attribute: SaveMilliseconds TextFunction: @Substring(@Value(ReportDate),20,7)

 

 

2)

 

Then use DateFormatter to tell FME ReportDate is a Date field (losing milliseconds)

 

'

 

 

3)

 

Attribute Manager: Set value of ReportDate

 

TextFunction: @Value(ReportDate).@Value(SaveMilliseconds)

 

 

Remove column SaveMilliseconds

 

----------------------------------------------------------------------------------------------------

 

This allowed me to successfully write a datetime2 field in SQL server with milliseconds using a SQL Spatial Writer.

 

 


Forum|alt.badge.img
Hi @kiwigis

 

I guess, you get values read as strings. If you visualize them using Data Inspector, they are displayed in the same 2016-02-16 22:11:15.8990000 format, am I correct?

 

To write these values back into a database, you need to convert them into FME datetime format. This can be done either using @DateTimeParse() function or with DateTimeConverter:

 

 

'

 

 


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings