Esri ArcGIS Feature Service - Reads Dates as Epoch Milliseconds

Yuck!

Can confirm the same with an Enterprise Source Type with the Esri ArcGIS Feature Service Reader

​@braggken - have you confirmed that you can’t write the epoch time using the newer writer and conversion is required?

I’ve not managed to get the newer writer working at all yet...

Thanks for confirming ​@ebygomm - can you tell me your version of FME please?
No I can’t write epoch time with the new writer, at least not an existing AGOL feature layer.  The other hitch I noticed is that where there was already a GlobalID field in the layer it seems we now have to send a guid along whereas before that was not needed.

We are going to have to migrate our Enteprise readers/writers also which will be a much bigger job.   

Using the same package (Esri ArcGIS Connector 3.11.1) I've tested connecting from

• FME 2024.1.0.0 Build 24612
• FME 2025.1.0.1 Build 25609 

Further to the above, I’ve now managed to update an Enterprise feature service using the Esri ArcGIS Feature Service writer - I get a permissions issue if i try to use token authentication but it worked with Oauth authentication. 

I can update a date field sending either milliseconds since epoch or an FME date time (attribute type needs to be datetime) to the writer

Esri data types for Feature Services are always such a pain… Especially the datetime datatypes. 

To convert UNIX millisecond epoch time to YYYYMMDD format, you need to use the AttributeManager’s Arithmetic Editor to first convert the UNIX milliseconds to seconds (e.g., @Value(ATTRIBUTENAME)/1000). Then, use the DateTimeConverter transformer to format UNIX seconds into YYYYMMDD. Depending on your workflow, you could create a dynamic workspace that only rectifies the date values for Feature Services.

Additionally, there’s also a quirk with Esri Feature Services where you cannot write the value 19700101000000 and Esri converting Date Time value to local time zones (Note: I am not sure if this behavior is still present with Esri Feature Services). My memory is a bit hazy, but you can find the thread here:

Yes thanks ​@geospatialize  as I mentioned in the inital post,  “Obviously this can be worked around ..“  but its good to have posted the details.
But the issue is the way datetimes are read by the new reader has changed. We have over 100 workspaces that read and write AGOL and Portal many of which manipulate datetimes within.  Also rather annoyingly, the new datetime format cannot be handled by DateTimeConverter so as you pointed out, the work-around involves an additional step for every datetime attribute being handled to convert from milliseconds to seconds. 

It certainly gives no indication in the article “Working with Esri ArcGIS Feature Services in FME” that there is a fundamental change in the data being returned when using the new reader v the old. And it concerns me that there might be other similar discrepancies. 

About the date using the new Writer, I have also been able to insert dates in FME format, but it keeps throwing errors when I use dates in milliseconds (even though the value is exactly the same as the ArcGIS feature Reader returns).

I have made sure to set the type as “datetime” with An AttributeManager. Did you do anything else to make it work with milliseconds?

Hi folks,

I've submitted a ticket for this behavior (FMEENGINE-87880).

The current workaround for the ArcGIS Feature Service Writer to accept UNIX milliseconds (if Data type is String) is to set the Attribute Definition as Manual and specify the Attribute with the Date value in an arithmetic editor within the Writer's User Attributes.

I apologize for this inconvenience. Thank you for your understanding.

So i hadn’t tried to insert a new feature with a date in milliseconds, just update an existing and that worked with no other changes other than setting a new value in the AttributeCreator (I thought it wasn’t updated for a while until i realised that although milliseconds are returned, only whole seconds seem to be accepted, i.e. a value of 1757434408001 ends up as 1757434408000)

I don’t even want to look into how this impacts dynamic workflows….

Hi folks,

I've submitted a ticket for this behavior (FMEENGINE-87880).

The current workaround for the ArcGIS Feature Service Writer to accept UNIX milliseconds (if Data type is String) is to set the Attribute Definition as Manual and specify the Attribute with the Date value in an arithmetic editor within the Writer's User Attributes.

I apologize for this inconvenience. Thank you for your understanding.

What about the ArcGIS Feature Service Reader?

I’ve tried all of the above workarounds for a dynamic ArcGIS Feature Service reader, which shows date fields as ‘Date’ in the Attributes list within the Navigator tree, and int64 in the AttributeManager.

I really don’t want to use a non-dynamic reader because I’m reading hundreds of attributes (don’t ask...), and it will be a massive pain to manage them manually.

I can’t, for the life of me, get DateTimeConverter to recognise any of the time attributes from the ArcGIS Feature Service reader (connected to an ArcGIS Online instance), irrespective of whether I convert them, change the data type, or divide by 1000 to bring them seconds units.

Example time: 1760436656063

The only other potential workaround I can think of, is to write a Python Caller that does the conversion but that’s going to be a pain, and waste work, to implement.

Any ideas?

​@braggken the article was updated ten days ago and gives the detail to this question.

Hope this will help folks who find this issue. 

The way its read from the ArcGIS Data Store has not changed. Safe and ESRI both made it easy for end user by converting them whether its in the Workbench app in the legacy formats or the Esri web online maps and feature service tables.

Why Safe decided to leave the conversion for workspace authors like us in the new ESRI ArcGIS Feature Servcice format must have been given due consideration. I would assume keeping this in your workflow as epoch time would address end users demand to allow macine readable analyses that will later be converted to human readable downstream.

There will be pains with migrating older formats expecially for dynamic workflows and that would probably be another good article to ask for ​@ebygomm.

+

For anyone that has had continued problems attempting to use the DateTimeConverter, I found a silly solution in another thread that resolved it for me.

After converting Epoch milliseconds to Epoch seconds using the Arithmetic Editor in Attribute Manager as noted above (@Value(ATTRIBUTENAME)/1000)

When feeding the result into the DateTimeConverter, do not use the down-arrow to select the attribute to convert. This will cause it to use the literal string instead of the actual value, and result in an error.

Instead, use the three dots (...) and select the attribute in the list. It should run without error now.

​@TandraAtSafe I am using the Esri ArcGIS Feature Service Reader in package version 3.13.2 but still seeing the same issue with the dates in FME Form 2024.2.2.0 (20250114 - Build 24820 - WIN64).

If I try to run it through DateTimeConverter I get the following rejection message:

“CREATIONDATE: The epoch time is out of range. Please ensure it is specified in seconds, and not in milliseconds or microseconds.”

FME Hub says the package was last updated Oct 31.

Is there a newer version of the package that resolves this issue?

Similarly to braggken my workbenches were built on YYYYMMDDHHMMSS format and retrofitting each one will be a pain.

Unfortunately we still experience the same problems with version 3.13.2 just as mentioned by sammy.

The history of the package shows the following fix at 3.13.2:

• Writer: Ensure epoch millisecond date format is accepted. (FMEENGINE-87880)

This suggests something is fixed for the writer, but the problem described in this topic involves the reader. And the different behaviour to the old reader when reading date fields.