Skip to main content
Question

Issues on Date/Time User Parameter

takashi
Contributor
Forum|alt.badge.img+19

Related to this post: Datetime: %s / Epoch Time / Unix Time

Great enhancements regarding date/time manipulation have been done in FME 2017.0, but I think there still is a room to improve the behavior of Date/Time type user parameter on Workbench. 1. Currently (FME 2017.0.0.1), a Date/Time type parameter edit field only accepts a datetime string formatted with %Y%m%d%H%M%S, and rejects decimal places of seconds and a timezone specifier (UTC offset). It would be better if the field could accept datetime representation formatted with Standard FME Date/Time Format, which allows both decimal places and a timezone specifier optionally.

2. Even if you entered a datetime formatted with %Y%m%d%H%M%S successfully, it will appear with a local format in the Navigator window and the re-opened edit parameter dialog. The implicit formatting is also done when selecting a datetime from the date picker tool. It would be better if the datetime would always appear in the parameter field with Standard FME Date/Time Format. No need to format it automatically with any local format.

3. A datetime parameter value without timezone specifier will be interpreted as a local time. It's OK, but it will be converted to a UTC datetime with timezone specifier implicitly when configuring the mapping file. I think it would be better if a datetime representation without timezone specifier would also be treated as a local time without timezone, as-is. In general, any implicit conversion could cause unnecessary confusion.

I submitted a support case C120294 about the 3rd issue 5+ months ago, and also raised other issues through the discuss about the case, but there wasn't any change for the FME 2017.0 release. What is the current status about the case?

17 replies

takashi
Contributor
Forum|alt.badge.img+19
  • takashiContributor
  • Author
  • Contributor
  • March 23, 2017
In addition, the datetime representation 3? 23, 2017 12:00:00 ?? shown in the screenshot above is very strange for us. If you intend to display a datetime with a popular local format, it should be:

 

2017?3?23? 00:00:00 or 2017?3?23? 00?00?00?

 

in Japanese locale, like this (from Japanese Windows)

 

 

However, it might be hard to convert between standard date/time format and the popular local format here. This is also one of reasons for that I think it would be better if a datetime could always be shown with Standard FME Date/Time Format, unless the user format that explicitly.

 

fmelizard
Contributor
Forum|alt.badge.img+17
  • fmelizardContributor
  • Contributor
  • March 23, 2017
Hi @takashi. Thank you for the feedback. Is there a reason you placed the feedback in the Community Moderation space, rather than in the post Datetime: %s / Epoch Time / Unix Time? It appears that your post may also be suitable for the Ideas section, as it sounds to me like enhancements to Date/Time behavior.

 

 

In regards to your case, C120294, Laura is the case owner, and I have requested she or someone else from support follow up with you. Thanks!

 

takashi
Contributor
Forum|alt.badge.img+19
  • takashiContributor
  • Author
  • Contributor
  • March 24, 2017

Hi @NatalieAtSafe and @LauraAtSafe, thanks for your response.

I have submitted approx 50 cases about FME 2017 defects or enhancements in the beta stage. Most of them have been resolved in the release version or earlier, but the Date/Time parameter issue has been left behind. For the cases, I basically wait for your update and will accept your final decision. However, regarding the Date/Time parameter issues, long time has passed since I submitted the case firstly, and I believe that these are critical issues which should be resolved as soon as possible, since the enhancement of Date/Time manipulation is one of the major points of update in FME 2017.

About the 1st and 2nd issues:

  1. Rejecting decimal places and timezone identifier is apparently inconsistent with the Standard FME Date/Time Format specifications.
  2. When a workspace has a Date/Time parameter with a default value, the user has to re-enter a datetime value with %Y%m%d%H%M%S format every time to run with "Run with Prompt" mode, even if it was not necessary to change the default value.

Those are obvious issues which can be detected easily, and I therefore think those had to be detected in the QA process before releasing, even if no user submitted any case in the beta stage. The Date/Time parameter issue indicates that there could be a serious defect in your QA process itself, so I hesitated to publish it into a general space in the Q&A; Forum or the Idea. That is, I was afraid that my post could bring concern about the entire quality of FME to other users.

About the 3rd issue, I agree that the contents itself is suitable to post into the Idea site. However, the datetime format shown in the screenshot (Navigator) is not good, so I didn't want to show it in public. The format is not popular in Japanese locale, is just a literal (mechanical) translation from a format in English locale - '%B %d, %Y %I:%M:%S %p'. If a Japanese potential customer looked at this, they might believe that FME could not be used in Japanese locale.

That's the reason why I posted it to the Community Moderation space, though I know it's not the best space.

I'm always nervous about defects of FME as a reseller. Because, even just a small issue could cause a fatal damage to the reliability of a software especially if the issue can be detected easily.

takashi
Contributor
Forum|alt.badge.img+19
  • takashiContributor
  • Author
  • Contributor
  • March 25, 2017
takashi wrote:

Hi @NatalieAtSafe and @LauraAtSafe, thanks for your response.

I have submitted approx 50 cases about FME 2017 defects or enhancements in the beta stage. Most of them have been resolved in the release version or earlier, but the Date/Time parameter issue has been left behind. For the cases, I basically wait for your update and will accept your final decision. However, regarding the Date/Time parameter issues, long time has passed since I submitted the case firstly, and I believe that these are critical issues which should be resolved as soon as possible, since the enhancement of Date/Time manipulation is one of the major points of update in FME 2017.

About the 1st and 2nd issues:

  1. Rejecting decimal places and timezone identifier is apparently inconsistent with the Standard FME Date/Time Format specifications.
  2. When a workspace has a Date/Time parameter with a default value, the user has to re-enter a datetime value with %Y%m%d%H%M%S format every time to run with "Run with Prompt" mode, even if it was not necessary to change the default value.

Those are obvious issues which can be detected easily, and I therefore think those had to be detected in the QA process before releasing, even if no user submitted any case in the beta stage. The Date/Time parameter issue indicates that there could be a serious defect in your QA process itself, so I hesitated to publish it into a general space in the Q&A; Forum or the Idea. That is, I was afraid that my post could bring concern about the entire quality of FME to other users.

About the 3rd issue, I agree that the contents itself is suitable to post into the Idea site. However, the datetime format shown in the screenshot (Navigator) is not good, so I didn't want to show it in public. The format is not popular in Japanese locale, is just a literal (mechanical) translation from a format in English locale - '%B %d, %Y %I:%M:%S %p'. If a Japanese potential customer looked at this, they might believe that FME could not be used in Japanese locale.

That's the reason why I posted it to the Community Moderation space, though I know it's not the best space.

I'm always nervous about defects of FME as a reseller. Because, even just a small issue could cause a fatal damage to the reliability of a software especially if the issue can be detected easily.

I found the format '%B %d, %Y %I:%M:%S %p' in English locale is acceptable. I guess it's the reason why the 2nd issue has been overlooked in your QA process. Isn't it?

 

 

But others are not acceptable...

 

 

david_r
Evangelist
  • david_rEvangelist
  • March 27, 2017

Very interesting findings, Takashi. Just goes to show that date/time handling is always much more complex that one might think.

Hopefully this will all be sorted out for 2017.1? As it stands, I'm very reluctant to use (or recommend) this new functionality.

fmelizard
Contributor
Forum|alt.badge.img+17
  • fmelizardContributor
  • Contributor
  • March 27, 2017

Thanks for alerting us @takashi and for your great passion for making FME the best it can be. As a result I'm going to also ask our release team to ensure we review your cases as part of our close down -- this definitely was something we shouldn't have missed. I've asked @LenaAtSafe to work with the developer team here and also advise us in this post as to the planned time for fixing this.

L
Forum|alt.badge.img

Hi @takashi

 

thank you for summarizing datetime published parameter issues and sharing this report with us. @LauraAtSafe has filed 3 PRs based on your report and we have them as high customer impact now. I can not guarantee this yet, but I very much hope the issues will be fixed in 2017.1.

 

To iterate:
  1. datetime parameter default value should accept fractional seconds;
  2. datetime parameter default value should accept time offset;
  3. unzoned datetime value should not be assumed local and recalculated into UTC;
  4. any default value should be preserved as is (i.e. not recalculated into UTC);
  5. any system datetime format should be OK for datetime parameter default value.

Regarding #3: I think it is safer to leave unzoned as unzoned. If the user knows that the time is actually local and having the values as zoned datetime values is important, they should type in/pick time with offset (as #2 will allow) as the default value OR tag the default value with offset later (this functionality is coming).

Regarding #5: unfortunately, I don't think we will be able to implement non-English datetime strings parsing now. I.e. if I type in 27 ????? 2017 ???? in Russian, FME 2017 will not be able to parse this string as 20170327. This is not about Russian - or Japanese - characters, this is due to the fact that at the moment only English month/week day names are used for parsing. However, if a user types in 20170327 which is later displayed as 27 ????? 2017 based on the system datetime format setting, this string should stay perfectly valid/editable.

 

However, the datetime format shown in the screenshot (Navigator) is not good, so I didn't want to show it in public. The format is not popular in Japanese locale, is just a literal (mechanical) translation from a format in English locale - '%B %d, %Y %I:%M:%S %p'.

I am surprised: I have tried several different formats and only Japanese so far gets translated; with other formats, long datetime format seems to be applied. We will investigate this further. For any internationalization test, Japanese is the best choice as it helps us discover majority of problems. Thank you for pointing out this issue specifically.

We very much appreciate your input. I am not sure what happened to your original datetime parameter report - we can not locate it. I apologize for this.

While we do very thorough testing, we can not detect all issues. By sharing your user experience you help us keep providing all FME users with high quality tool.

I also would like to thank you for your reply in https://knowledge.safe.com/questions/41560/datetime-s-epoch-time-unix-time.html You provided us with a lot of food for thoughts. With your input, we decided to alter some of our original plans. We are still working on %s support and I will post some updates in https://knowledge.safe.com/questions/41560/datetime-s-epoch-time-unix-time.html shortly.

Please share any datetime ideas/suggestions/concerns you might have. The more we hear from users, the better the new functionality will be.

 

 

 

 

L
Forum|alt.badge.img
david_r wrote:

Very interesting findings, Takashi. Just goes to show that date/time handling is always much more complex that one might think.

Hopefully this will all be sorted out for 2017.1? As it stands, I'm very reluctant to use (or recommend) this new functionality.

Hi @david_r

 

could you please share your concerns regarding the new datetime functionality? I am involved in Datetime Project, my role is to gather users' requests and make sure the new functionality will indeed solve users' problems. If there is a task you are not sure the new functionality will help with, or concern regarding correctness/interpretation, or an enhancement request - please share and we will follow up and do our best to deliver the high quality functionality.

 

So far, we have surrounded development with a lot of research and testing. As the functionality is new, we have a luxury to rethink and redo if what seemed to be a perfect solution... two weeks ago doesn't feel that perfect today. We really appreciate any feedback.

 

Speaking of feedback: if you have any thoughts regarding epoch time or have ever worked with seconds from epoch and would like to share your experience, please add to https://knowledge.safe.com/questions/41560/datetime-s-epoch-time-unix-time.html discussion.

 

takashi
Contributor
Forum|alt.badge.img+19
  • takashiContributor
  • Author
  • Contributor
  • March 28, 2017

Hi @daleatsafe and @LenaAtSafe, thanks for your considerations and investigation on this.

Possible Date/Time formats are different depending on locale, and there could also be several formats in the same locale. I think it's hard to support all the possible formats for every locale as the Date/Time type parameter input. In my personal opinion, it would be better if the parameter field and Navigator could only accept/display the Standard FME Date/Time format and/or some formats which don't depend on any locale, e.g. ISO 8601. If you did so, the thing would be simpler and also more robust, I think.

fmelizard
Contributor
Forum|alt.badge.img+17
  • fmelizardContributor
  • Contributor
  • March 28, 2017
takashi wrote:

Hi @daleatsafe and @LenaAtSafe, thanks for your considerations and investigation on this.

Possible Date/Time formats are different depending on locale, and there could also be several formats in the same locale. I think it's hard to support all the possible formats for every locale as the Date/Time type parameter input. In my personal opinion, it would be better if the parameter field and Navigator could only accept/display the Standard FME Date/Time format and/or some formats which don't depend on any locale, e.g. ISO 8601. If you did so, the thing would be simpler and also more robust, I think.

I completely agree.

 

 

takashi
Contributor
Forum|alt.badge.img+19
  • takashiContributor
  • Author
  • Contributor
  • March 28, 2017
lenaatsafe wrote:

Hi @takashi

 

thank you for summarizing datetime published parameter issues and sharing this report with us. @LauraAtSafe has filed 3 PRs based on your report and we have them as high customer impact now. I can not guarantee this yet, but I very much hope the issues will be fixed in 2017.1.

 

To iterate:
  1. datetime parameter default value should accept fractional seconds;
  2. datetime parameter default value should accept time offset;
  3. unzoned datetime value should not be assumed local and recalculated into UTC;
  4. any default value should be preserved as is (i.e. not recalculated into UTC);
  5. any system datetime format should be OK for datetime parameter default value.

Regarding #3: I think it is safer to leave unzoned as unzoned. If the user knows that the time is actually local and having the values as zoned datetime values is important, they should type in/pick time with offset (as #2 will allow) as the default value OR tag the default value with offset later (this functionality is coming).

Regarding #5: unfortunately, I don't think we will be able to implement non-English datetime strings parsing now. I.e. if I type in 27 ????? 2017 ???? in Russian, FME 2017 will not be able to parse this string as 20170327. This is not about Russian - or Japanese - characters, this is due to the fact that at the moment only English month/week day names are used for parsing. However, if a user types in 20170327 which is later displayed as 27 ????? 2017 based on the system datetime format setting, this string should stay perfectly valid/editable.

 

However, the datetime format shown in the screenshot (Navigator) is not good, so I didn't want to show it in public. The format is not popular in Japanese locale, is just a literal (mechanical) translation from a format in English locale - '%B %d, %Y %I:%M:%S %p'.

I am surprised: I have tried several different formats and only Japanese so far gets translated; with other formats, long datetime format seems to be applied. We will investigate this further. For any internationalization test, Japanese is the best choice as it helps us discover majority of problems. Thank you for pointing out this issue specifically.

We very much appreciate your input. I am not sure what happened to your original datetime parameter report - we can not locate it. I apologize for this.

While we do very thorough testing, we can not detect all issues. By sharing your user experience you help us keep providing all FME users with high quality tool.

I also would like to thank you for your reply in https://knowledge.safe.com/questions/41560/datetime-s-epoch-time-unix-time.html You provided us with a lot of food for thoughts. With your input, we decided to alter some of our original plans. We are still working on %s support and I will post some updates in https://knowledge.safe.com/questions/41560/datetime-s-epoch-time-unix-time.html shortly.

Please share any datetime ideas/suggestions/concerns you might have. The more we hear from users, the better the new functionality will be.

 

 

 

 

Regarding #3, I have opposite view. That is, unzoned datetime value should be assumed local time, should not be recalculated to UTC datetime.

 

It's just my personal view. I will respect your final decision.

 

david_r
Evangelist
  • david_rEvangelist
  • March 28, 2017
lenaatsafe wrote:
Hi @david_r

 

could you please share your concerns regarding the new datetime functionality? I am involved in Datetime Project, my role is to gather users' requests and make sure the new functionality will indeed solve users' problems. If there is a task you are not sure the new functionality will help with, or concern regarding correctness/interpretation, or an enhancement request - please share and we will follow up and do our best to deliver the high quality functionality.

 

So far, we have surrounded development with a lot of research and testing. As the functionality is new, we have a luxury to rethink and redo if what seemed to be a perfect solution... two weeks ago doesn't feel that perfect today. We really appreciate any feedback.

 

Speaking of feedback: if you have any thoughts regarding epoch time or have ever worked with seconds from epoch and would like to share your experience, please add to https://knowledge.safe.com/questions/41560/datetime-s-epoch-time-unix-time.html discussion.

 

Hi Lena, sorry I should've been more specific. I've had several bad experiences over the years using newly developed date/time libraries, which makes me very reluctant to use any new ones before they're been thoroughly tested and they've got some real world use under their belt. I'll note that none of these bad experiences were with FME, however :-)

 

For our users, the primary concern is related to locale and regional settings, these things must work perfectly out of the box. I know it's a complex issue, so I appreciate the efforts by @takashi here. I'll of course be sure to flag any issues I might come across myself.

 

david_r
Evangelist
  • david_rEvangelist
  • March 28, 2017
takashi wrote:
Regarding #3, I have opposite view. That is, unzoned datetime value should be assumed local time, should not be recalculated to UTC datetime.

 

It's just my personal view. I will respect your final decision.

 

I agree with @takashi on no. 3. Amongst our users almost nobody uses UTC time stamps, 99% would assume local time when no TZ is specified.

 

L
Forum|alt.badge.img
david_r wrote:
Hi Lena, sorry I should've been more specific. I've had several bad experiences over the years using newly developed date/time libraries, which makes me very reluctant to use any new ones before they're been thoroughly tested and they've got some real world use under their belt. I'll note that none of these bad experiences were with FME, however :-)

 

For our users, the primary concern is related to locale and regional settings, these things must work perfectly out of the box. I know it's a complex issue, so I appreciate the efforts by @takashi here. I'll of course be sure to flag any issues I might come across myself.

 

Locale and regional settings? Could you please elaborate on this? There will be some limitations in FME 2017, e.g. %B will only mean month name in English. Adding other languages support is not trivial (which I didn't expect, to be honest), but we definitely need to know if this is important for users. It will be possible to parse/format datetime values from/into regional formats - is it really important to provide users with a ready to use flag meaning regional format or is it OK to offer tools to construct a format string that matches regional format?

 

Regarding the testing: we did extensive testing and continue testing the new features (we are adding some in 2017.1). This can't guarantee that there are no bugs/logical mistakes left :) But we are working hard on QA. If you find any bugs or inconsistencies, please let us know ASAP - we will find resources to fix the issues (in other words, all problem reports will be treated as high priority).

 

 

L
Forum|alt.badge.img
takashi wrote:
I found the format '%B %d, %Y %I:%M:%S %p' in English locale is acceptable. I guess it's the reason why the 2nd issue has been overlooked in your QA process. Isn't it?

 

 

But others are not acceptable...

 

 

Yes, you are right, %B %d, %Y %I:%M:%S %p is one of the acceptable formats on English locale. I've noticed that with any system format settings, the default value display doesn't really match the long format although the correct language is used. I previously claimed that with other than Japanese format settings the display matches the system long format - I was wrong, I didn't pay enough attention to details. The key difference is that with languages I can understand the display... was not that awkward (and it was not always the same) while with Japanese it was absolutely wrong. I guess we need a native Japanese speaker on our QA team to notice issues like this :)

 

takashi
Contributor
Forum|alt.badge.img+19
  • takashiContributor
  • Author
  • Contributor
  • March 28, 2017
lenaatsafe wrote:

Hi @takashi

 

thank you for summarizing datetime published parameter issues and sharing this report with us. @LauraAtSafe has filed 3 PRs based on your report and we have them as high customer impact now. I can not guarantee this yet, but I very much hope the issues will be fixed in 2017.1.

 

To iterate:
  1. datetime parameter default value should accept fractional seconds;
  2. datetime parameter default value should accept time offset;
  3. unzoned datetime value should not be assumed local and recalculated into UTC;
  4. any default value should be preserved as is (i.e. not recalculated into UTC);
  5. any system datetime format should be OK for datetime parameter default value.

Regarding #3: I think it is safer to leave unzoned as unzoned. If the user knows that the time is actually local and having the values as zoned datetime values is important, they should type in/pick time with offset (as #2 will allow) as the default value OR tag the default value with offset later (this functionality is coming).

Regarding #5: unfortunately, I don't think we will be able to implement non-English datetime strings parsing now. I.e. if I type in 27 ????? 2017 ???? in Russian, FME 2017 will not be able to parse this string as 20170327. This is not about Russian - or Japanese - characters, this is due to the fact that at the moment only English month/week day names are used for parsing. However, if a user types in 20170327 which is later displayed as 27 ????? 2017 based on the system datetime format setting, this string should stay perfectly valid/editable.

 

However, the datetime format shown in the screenshot (Navigator) is not good, so I didn't want to show it in public. The format is not popular in Japanese locale, is just a literal (mechanical) translation from a format in English locale - '%B %d, %Y %I:%M:%S %p'.

I am surprised: I have tried several different formats and only Japanese so far gets translated; with other formats, long datetime format seems to be applied. We will investigate this further. For any internationalization test, Japanese is the best choice as it helps us discover majority of problems. Thank you for pointing out this issue specifically.

We very much appreciate your input. I am not sure what happened to your original datetime parameter report - we can not locate it. I apologize for this.

While we do very thorough testing, we can not detect all issues. By sharing your user experience you help us keep providing all FME users with high quality tool.

I also would like to thank you for your reply in https://knowledge.safe.com/questions/41560/datetime-s-epoch-time-unix-time.html You provided us with a lot of food for thoughts. With your input, we decided to alter some of our original plans. We are still working on %s support and I will post some updates in https://knowledge.safe.com/questions/41560/datetime-s-epoch-time-unix-time.html shortly.

Please share any datetime ideas/suggestions/concerns you might have. The more we hear from users, the better the new functionality will be.

 

 

 

 

And regarding #5, I think it's enough that the parameter field and Navigator could only accept/display the Standard FME Date/Time format and/or some formats which don't depend on any locale, e.g. ISO 8601. No need to allow using any locale (system) specific datetime format here, since the user can use the DateTimeFormatter transformer or @DateTimeFormat function in the workflow whenever they needs to format the value with a specific format.

 

L
Forum|alt.badge.img
takashi wrote:
Regarding #3, I have opposite view. That is, unzoned datetime value should be assumed local time, should not be recalculated to UTC datetime.

 

It's just my personal view. I will respect your final decision.

 

I am not suggesting to recalculate unzoned time into UTC assuming it is local. I actually would prefer it to stay unzoned. It is risky to assume the time zone for unzoned values - especially when a workspace migrates to FME Server/Cloud.

 

I completely agree with both of you - no values should be recalculated into UTC implicitly.

 

 

Reply


Helpful Members This Week

Community Stats

30,942
Posts
117,252
Replies
38,742
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.