Skip to main content
Solved

DataTimeConverter corrupts data !?!

  • June 9, 2022
  • 9 replies
  • 24 views
lifalin2016
Supporter
Forum|alt.badge.img+38

Hi list.

I have a dataset with mixed date/time formats in a single field.

I want to create a cascade to transform these, i.e. "Try this format", and if it fails, "Try that format", etc.

HOWEVER, when data are output on DateTimeConverter's rejected port (signalling a data error), the field value is cleared, rendering a cascaded approach impossible. Why is that ??

Is there a way to avoid this unwarranted nulling of a possible valid field value ?

Cheers.

Best answer by lifalin2016

Actually, it just dawned on me, that it's defined in this file: C:\Program Files\FME 2022\transformers\fmesuite.fmx

The code seems to be Tcl/Tk, which I never really learned.

Are there any Tcl/Tk programmers out there, that may be able to decipher it and suggest a fix ?

YES!!!!

I did manage to decipher the TclTk code myself, and offer a correction.

This is the original code piece:

if {[string length $newAttrVal] == 0} {
         if {{$(PASSTHROUGH)} eq {NO} || [string length $oldAttrVal] > 0} {
            FME_SetAttribute fme_rejection_code "INVALID_INPUT";
            break;
         } elseif {!$oldAttrExists} {
            FME_UnsetAttributes $attr;
         } elseif {!$oldAttrNull} {
            FME_SetAttribute $attr {};
         };
      }

This is my fix:

if {[string length $newAttrVal] == 0} {
         if {{$(PASSTHROUGH)} eq {NO} || [string length $oldAttrVal] > 0} {
            FME_SetAttribute $attr $oldAttrVal;
            FME_SetAttribute fme_rejection_code "INVALID_INPUT";
            break;
         } elseif {!$oldAttrExists} {
            FME_UnsetAttributes $attr;
         } elseif {!$oldAttrNull} {
;#            FME_SetAttribute $attr {};
            FME_SetAttribute $attr $oldAttrVal;
         };
      }

And it works 🙂

Now I can use the transformers in a cascading manner.

Yippee!

--------------------- edit:

Actually, the fix need only be added to the "INVALID_INPUT" block:

      if {[string length $newAttrVal] == 0} {
         if {{$(PASSTHROUGH)} eq {NO} || [string length $oldAttrVal] > 0} {
            FME_SetAttribute $attr $oldAttrVal;
            FME_SetAttribute fme_rejection_code "INVALID_INPUT";
            break;
         } elseif {!$oldAttrExists} {
            FME_UnsetAttributes $attr;
         } elseif {!$oldAttrNull} {
            FME_SetAttribute $attr {};
         };
      }

And the updated transformer definition can be kept in a separate FMX file in the same folder as fmesuite.fmx. Just remember to increase its version number.

-- Cheers, Lars I.
    This post is closed to further activity.
    It may be an old question, an answered question, an implemented idea, or a notification-only post.
    Please check post dates before relying on any information in a question or answer.
    For follow-up or related questions, please post a new question or idea.
    If there is a genuine update to be made, please contact us and request that the post is reopened.

    9 replies

    david_r
    Celebrity
    • david_rCelebrity
    • 8391 replies
    • June 9, 2022

    Unfortunately this is a known issue, you'll find several threads on these forums with users jumping through hoops to work around it, e.g. https://community.safe.com/s/question/0D54Q000080hbnrSAA/datetimeconverter-rejected-port-null

    It's regrettable that the issue seems to still be there in 2022.0.

    lifalin2016
    Supporter
    Forum|alt.badge.img+38
    • lifalin2016Supporter
    • Author
    • Supporter
    • 592 replies
    • June 9, 2022

    And it would such an incredibly easy fix, so why is it that hard to fix it ?

    -- Cheers, Lars I.
    lifalin2016
    Supporter
    Forum|alt.badge.img+38
    • lifalin2016Supporter
    • Author
    • Supporter
    • 592 replies
    • June 9, 2022

    And it would such an incredibly easy fix, so why is it that hard to fix it ?

    Actually, it just dawned on me, that it's defined in this file: C:\\Program Files\\FME 2022\\transformers\\fmesuite.fmx

    The code seems to be Tcl/Tk, which I never really learned.

    Are there any Tcl/Tk programmers out there, that may be able to decipher it and suggest a fix ?

    -- Cheers, Lars I.
    lifalin2016
    Supporter
    Forum|alt.badge.img+38
    • lifalin2016Supporter
    • Author
    • Supporter
    • 592 replies
    • Best Answer
    • June 9, 2022

    Actually, it just dawned on me, that it's defined in this file: C:\Program Files\FME 2022\transformers\fmesuite.fmx

    The code seems to be Tcl/Tk, which I never really learned.

    Are there any Tcl/Tk programmers out there, that may be able to decipher it and suggest a fix ?

    YES!!!!

    I did manage to decipher the TclTk code myself, and offer a correction.

    This is the original code piece:

    if {[string length $newAttrVal] == 0} {
             if {{$(PASSTHROUGH)} eq {NO} || [string length $oldAttrVal] > 0} {
                FME_SetAttribute fme_rejection_code "INVALID_INPUT";
                break;
             } elseif {!$oldAttrExists} {
                FME_UnsetAttributes $attr;
             } elseif {!$oldAttrNull} {
                FME_SetAttribute $attr {};
             };
          }

    This is my fix:

    if {[string length $newAttrVal] == 0} {
             if {{$(PASSTHROUGH)} eq {NO} || [string length $oldAttrVal] > 0} {
                FME_SetAttribute $attr $oldAttrVal;
                FME_SetAttribute fme_rejection_code "INVALID_INPUT";
                break;
             } elseif {!$oldAttrExists} {
                FME_UnsetAttributes $attr;
             } elseif {!$oldAttrNull} {
    ;#            FME_SetAttribute $attr {};
                FME_SetAttribute $attr $oldAttrVal;
             };
          }

    And it works 🙂

    Now I can use the transformers in a cascading manner.

    Yippee!

    --------------------- edit:

    Actually, the fix need only be added to the "INVALID_INPUT" block:

          if {[string length $newAttrVal] == 0} {
             if {{$(PASSTHROUGH)} eq {NO} || [string length $oldAttrVal] > 0} {
                FME_SetAttribute $attr $oldAttrVal;
                FME_SetAttribute fme_rejection_code "INVALID_INPUT";
                break;
             } elseif {!$oldAttrExists} {
                FME_UnsetAttributes $attr;
             } elseif {!$oldAttrNull} {
                FME_SetAttribute $attr {};
             };
          }

    And the updated transformer definition can be kept in a separate FMX file in the same folder as fmesuite.fmx. Just remember to increase its version number.

    -- Cheers, Lars I.
    david_r
    Celebrity
    • david_rCelebrity
    • 8391 replies
    • June 9, 2022

    Actually, it just dawned on me, that it's defined in this file: C:\\Program Files\\FME 2022\\transformers\\fmesuite.fmx

    The code seems to be Tcl/Tk, which I never really learned.

    Are there any Tcl/Tk programmers out there, that may be able to decipher it and suggest a fix ?

    Thanks for sharing!

    DanAtSafe
    Safer
    Forum|alt.badge.img+19
    • DanAtSafeSafer
    • Safer
    • 344 replies
    • June 9, 2022

    Unfortunately this is a known issue, you'll find several threads on these forums with users jumping through hoops to work around it, e.g. https://community.safe.com/s/question/0D54Q000080hbnrSAA/datetimeconverter-rejected-port-null

    It's regrettable that the issue seems to still be there in 2022.0.

    FYI, the unresolved known issue is FMEENGINE-56103

    lifalin2016
    Supporter
    Forum|alt.badge.img+38
    • lifalin2016Supporter
    • Author
    • Supporter
    • 592 replies
    • June 10, 2022

    FYI, the unresolved known issue is FMEENGINE-56103

    Hi Dan.

    Did you see my fix ?

    It's a very small thing to fix in the fmesuite.fmx file, so couldn't it, and similar fixes, be escalated to be fixed asap?

    In the meantime:

    Will any FMX file in the same folder be read upon start, so one can add custom updated versions as a seperate file?

    -- Cheers, Lars I.
    lifalin2016
    Supporter
    Forum|alt.badge.img+38
    • lifalin2016Supporter
    • Author
    • Supporter
    • 592 replies
    • June 10, 2022

    FYI, the unresolved known issue is FMEENGINE-56103

    Hi Dan.

    I went ahead and tried it, and yes, a seperate FMX file will indeed be read upon startup, so such customizations can be kept in its own file. Easier to distribute too.

    -- Cheers, Lars I.
    DanAtSafe
    Safer
    Forum|alt.badge.img+19
    • DanAtSafeSafer
    • Safer
    • 344 replies
    • June 11, 2022

    FYI, the unresolved known issue is FMEENGINE-56103

    Yes, I saw your fix and I've escalated the issue. Thanks!

    Community Stats

    32,065
    Posts
    121,732
    Replies
    39,691
    Members

    Latest FME

    Terms & ConditionsCookie settingsAccessibility statement
    A product of
    Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.