Not sure whether it's just because it's Friday afternoon and I'm missing something obvious but I'm seeing some strange behaviour with the attribute validator. Same input, same test, different results
Page 1 / 1
That is unusual. And the Inspector actually showed 1 feature for TEST1 and 2 features for TEST2? Or is it a glitch in the feature count?
Really strange. I was not able to reproduce the symptom. Please contact Safe support, sending the workspace and a minimal dataset which can reproduce that.
Does it make a difference if you attach Loggers instead of Inspectors? I've sometimes seen Inspectors somehow affect the transformer before them.
The inspector does actually show 1 feature for TEST1 and then 2 features (the same plus one additional) for TEST2.
Using a stringsearcher with the same regex finds a match for all 14,009 features.
I will see if i can create a workspace exhibiting the same behaviour which I can share next week.
This is with FME 2016.1
The other obvious question is, which is right? Should two features have failed, or just one? If one, what was the extra? A feature that should have passed or a duplicate of the fail?
The other obvious question is, which is right? Should two features have failed, or just one? If one, what was the extra? A feature that should have passed or a duplicate of the fail?
I've attached some data that I've experienced the issue with (now as csv, but original was shape)
Bizarrely after saving the workspace to share here I now see the following behaviour. The features failing now are not the ones that were failing before!
attribute-validator-test-data.csv
I've attached some data that I've experienced the issue with (now as csv, but original was shape)
Bizarrely after saving the workspace to share here I now see the following behaviour. The features failing now are not the ones that were failing before!
I feel it's a critical issue regarding the reliability of the AttributeValidator transformer. Hope someone from Safe clarifies our questions, including:
- In what condition does such a strange behavior occur? And is there any way to avoid this?
- Do the validation rules other than the Contains Regex work definitely fine?
- Does the problem have been resolved completely in FME 2017.0 RC?
I've attached some data that I've experienced the issue with (now as csv, but original was shape)
Bizarrely after saving the workspace to share here I now see the following behaviour. The features failing now are not the ones that were failing before!
Hello all, very sorry about all this. I can confirm that we found this exact problem ourselves during 2017.0 release testing and made the fix in FME 2017. We requested a backport to 2016.x as well, though I don't know when the next build is scheduled but will report back here when I find out.
The root problem was introduced late in the development FME 2016.0. We'd consolidated all our regular expression processing into a unicode centric single function call. The downside was that many translations were WAY slower than before. So we implemented a caching mechanism to avoid recomputing regular expression matches that we'd recently seen before in any transformer. (So the Tester/AttributeValidator/TestFilter all go through the same place.)
I do not know exactly the specifics, but it is possible in the right situation (which seems to have been duplicated here) to get this cache to do the wrong thing. Obviously once we discovered this we jumped right on it. And then a few days later @egomm discovered it as well.
I'll check on the status of a 2016.1 update and report back.
Sorry about this but thanks for confirming that the problem isn't there in 2017.0
Hello all, very sorry about all this. I can confirm that we found this exact problem ourselves during 2017.0 release testing and made the fix in FME 2017. We requested a backport to 2016.x as well, though I don't know when the next build is scheduled but will report back here when I find out.
The root problem was introduced late in the development FME 2016.0. We'd consolidated all our regular expression processing into a unicode centric single function call. The downside was that many translations were WAY slower than before. So we implemented a caching mechanism to avoid recomputing regular expression matches that we'd recently seen before in any transformer. (So the Tester/AttributeValidator/TestFilter all go through the same place.)
I do not know exactly the specifics, but it is possible in the right situation (which seems to have been duplicated here) to get this cache to do the wrong thing. Obviously once we discovered this we jumped right on it. And then a few days later @egomm discovered it as well.
I'll check on the status of a 2016.1 update and report back.
Sorry about this but thanks for confirming that the problem isn't there in 2017.0
The team advises that FME 2016.1.3.2 has been verified and released on safe.com @ http://www.safe.com/support/support-resources/fme-downloads/ Thanks for letting us know @egomm & @takashi
Thanks for the fix!