Solved

Several issues encountered since upgrading from FME 2020.2.5.0 to the latest version yesterday


Badge +10

The upgrade to FME Desktop 2021.0.3 Build 21326 was carried out yesterday morning (Desktop / Server)

Issue 1

I have a workspace which used to work until the upgrade.

Simply put, a downloaded CSV file is read then written into a truncated Postgis table.

This table is responsible for the functioning of this site including its contents as FME generated. (https://molevalley.cloudmappin.com/mmv/)

 

The scheduled workflow reported that it was successful through an e-mail notification. However when I checked the site this morning, the address lookup failed to work therefore no one could use the site.

 

During investigation I realised that the table was empty. (Truncated but no data)

 

EvoCall2Postgis-EmptyTbl 

Took a closer look and saw the following:

EvoCall2Postgis-Line 

Features were read as line geom whereby the original CSV is read as point data.

PS: The reason why the Vertex transformer is used is to ensure the correct X/Y (Easting/Northing) fields are used

 

Data detected as point geom when bypassing the Vertex transformer

EvoCall2Postgis-Point 

The "fix" was to use the "Geometry Remover"

This issue affected quite a lot of tables

EvoCall2Postgis-AffectedTbl 

Issue 2 (Same workspace)

Since FME was passing through the line geom features, the transaction log displayed the following as Warnings:

EvoCall2Postgis-WarningLineGeom 

Upon job completion I get a successful translation instead of a failed one with zero Output .

EvoCall2Postgis-FeaturesWritten 

Issue 3:

Different workspace but same behaviour, meaning a failed workspace completed as successful.

 

MyMV-PlanningHTML-ErrorSuccessErrors

MyMV-PlanningHTML-ErrorSuccess2 

Zero features written

MyMV-PlanningHTML-NoFeaturesWritten 

Thanks

Sameer

icon

Best answer by virtualcitymatt 15 June 2021, 17:26

View original

4 replies

Userlevel 4
Badge +26

I was able to reproduce your issue. Although in my opinion the FME 2020 functionality was wrong and it has been fixed in FME 2021.

In your VertexCreator_2 the mode it likely set to "Add point" when it should probably be set to "Replace with point".

"Add point" will add an additional point/vertex to the feature, in the case where the input feature is a point this would then create a line feature (with two duplicated points). In FME 2020 a point wasn't added because it was in the same location so the feature remained a point. FME no longer seems to ignore the fact that the point is in the same location and you end up with a line feature.

The solution would be to switch the VertexCreator_2 mode to "Replace with point".

 

As for the ERROR in the log and FME reporting successful, this is because the ERROR was because of a 'Rejection' rather than a real ERROR. The workspace is set to ignore Rejections and continue. If the workbench was set to Terminate on rejection then this would cause a failure.

Badge +10

I was able to reproduce your issue. Although in my opinion the FME 2020 functionality was wrong and it has been fixed in FME 2021.

In your VertexCreator_2 the mode it likely set to "Add point" when it should probably be set to "Replace with point".

"Add point" will add an additional point/vertex to the feature, in the case where the input feature is a point this would then create a line feature (with two duplicated points). In FME 2020 a point wasn't added because it was in the same location so the feature remained a point. FME no longer seems to ignore the fact that the point is in the same location and you end up with a line feature.

The solution would be to switch the VertexCreator_2 mode to "Replace with point".

 

As for the ERROR in the log and FME reporting successful, this is because the ERROR was because of a 'Rejection' rather than a real ERROR. The workspace is set to ignore Rejections and continue. If the workbench was set to Terminate on rejection then this would cause a failure.

@virtualcitymatt​ 

Thanks for providing clarification. Yes what you stated makes sense.

I assumed the Vextex setting was correct as no issues reported prior.

 

In regards to the ERROR I just had not seen it before.

Even the "FailuresByWorkspace" (Dashboards) is returning a warning which wasn't the case.

FME-Server2021-ErrorLogsFailureByWorkspace-Warning 

Regarding Issue 2: Why did the job still complete with no data written, subsequently ending up with a truncated table? I think FME should have terminated the translation with errors no, unless I'm mistaken?

Thanks for your help.

 

Userlevel 4
Badge +26

@virtualcitymatt​ 

Thanks for providing clarification. Yes what you stated makes sense.

I assumed the Vextex setting was correct as no issues reported prior.

 

In regards to the ERROR I just had not seen it before.

Even the "FailuresByWorkspace" (Dashboards) is returning a warning which wasn't the case.

FME-Server2021-ErrorLogsFailureByWorkspace-Warning 

Regarding Issue 2: Why did the job still complete with no data written, subsequently ending up with a truncated table? I think FME should have terminated the translation with errors no, unless I'm mistaken?

Thanks for your help.

 

Interesting, I haven't seen the new UI of FME Server 2021 yet.

 

The data wasn't written to the table because they were all lines and so were skipped because of their geometry type. The table was truncated because the writer is set to truncate the table, but because there was no 'valid' data to fill the table it ended up empty.

 

When it comes to writing, the behaviour is different between formats sometimes. Mostly invalid geometries for a feature type are just skipped with a warning (not skipped). Some formats will cause an ERROR and result in a failed workspace. I'm not sure why there is a difference.

 

In any case, because these skipped geometries were just Warnings FME continued as usual, there were no ERRORS detected which should have cause the workspace to stop. The only 'ERRORS' detected were the ones which were from the Rejected port of the PointOnAreaOverlayer (again because of the line geometries).

 

One thing you can do, is to build in some validation. If you use a FeaureWriter to write to your database you can do some post writing validation checks (perhaps a call to the database count the number of rows). You can then use Terminator transformer to force an error when certain conditions are met (if the number of rows does not match what you expect). This can be a nice way to control and validate which can be important for production workflows.

Badge +10

@virtualcitymatt​ 

Thanks for providing clarification. Yes what you stated makes sense.

I assumed the Vextex setting was correct as no issues reported prior.

 

In regards to the ERROR I just had not seen it before.

Even the "FailuresByWorkspace" (Dashboards) is returning a warning which wasn't the case.

FME-Server2021-ErrorLogsFailureByWorkspace-Warning 

Regarding Issue 2: Why did the job still complete with no data written, subsequently ending up with a truncated table? I think FME should have terminated the translation with errors no, unless I'm mistaken?

Thanks for your help.

 

Thanks Matt

In regards to your suggestion, that's exactly what I was going to do albeit chaining a new workspace doing SQL Count and get notified if table is empty. (Automation)

I'll look into the FeatureWriter option too as it looks more promising with damage limitation.

 

Because FME was used I could not utilise Postgres rollback function as it's not a script.

 

Cheers

Sameer

Reply