Hello,
I am performing a data stream in FME Flow that reads two independent datasets with the same schema from the BC Wildfire Service’s (BCWS) Public ArcGIS Online Feature Service. These datasets are the BCWS current fires found on their Wildfire Dashboard:
- Known fire start locations (points)
- Wildfire perimeters (polygons)
What my data stream does:
- (Read) Read both points and polygons of current wildfires, also read a previous copy of both datasets merged as a PostGIS table →
- (FeatureJoiner) Join and overwrite points where polygons exist →
- (ChangeDetector) Detect major changes from the PostGIS table: point-to-polygon updates, polygon area increases, and fire status indicators →
- (Write) Write all changes to the saved copy in PostGIS, and write the major changes to a PostgreSQL update logging table.
This workflow runs fine when performed manually. However I noticed when I apply the workbench to run as a data stream, my logging table will go crazy every once in a while and log every feature in the dataset being changed from POINT to POLYGON or from POLYGON to POINT.
My suspicions on the source of the problem:
I previously ran this with a database trigger suspecting that FME was writing to the table in a way that the trigger didn’t like, but when I replaced the trigger with the log writer, the same issues occurred.
I suspect that one or both Feature Service readers runs into a bad request every once in a while and gets skipped over, leading to the FeatureJoiner not properly overwriting geometry, and writing this change to the PostGIS table and the logging table.
Questions:
Is there a way to detect when an ArcGIS Feature Service reader does not read the data as intended? Other than turning off geometry monitoring in my workflow, are there any potential fixes I could try?
SCREENSHOTS:
