Incremental extract of Intergraph data to Smallworld

Depending on the amount of data (and only suitable with a smaller amount) you could read data from both Intergraph and Smallworld and use the ChangeDetector transformer (on a limited number of attributes and/or the geometry) to determine the differences.

Then use the attribute fme_db_operation (INSERT or DELETE) to write the differences to Smallworld.

I expect this would take too much resources for a large dataset, but would work for a smaller set.

Thanks Erik, but I am looking to get an incremental update of about 150 substations (1300 circuits), so I am not sure that would work. Depending on how fast FME server can extract the data each night, we could potentially load it into Smallworld in a sub-alternative and then do a difference stream to post the data up.

I do agree with @erik_jan's answer on ChangeDetector for smaller datasets (because of how FME loads the data into memory).

There's some other options for change detection here that might give you some ideas: https://knowledge.safe.com/articles/33899/change-detection-2.html

I don't have experience with Intergraph, so I can't comment on timestamp and versions on your source, which would be the way to go if it's an option. If it's not, what about running multiple FME instances through the ChangeDetector for smaller subsets of your dataset, such as by particular circuit or asset type? Perhaps you could have multiple workspaces or a generic workbench you call through the FMEServerJobSubmitter that checks smaller subsets of data, so you aren't identifying changes on everything at once.

Erik, I believe they have GTech, in which case we can use the modifications dates, but then how can we tell which geometries have changed? Are those stored somehow in Oracle too? I thought the DGN files were used for that, no?

Hi Robert, GTech stores the geometries in Oracle Spatial tables.

So, that should not be the issue.

Fantastic! Thanks for the help Erik.