I am looking to do an initial extract of a utility customer's Intergraph data to Smallworld using FME workbench. Is there a way to do an incremental extract on a nightly basis, without completely reloading the entire database?
In other words, is there a way to identify daily changes in Intergraph data (ie time stamp, versions, etc) ?
@MarkAtSafe
Depending on the amount of data (and only suitable with a smaller amount) you could read data from both Intergraph and Smallworld and use the ChangeDetector transformer (on a limited number of attributes and/or the geometry) to determine the differences.
Then use the attribute fme_db_operation (INSERT or DELETE) to write the differences to Smallworld.
I expect this would take too much resources for a large dataset, but would work for a smaller set.
Thanks Erik, but I am looking to get an incremental update of about 150 substations (1300 circuits), so I am not sure that would work. Depending on how fast FME server can extract the data each night, we could potentially load it into Smallworld in a sub-alternative and then do a difference stream to post the data up.
What Intergraph system are you referring to (FRAMME, GTECH, GeoMedia)?
If I recall right:
FRAMME has no modification history as it stores geometry in DGN
GTECH (Oracle tables) has modification dates, that can be used
GeoMedia has a system table called ModificationLog that can be used.
Hope this helps.
I do agree with @erik_jan's answer on ChangeDetector for smaller datasets (because of how FME loads the data into memory).
There's some other options for change detection here that might give you some ideas: https://knowledge.safe.com/articles/33899/change-detection-2.html
I don't have experience with Intergraph, so I can't comment on timestamp and versions on your source, which would be the way to go if it's an option. If it's not, what about running multiple FME instances through the ChangeDetector for smaller subsets of your dataset, such as by particular circuit or asset type? Perhaps you could have multiple workspaces or a generic workbench you call through the FMEServerJobSubmitter that checks smaller subsets of data, so you aren't identifying changes on everything at once.
What Intergraph system are you referring to (FRAMME, GTECH, GeoMedia)?
If I recall right:
FRAMME has no modification history as it stores geometry in DGN
GTECH (Oracle tables) has modification dates, that can be used
GeoMedia has a system table called ModificationLog that can be used.
Hope this helps.
Erik, I believe they have GTech, in which case we can use the modifications dates, but then how can we tell which geometries have changed? Are those stored somehow in Oracle too? I thought the DGN files were used for that, no?
Erik, I believe they have GTech, in which case we can use the modifications dates, but then how can we tell which geometries have changed? Are those stored somehow in Oracle too? I thought the DGN files were used for that, no?
Hi Robert, GTech stores the geometries in Oracle Spatial tables.
So, that should not be the issue.
What Intergraph system are you referring to (FRAMME, GTECH, GeoMedia)?
If I recall right:
FRAMME has no modification history as it stores geometry in DGN
GTECH (Oracle tables) has modification dates, that can be used
GeoMedia has a system table called ModificationLog that can be used.
Hope this helps.
Fantastic! Thanks for the help Erik.