Question

Whats the BEST way to update values when matched data changes?

  • 10 June 2021
  • 4 replies
  • 19 views

Badge +3

I find it REALLY difficult to use FME for feature change detection.

I'm using an api to grab weather data and store it in a table. Occasionally the api might send an updated value for one of the records.

So I might have STATION_NAME, DATETIME and VALUE from the api to match against the existing record.

I can use the FEATURE_JOINER to join the STATION_NAME and DATETIME, but then need to update the value.

Any idea what the best method is? I have tried the FEATURE_MERGER and CHANGE_DETECTOR, but neither seem to be simple to set up or understand...


4 replies

Userlevel 6
Badge +31

The best way is hard to say without knowing the details so the answer is "it depends"...

  • If you have a small volume of data to test against a small volume of data stored, which I think is your situation, I would load both sets and use the ChangeDetector.
  • If you have a small volume of data to test against a large volume of data stored, I would try to make use of database functionality and only load records where STATION_NAME and DATETIME matches to test for changes.

Small can vary but loading a large dataset for processing really can bog down you process. And if you need to check for changes every hour, and the process runs for more than an hour you need to change your strategy.

 

I think the ChangeDetector is not that hard to work with. When learning something new it helps to generate a small test environment. For example two datasets having one unchanged, one updated, one deleted and one updated record. Feed them both to the ChangeDetector and see if it works as expected. I attached a workspace demonstrating this.

Badge +3

Hi Niels,

 

Thank you - I have a small amount of data to check against a large volume of stored data. It's indexed, but takes a long time to check. I'm currently using a Feature_Joiner and have attached a Database Updater to the Joined port and a Writer to the Unjoined Left port.

Cheers for the sample workflow - that is a great help as I find the terminology around the ChangeDetector mightily confusing!

Badge +2

@nedwaterman​ These two webinars might give you some ideas on how to solve your change detection problem:

 

Badge +3

@nedwaterman​ These two webinars might give you some ideas on how to solve your change detection problem:

 

Thanks Mark - I will take a look!

Reply