Skip to main content
Question

Read or filter values based on attribute from another source

  • May 2, 2018
  • 5 replies
  • 258 views
M
Forum|alt.badge.img

Hi,

I'm reading a bunch of data from a log database, transforming it and storing it somewhere else. I leave the unique ID attribute of the log message alone.

So I want to make it so my workbench only reads the logging rows I haven't processed yet. The ID numbers are ascending so my idea was to check the destination database using SQL to get the highest ID stored in there, and only read/filter from the source based on that.

Now: How do I do this? I can get the current highest ID processing using the SQLCreator, but how do i read/filter based on that?

Thanks!

This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

5 replies

redgeographics
Celebrity
Forum|alt.badge.img+60
  • redgeographicsCelebrity
  • Celebrity
  • 3703 replies
  • May 2, 2018

You can use a Tester or TestFilter to select the log messages whose ID is higher than the one you got out of the database.

FME rocks! \m/
M
Forum|alt.badge.img
  • mvdorst
  • Author
  • 17 replies
  • May 2, 2018

But when input the SQLCreator and the log database into a tester, there is on row the the max value to test against and a lot of logging rows the ID values. Comparing them doesn't work.

M
Forum|alt.badge.img
  • mvdorst
  • Author
  • 17 replies
  • May 2, 2018

You can use a Tester or TestFilter to select the log messages whose ID is higher than the one you got out of the database.

But when input the SQLCreator and the log database into a tester, there is on row the the max value to test against and a lot of logging rows the ID values. Comparing them doesn't work.

 

 

redgeographics
Celebrity
Forum|alt.badge.img+60
  • redgeographicsCelebrity
  • Celebrity
  • 3703 replies
  • May 2, 2018

If you do a featurejoin you can add the id from the database to all records

FME rocks! \m/
takashi
Celebrity
  • takashiCelebrity
  • 7843 replies
  • May 5, 2018

If the two database tables belong to the same database, I think you can extract only required records from the source table with a single SQL statement something like this.

select * from source where (select max(id) from destination) < id

Alternatively, you can also set this expression to the WHERE Clause parameter in the reader which reads the source table.

(select max(id) from destination) < id
Why not inspect features with Visual/Data Preview and Feature/Record Information before writing them into a destination dataset?

Community Stats

32,086
Posts
121,848
Replies
39,729
Members

Latest FME

Terms & ConditionsCookie settingsAccessibility statement
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.