Skip to main content
Question

Creating ID values from MAX of existing dataset


Hello FME'rs

I have a problem that either I've been staring at for too long or maybe I'm just way off base on. Either way, I can't seem to get the desired output:

I have (2) datasets: an oracle non-spatial table and a xlsx file

I am trying to add rows from my xlsx file to the db table

There is no "ID" field in my xlsx file, so I would like the new rows from the xlsx to populate the "ID" field(s) where the existing table left off (max+1)

I have tried to add the existing table as a reader --> run through a StatisticsCalculator to get "_max" --> through AttributeCreator for the new features, set "ID" to @Value(_max)+1 --> my result is "+1" on all of the "ID" fields instead of a sequential numbering.

What step(s) am I missing?

Thank you for your help!

5 replies

todd_davis
Supporter
Forum|alt.badge.img+21
  • Supporter
  • June 8, 2016

Why don't you just make the id an identity column or use a sequence in Oracle.

If you use a sequence, you can ustilise some advanced table handling in the featuretypes parameters.

If not, and you want to do it all in FME. You need to get the max of the initial table (you are using a statisticscalculator but I would use SQLCreator and write a query to get the max using a SQL statement as that would use the databases power and not read every feature into FME).

You then need to do an unconditional merge to your new features. So that may be a featuremerger with "1" as the value of the requestor and "1" in the value of the supplier

Next comes the counter. In "Count_Start" create an arithmetic function of _max attribute + 1

Then that should give you the next value for the ids, to be written out into the table


pratap
Contributor
Forum|alt.badge.img+11
  • Contributor
  • June 9, 2016

Hi,

I understand that "set "ID" to @Value(_max)+1 --> my result is "+1" means if the max value is 99 then 1st record in excel will be 100 and 2nd record is also 100 instead of 101.

If this is the situation then I propose the following method...

1. Add counter transformer with starting value 1 with local (so each record will have attribute 1st records will have 1 and 2nd record will have 2)

2. Add the _max value to the attribute created in step 1 with attributecreator

Result will be as shown below...

1st record: _max=99 and _counter=1 results 100

2nd record:_max=99 and _counter=2 results 101


gio
Contributor
Forum|alt.badge.img+15
  • Contributor
  • June 9, 2016

You can either use a SQL creator (or executor) to read the table and select a count( ID) as max. Or max(rownum) as max.

Then set a counter and have it start on max+1 where fme_feature_type = "your excel"

Or set a statisticscalculator and get the "Total Count Attribute" for "ID" on your SQL Reader and use the counter and have it start on _count+1 (_count is defaultname in statistics caclulator) where fme_feature_type = "your excel"


Forum|alt.badge.img+5

All of the other 3 answers here are good solutions. Basically you want a SQLExecutor to find the current max value and a Counter to start counting from MaxValue+1

But timing is everything here.

For example, if you put the SQLExecutor in the main data stream, before the Counter, then every feature will trigger it (ie if you have 1000 features you will make 1000 queries to the database). This is not efficient, and there are risks. What happens if the first feature is written to the database before the second feature runs the SQLExecutor? What happens if the Writer locks the database so the SQLExecutor can't read it (will they sit waiting for each other)?

So, to be safe, I would do as @todd_davis suggests. I would place a SQLCreator that queries the database to find the current maximum. Then I would merge that information onto the main features using a FeatureMerger with a fixed (unconditional) merge. Then add the Counter to start counting from MaxValue+1 (use the arithmetic editor in the Count Start parameter).

Hope that helps. Great answers everyone.


Forum|alt.badge.img
  • June 28, 2018
mark2catsafe wrote:

All of the other 3 answers here are good solutions. Basically you want a SQLExecutor to find the current max value and a Counter to start counting from MaxValue+1

But timing is everything here.

For example, if you put the SQLExecutor in the main data stream, before the Counter, then every feature will trigger it (ie if you have 1000 features you will make 1000 queries to the database). This is not efficient, and there are risks. What happens if the first feature is written to the database before the second feature runs the SQLExecutor? What happens if the Writer locks the database so the SQLExecutor can't read it (will they sit waiting for each other)?

So, to be safe, I would do as @todd_davis suggests. I would place a SQLCreator that queries the database to find the current maximum. Then I would merge that information onto the main features using a FeatureMerger with a fixed (unconditional) merge. Then add the Counter to start counting from MaxValue+1 (use the arithmetic editor in the Count Start parameter).

Hope that helps. Great answers everyone.

@mark2catsafe

 

This question is exactly what I am trying to do, I have an exisitng oracle table, I have an ID field "FME_UID" and I am trying to to do as you suggested using the sql creator and feature merger. I have the sql creator working great but I am unsure how to setup the feature merger transformer.

 

 


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings