Skip to main content
Question

Excel Writer: How to keep edits and append new features


kibailey
Contributor
Forum|alt.badge.img+2

I think what I’m trying to do should be pretty straightforward and that I’m just missing something obvious, so I apologize if this question is not great …

I have a workbench that uses sqlcreator to connect to some data in my db. I do a few transformations and then finish everything up with an excel writer. I’m having an issue configuring the excel writer correctly. 

1.New records are being added to the db, and I occasionally need to rerun the workbench. I want the new records to be appended to original excel output. 2.Also in between runs, I am writing notes and other relevant information in the excel sheet on empty columns I created in the workbench. I want to keep the text I’ve written.

The first time I ran the workbench I had the writer configured to INSERT. No problem. I thought I simply needed to change it to UPDATE and it would start to append new features to rows below the original, but instead I get an error because I don’t have a field to put in the Row Number Attribute field. I tried putting a field that holds a unique ID (messageid), but that just results in an error about the range of rows being incorrect. I’m not sure how much it matters because after looking up what UPDATE is actually supposed to do, I’m no longer sure it will do what I need it to.

Keeping it on INSERT, adds duplicate columns to the excel sheet, so that is obviously a dead end.

Do I need to change the UPDATE configuration and/or excel parameters or do I need some other type of transformer?

Thanks inf advance! 

Note: Not sure if its important but the info in the db is for errors that need to be reprocessed, so when reprocessed correctly the records are removed from the db table. I need to keep old records in the excel.

 

4 replies

ebygomm
Influencer
Forum|alt.badge.img+31
  • Influencer
  • January 9, 2025

Insert is the option you need to append new rows to an existing worksheet. Update would be for updating records within the worksheet already.

 

If you are seeing duplicate columns added to the spreadsheet, i suspect there’s an issue in how your writer is set up with a mismatch in column names


kibailey
Contributor
Forum|alt.badge.img+2
  • Author
  • Contributor
  • January 13, 2025

@ebygomm thanks for the help. Sorry, not duplicate columns, duplicate rows. First run, I may have 650 rows, if i rerun the workbench again before any edits are made to the DB, than I will get another 650, the same 650 rows added to the bottom of the excel sheet. 1300 total

 

If DB gets 3 new records between first and second time running the workbench, then it will be 650+653 for 1303 total. When it should be 653, all unique rows. 

 

If DB drops 3 records between first and second time running the workbench, then it will be 650+ 650 for a total 1300 total. When what I would want is for it to stay at 650, I don’t want it to remove the 3 records that were removed from the DB.

 

Is what I’m aiming for possible?


geomancer
Evangelist
Forum|alt.badge.img+47
  • Evangelist
  • January 14, 2025

You only want to insert the new records from the database as rows in your Excel sheet, and leave the existing rows alone, so you should only pass the new records to the Excel writer.

Therefore you need a way to identify the new records.

One way would be to read the entire Excel sheet, and compare the records from the database with the rows from Excel to determine what records are new.

Or maybe there is a date field you can use: read the last date that is present in the Excel sheet, and change the SQLCreator to read only the records that were created after that date. Something like this could also work with an ID field from the database.

 


kibailey
Contributor
Forum|alt.badge.img+2
  • Author
  • Contributor
  • January 17, 2025

@geomancer thank you. I decided to add a matcher, and the excel file, and the push the features that don’t match into the excel sheet to be updated. Works like a charm. Your suggestions help me get there. 


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings