Question

Can I take features written count from Logfile using regular expression search?

  • 22 October 2021
  • 4 replies
  • 2 views

Badge +3

Sorry Team, This is not one question.

 

I have one fme file, which writes (conventional writers)to 50 PostGIS tables. I need table names and total features written counts from fme.Since I am writing to a staging database, I need to do validation of counts, before making / swapping these tables as the main tables for the application.

 

once I am okay with the previous counts(count from the previous run) and current counts(comparing prev run with the current fme run), we are copying the tables to the main DB.

 

table1 100000

table2 50

table3 6895698

....

like this

to write these info (table_name , total_count)to one more postgres table ie fme_written_status table.

 

I know our feature writer does this job, but I need to replace all the writers.

summary port gives this output with total_features_written.

But not one feature writer, I need 50 separate feature writers. because summary port gives total, not table-wise.

 

can I replace all normal writers to feature writer transformers in one go?

 

can I achieve this using one features writer by grouping table names?

otherwise, fme will open multiple Postgres connections.

 

I am also thinking...

after fme is done, why don't we read the log file(anyway it is a text file) and get the table names and their counts using regex search? Has anyone done this before?

 

 


4 replies

Userlevel 6
Badge +33

This can easily be done using the LogMessageStreamer.

Userlevel 4

The FeatureWriter will not only give you "_total_features_written", but it will also give you a list "_feature_types{}.count" with an associated "_feature_types{}.name" per output feature type. Is this perhaps what you're looking for?

 

Badge +3

This can easily be done using the LogMessageStreamer.

i tried, Logmessagestreamer will not give total written counts.

Userlevel 6
Badge +33

i tried, Logmessagestreamer will not give total written counts.

My bad, I assumed this and did not test before answering.

 

One way to do this is to create a parent workspace with a workspacerunner running the write process followed by a text file reader to read the log and go from there.

Reply