Question

Data check in workbench

  • 8 February 2022
  • 5 replies
  • 1 view

Badge +10

Hello. I have a question. Does anyone know how to check inside the workspac that the workbench has processed all the data at the beginning of the WS? It happened to me several times that during a large WS the workbench looked like it was finished properly, but not all the data was retrieved during the data transformation. I came up with a counter at the beginning and end of the WS, but it's probably not the perfect solution. Another thing is that the counter reads over and over again, if WS is automated, I would need to somehow reset it when it's done. I need something that will be obvious at first glance, the search in the log will take me a long time.

 

Thanks a lot


5 replies

Badge +2

You should be able to get this information from the translation log in the Features Read Summary and Features Written Summary sections at the bottom of the log file - you can compare how many features were read in and written out:

imageIf you don't want to manually parse/search the log file, you can use FME for that too by utilizing the WorkspaceRunner transformer and reading the .log file that is produced with a CSV reader (it's just a pipe-delimited CSV):

imageYou could also write this to a separate file if you wanted to keep a history table of the job details (i.e. read in, written out, start/end time, etc.).

Badge +2

@fmesafe.podpora​ If you're reading from a database, you can add a COUNT(*) SQL query in a SQLCreator or SQLExecutor. Something like:

SELECT COUNT( *) as "Number of Rows"
FROM mytable;

But ideally we should try and track down why you think FME is not reading all the records from your source data

 

Badge +10

You should be able to get this information from the translation log in the Features Read Summary and Features Written Summary sections at the bottom of the log file - you can compare how many features were read in and written out:

imageIf you don't want to manually parse/search the log file, you can use FME for that too by utilizing the WorkspaceRunner transformer and reading the .log file that is produced with a CSV reader (it's just a pipe-delimited CSV):

imageYou could also write this to a separate file if you wanted to keep a history table of the job details (i.e. read in, written out, start/end time, etc.).

printThank you for answer. I understand your print screen, but it's not so clear to me, an error somewhere in the settings? Total features read in log I should have 12 020, I only have one. I would need to perform an element level check. I don't know if I'm explaining it correctly.

Badge +10

It would be enough if I could check the number of input elements at the input ws and the number of written elements at the output ws..

Badge +10

@fmesafe.podpora​ If you're reading from a database, you can add a COUNT(*) SQL query in a SQLCreator or SQLExecutor. Something like:

SELECT COUNT( *) as "Number of Rows"
FROM mytable;

But ideally we should try and track down why you think FME is not reading all the records from your source data

 

Thanks for the reply. Can you please specify? 

Reply