The LogNormalizer custom transformer is designed to read and normalize log files.
It attempts to extract and standardize key information such as the timestamp and severity level, producing structured attributes that can be used for further processing in FME.
Originally developed to handle FME Flow log files, it can also be applied to log files from other applications.
How It Works
Each input feature triggers the parsing of a log file.
Typically, it is combined with the Directory and File Pathnames reader to list all log files within a directory and its subdirectories.
The transformer uses different regular expressions to detect:
The severity level of each log line
The timestamp format
When a timestamp is found, it is converted and normalized into both FME and ISO 8601 formats.
If no timestamp is detected for a line, the transformer attempts to reuse the timestamp of the previous line and marks it as "unknown".
Parameters
Log File(s) to Parse
Type: File path
Description: Path(s) to the log file(s) to be parsed.
Output Attributes
Each log line is output as a feature with the following attributes:
Attribute
Description
_log.timestampFME
Timestamp converted to FME datetime format
_log.timestampISO
Timestamp converted to ISO 8601 format
_log.isDateTimeKnown
yes / no – indicates whether a timestamp was detected for this line. If no, the timestamp is inherited from the previous line.
_log.severity
Detected severity level: INFO, WARN, ERROR, or UNKNOWN
_log.message
The log message text, without the timestamp
_log.dataset
The source log file from which the line originates
Typical Usage
Normalize FME Flow log files for further analysis
Parse third-party application logs and extract structured information
Combine with readers like Directory and File Pathnames to batch-process multiple log files
Would you like to know more? Click here to find out more details!