How can I extract total transfer time from "http caller" transformer

The other way (without python) would be to parse the logfile, but admittedly the method provided by @david_r is much neater!

Thanks to David and Itay. It worked. Great!!

I used David's method. mmuuaahh

Before writing to you i used this python, (ha ha... this is very basic method)

from datetime import datetime
def FeatureProcessor(feature):
    t11 = feature.getAttribute("time1")    
    t21 = feature.getAttribute("time2") 
    t1 = datetime.strptime(t11, "%Y%m%d%H%M%S")
    t2 = datetime.strptime(t21, "%Y%m%d%H%M%S")
    t2 - t1
    # datetime.timedelta(0, 11102)
    day_delta=(t2 - t1).seconds
    feature.setAttribute("Time_taken", day_delta)





here i took two timestamper , before and after httpcaller and find the difference
in seconds. But i feel yours is better way. Thanks again Mr David
   
   

I wonder if syntax has changed for the solution.

I wanted to get the ProcessID from the message that contains START and I get an error on ProcessFeature.

Here's my error message

My workspace is basically a Creator with one object, a PythonCaller and an inspector.

Here's my code for the startup Python script

import fmeobjects
import re

def fme_log_callback(severity, str):
     match = re.search(r"ProcessID: (\d+)", str)
     if match:
        global processID
        processID = match.groups()[0]

fmelog = fmeobjects.FMELogFile()
fmelog.setCallBack(fme_log_callback)

Here's my code in the Python Caller.

import fmeobjects

def GetProcessID(feature):
global processID

    if processID:
         feature.setAttribute('pid', processID)

I'd also like to get from the log, messages that contain "FME Session Duration" and the "END". From each line, I'll parse off the translation details that I need to store as translation history. Would the Callback method work for the END log message? @david_r, your answer in this thread is so far the best and closest solution I have found so far.

In the modern FME, the LogMessageStreamer could be an option.

e.g.

@takashi thanks for your response. I am already using the LogMessageStreamer for my httpcallers - I have 120 of these httpcallers in my workspace and I'm capturing the total transfer time for each transformer. I need the history log of the entire translation. However, as far as I know, LogMessageStreamer does not capture the log streams from the beginning and the end of the translation where START, FME Session Duration and END are. Perhaps I should create a new question, since this is not really the same. The python solution is probably the answer but I can't seem to make it work.

I can relate to what @itay said about documentation and as far as Safe's examples, they have a Nov 2019 blog update with three samples using callback from @daveatsafe. To your point David on how callbacks can be confusing, in spite of these samples from Safe, I still can't accomplish what I needed to do to retrieve three specific log messages into my translation. So I'm relying on your sample and callback documentation to solve my problem. Not there yet, so any ideas are welcome.

import fmeobjects
import re

def fme_log_callback(severity, str):
     match = re.search(r"ProcessID: (\d+)", str)
     if match:
        global processID
        processID = match.groups()[0]

fmelog = fmeobjects.FMELogFile()
fmelog.setCallBack(fme_log_callback)

import fmeobjects

def GetProcessID(feature):
global processID

    if processID:
         feature.setAttribute('pid', processID)

Two things that jump out to me:

1. In the first block, if 'match' is undefined, then 'processID' will never be defined and you'll get the error in the screenshot
2. In the second block, there's a indentation error on line 4, but perhaps it's only a forum issue?

import fmeobjects
import re

def fme_log_callback(severity, str):
     match = re.search(r"ProcessID: (\d+)", str)
     if match:
        global processID
        processID = match.groups()[0]

fmelog = fmeobjects.FMELogFile()
fmelog.setCallBack(fme_log_callback)

import fmeobjects

def GetProcessID(feature):
global processID

    if processID:
         feature.setAttribute('pid', processID)

The main issue is that when the Python startup script executes, the process ID has already been written to the log file, meaning that those lines will never be captured by either the callback function or the LogMessageStreamer (which uses the callback internally).

If you need to extract the process ID, you could consider reading and parsing the log file in the startup script, e.g.

import fme
with open(fme.logFileName, 'r') as f:
    contents = f.read()
    # use regex to search for process ID in contents

import fme
with open(fme.logFileName, 'r') as f:
    contents = f.read()
    # use regex to search for process ID in contents

Thanks for stating the main issue. The indention is a forum issue.

Thanks also for the code. More of a python issue at this point. I really need to retrieve a few lines of the log and bring it to my workspace to process. I got it as far as retrieving the line but I still need a few more answered questions to get it done.

Hi @salvaleonrp,

Thanks for your question! I almost missed your post here as I realized this Q&A is from quite some time ago and already has an accepted answer. For better visibility that will get you some more help, would you be open to posting a new question to the Forums with your issue, what you've tried, and where you're stuck? You can always link back to this question for context.