Skip to main content
Solved

Extract Detailed Job messages from FME Flow Successful/Failed Jobs

  • December 19, 2024
  • 4 replies
  • 75 views

sunnywaygis
Contributor
Forum|alt.badge.img+2

Hi All, I want to extract detailed log messages from FME job running on FME Flow (Ver 2021.2.3). What options are available to extract this information along with basic job information like Workspace, Repository, user, source, start and end time. etc. 

Appreciate any feedback. 

Jay 

 

 

 

Best answer by hkingsbury

The easiest way to do this would be to use the HTTPCaller in FME, you can then use a Web Connection and this will handle your authentication. The header should be:

headers = {
    "Authorization": f"fmetoken token={token}",
    "Accept": "application/json"
}

The two endpoints you have there don’t exist, they should be /transformations/jobs/completed and /transformations/jobs/id/{job_id}.

To get the log, you’d then need to call /transformations/jobs/id/{jobid}/log. This will give you the log for the job which you can then parse if needed. Bare in mind that the logs aren’t kept forever (job records are, but no logs). These are cleared based on you System Cleanup configuration

View original
Did this help you find an answer to your question?

4 replies

hkingsbury
Celebrity
Forum|alt.badge.img+53
  • Celebrity
  • December 19, 2024

You can make use of the REST API in behind FME Flow - https://docs.safe.com/fme/html/FME_REST/apidoc/v3/#/

 

You specifically be looking under the transformations group. Likely a combination of these:
 

 


sunnywaygis
Contributor
Forum|alt.badge.img+2
  • Author
  • Contributor
  • December 31, 2024

 

Thanks hkingsbury for your response. I made some good progress using transformation group, when I run this code, I get this 401 response, I am not sure what I am missing here. 

 

Response [401]>
{}

 

import requests

base_url = "https://fmeserver:8443/fmerest/v3"
token = "tert434645646efere3445435"

headers = {
    "Authorization": f"Bearer {token}",
    "Accept": "application/json"
}

try:
    # Get all jobs
    response = requests.get(f"{base_url}/transformations/jobs", headers=headers)
    print(response) 
    jobs = response.json()
    print(jobs)

    # Get detailed information for each job
    for job in jobs:
        job_id = job['id']
        print(job_id)
        job_details = requests.get(f"{base_url}/transformations/jobs/{job_id}", headers=headers)
        print(job_details)
        # Process or store the job details as needed
        print(job_details.json())

except Exception as e:
    # handle it
    print (e)

 


hkingsbury
Celebrity
Forum|alt.badge.img+53
  • Celebrity
  • Best Answer
  • January 5, 2025

The easiest way to do this would be to use the HTTPCaller in FME, you can then use a Web Connection and this will handle your authentication. The header should be:

headers = {
    "Authorization": f"fmetoken token={token}",
    "Accept": "application/json"
}

The two endpoints you have there don’t exist, they should be /transformations/jobs/completed and /transformations/jobs/id/{job_id}.

To get the log, you’d then need to call /transformations/jobs/id/{jobid}/log. This will give you the log for the job which you can then parse if needed. Bare in mind that the logs aren’t kept forever (job records are, but no logs). These are cleared based on you System Cleanup configuration


sunnywaygis
Contributor
Forum|alt.badge.img+2
  • Author
  • Contributor
  • January 8, 2025

 Thanks hkingsbury for your pointers. I was able to come up with code that provides all the messages that I needed to generate. 

I really appreciate your help. 

import requests

base_url = "https://fmeserver:8443/fmerest/v3"
token = "6454574747ghfjhfurturturtu"

headers = {
    "Authorization": f"fmetoken token={token}",
    "Accept": "application/json"
}

try:
    response = requests.get(f"{base_url}/transformations/jobs/completed", headers=headers)
    data = response.json()

    items = data['items']
    for item in items:
        job_id = item['id']
        source_name = item['sourceName']

        job_details = requests.get(f"{base_url}/transformations/jobs/id/{job_id}/log", headers=headers)
        data_job_details = job_details.json()

        data_job_details_items = data_job_details['items']

        for data_job_details_item in data_job_details_items:
            status = data_job_details_item['status']
            if (status == 'WARN'):
                message = data_job_details_item['message']
                print("%s - %s - %s" % (job_id, source_name, message))

except Exception as e:
    print (e)

 

 

 

 


Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings