+2Hi All, I want to extract detailed log messages from FME job running on FME Flow (Ver 2021.2.3). What options are available to extract this information along with basic job information like Workspace, Repository, user, source, start and end time. etc.
Appreciate any feedback.
Jay
Best answer by hkingsbury
The easiest way to do this would be to use the HTTPCaller in FME, you can then use a Web Connection and this will handle your authentication. The header should be:
headers = {
"Authorization": f"fmetoken token={token}",
"Accept": "application/json"
}The two endpoints you have there don’t exist, they should be /transformations/jobs/completed and /transformations/jobs/id/{job_id}.
To get the log, you’d then need to call /transformations/jobs/id/{jobid}/log. This will give you the log for the job which you can then parse if needed. Bare in mind that the logs aren’t kept forever (job records are, but no logs). These are cleared based on you System Cleanup configuration
+63You can make use of the REST API in behind FME Flow - https://docs.safe.com/fme/html/FME_REST/apidoc/v3/#/
You specifically be looking under the transformations group. Likely a combination of these:
+2
Thanks hkingsbury for your response. I made some good progress using transformation group, when I run this code, I get this 401 response, I am not sure what I am missing here.
Response [401]>
{}
import requests
base_url = "https://fmeserver:8443/fmerest/v3"
token = "tert434645646efere3445435"
headers = {
"Authorization": f"Bearer {token}",
"Accept": "application/json"
}
try:
# Get all jobs
response = requests.get(f"{base_url}/transformations/jobs", headers=headers)
print(response)
jobs = response.json()
print(jobs)
# Get detailed information for each job
for job in jobs:
job_id = job['id']
print(job_id)
job_details = requests.get(f"{base_url}/transformations/jobs/{job_id}", headers=headers)
print(job_details)
# Process or store the job details as needed
print(job_details.json())
except Exception as e:
# handle it
print (e)
+63The easiest way to do this would be to use the HTTPCaller in FME, you can then use a Web Connection and this will handle your authentication. The header should be:
headers = {
"Authorization": f"fmetoken token={token}",
"Accept": "application/json"
}The two endpoints you have there don’t exist, they should be /transformations/jobs/completed and /transformations/jobs/id/{job_id}.
To get the log, you’d then need to call /transformations/jobs/id/{jobid}/log. This will give you the log for the job which you can then parse if needed. Bare in mind that the logs aren’t kept forever (job records are, but no logs). These are cleared based on you System Cleanup configuration
+2Thanks hkingsbury for your pointers. I was able to come up with code that provides all the messages that I needed to generate.
I really appreciate your help.
import requests
base_url = "https://fmeserver:8443/fmerest/v3"
token = "6454574747ghfjhfurturturtu"
headers = {
"Authorization": f"fmetoken token={token}",
"Accept": "application/json"
}
try:
response = requests.get(f"{base_url}/transformations/jobs/completed", headers=headers)
data = response.json()
items = data['items']
for item in items:
job_id = item['id']
source_name = item['sourceName']
job_details = requests.get(f"{base_url}/transformations/jobs/id/{job_id}/log", headers=headers)
data_job_details = job_details.json()
data_job_details_items = data_job_details['items']
for data_job_details_item in data_job_details_items:
status = data_job_details_item['status']
if (status == 'WARN'):
message = data_job_details_item['message']
print("%s - %s - %s" % (job_id, source_name, message))
except Exception as e:
print (e)