Skip to main content

We would like to convert a large number of scheduled FME flows we execute today via Autosys.  Autosys has built-in integration with our Incident Management system (ServiceNow).  So if any FME wokspace fails in the flow, it creates an incident in ServiceNow and our on call person is notified by ServiceNow.

Automations in FME Flow have the ability to execute a workspace on failure of a workspace, which could use the ServiceNow API.  However, we have many flows with many workspace executions in each flow.  We run 15 min, 30 min, hrly and daily flows.

What is the recommended best practice to integrate our incident management system into FME Flow.  Some of the items:

  1. When we publish a workspace to flow what are the recommended services (ie Job submitter, Notification Service)?
  2. Flow - in the automation do we connect every workspace failure to a Topic then to a ServiceNow API call workspace to send info like workspace name and error code/msg info too?  Or do we call the ServiceNow API call workspace only and if so where would we get the failure info?

Hello! We don’t currently have an integration with ServiceNow, but we have noted this as a connector that would be beneficial to a number of organizations. Please feel free to email me at joleen.drachenberg@safe.com if you feel comfortable sharing your organization and use cases, so we can investigate this further.


Hi @jbradfor 

When we publish a workspace to flow what are the recommended services (ie Job submitter, Notification Service)?

This depends what the workflow requirements are. The transformation services each deliver results back in different ways. The documentation goes over each service: https://docs.safe.com/fme/html/FME-Flow/ReferenceManual/transformation_svcs.html

For a simple webhook notification where FME sends data to another application, you could probably just use the job submitter service. If using FME Automations to receive data from another application, the translation services can not be used in Automations. This article also goes over different options for workflow integration: https://support.safe.com/hc/en-us/articles/25407774216461-Automating-Workflow-Submission-from-a-Third-Party-Application-To-FME-Flow. Instead, you could set up a webhook trigger to facilitate the integration. 

 

Flow - in the automation do we connect every workspace failure to a Topic then to a ServiceNow API call workspace to send info like workspace name and error code/msg info too?  Or do we call the ServiceNow API call workspace only and if so where would we get the failure info?

Instead of FME Topics, I recommend using Automations with the webhook trigger or schedule trigger. This article demonstrates how to set up an automation that monitors for failed jobs: https://support.safe.com/hc/en-us/articles/28372492971789-Send-Email-Notification-on-Failed-or-Cancelled-Jobs. For more complex requirements, you’ll probably have to review the FME Flow REST API’s transformation requests: https://docs.safe.com/fme/html/FME_REST/apidoc/v3/index.html#/

 

Hope that helps!

Best, 

Sanae


Funny...we are building something similar currently...when we are finding the time. We decided against Topics/automations/System Events, as we would need to build everything slightly differently based on the method that each process was called (schedule, webhook, automation, etc) and the scenarios are more broad. In our case we are building a “big” custom transformer to work with the FME Rest API to monitor jobs and have a range of options based on a range of inputs.

We are very aware that a single failure may not always be a sign of a problem, so we are looking at extra options such as consecutive failures or number of failures in a defined timeline.

But also other thing like:

  • specific parameters are be used
  • number of features output
  • etc

The most annoying problem currently is that the job api queries don’t have ability to bound by a time period in either V3 or V4 of the api, so we are querying a lot of job summaries (not a huge problem, but still more returned than we need)

 


Hi @todd_davis! Interesting use case, thanks for sharing. I really like the idea of time interval parameters for the get /jobs endpoint! I submitted an enhancement request with our development team for it (internal reference is FMEFLOW-23982), but I think it would also be a great candidate for the Ideas page. Please add the idea and use case if you can!

If you want to monitor the FME Flow system at a higher level, I recommend using the enhanced /healthcheck endpoints in V4: https://support.safe.com/hc/en-us/articles/25407381858445-Monitoring-FME-Flow-Availability-with-REST-API-Health-Checks#01J7EH72JEGPTX7Q7VXVBDCTJ3

The new includeDetails parameter returns a status for each FME Flow component could potentially be used in combination with job failures. 

Hope that helps! Thanks, Sanae


Reply