Skip to main content
  • EN
  • Home
  • Forums
  • FME Flow
  • Automation
  • Is there a way to automate build process for FME workspaces? We want to start using a repository (like GIT), have an automated process to pick FME things - workspaces, schedules, subscriptions, files, etc. from repository and deploy to Uat and produc

Hi Everyone,

I was hoping to find a definitive guide explaining how to use CI/CD automation with FME.

Right now we have very few workspaces and so we use a manual process to deploy FME things in production. We use Publications, Schedules, Subscriptions, workspaces, data files (for passwords), database connections, etc.

I looking for instructions to help with following:

  1. Integration with a source control like GIT so that we have versioning set up
  2. A guide/tutorial that can explain how to automate deployment of all things FME. If something cannot be deployed via automation, I would like to know that as well.
  3. Any pointers on automated testing of FME workspaces/schedules once they are deployed in an environment.

We use gitlab internally for our pipelines and I've been trying to create a CI/CD pipeline for a couple of our FME Server based products. It hasn't been very easy to be honest but does work...but was pretty hacky.

 

You wont find a definitive guide here as it's really tricky to work with FME and git. Every time you save a workspace it gets changed significantly so the idea of merging changes across multiple branches doesn't really work. This is fine if you are working on your own workspaces, however, as soon as you make a two branches and make different modifications you run into trouble trying to merge them.

I have heard that in FME 2022 there is a new workspace change detection tool where you can perform a visual review of changes - this is a big step in the right direction but still doesn't solve the issue of working across multiple branches as far as I can tell.

 

As far as CI/CD goes I guess it depends a lot on what you're looking to achieve. For us we deliver several components including manuals, formats, custom transformers and an FME Server Project with it's own set of components like publishers etc. Each project component is stored as a JSON file accessible by the build pipeline.

 

For automatic versioning in the workspace(s) I found it pretty tricky. I've created a private parameter which I tick each time a new build is made but it's very hacky.

 

In the pipeline I essentially use a workspace to build the FME Server project using the required components with the FME Server REST API. The project is then imported into a fresh FME Server docker instance where I can make calls REST called to perform various automated tests.

 

FME Server does have some built mechanisms to connect to git for version, however, it's only for workspaces really and not for projects as far as I can tell: https://docs.safe.com/fme/html/FME_Server_Documentation/WebUI/Version-Control.htm

 

I've submitted this CI/CD stuff as an abstract for the UC this year because I'm really interested to see how others are doing it.

In 2017 there was this talk which is pretty interesting: https://www.safe.com/presentations/working-with-fme-in-an-agile-software-development-lifecycle/

@sseth2011​ Did you use FME Server projects for this or just the api?

 

@virtualcitymatt​ - what parts were the most hacky? How did you resolve issues with connections? Our goal is to automate our deployments between DTAP environments.

Reply


Community Stats

30,352
Posts
115,344
Replies
38,087
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.