Solved

Query a workspace for resource dependencies


Badge

I am trying to publish workspaces via the REST api. Ideally I would like to programmatically get the workspaces dependencies before publishing to make sure the resources are on the server. When I manually publish via Desktop I get a warning about dependencies, I would like to access this info before publishing with REST. I am making the REST calls with Python so if I had a way to get the dependencies with Python that would be real neat.

icon

Best answer by jlutherthomas 7 November 2017, 20:26

View original

17 replies

Badge +2

Hi @2dews

 

 

We don't actually support publishing a workspace via the REST API at the moment, however this is something we'd like to do in the future. Also determining all of the workspace dependencies, and FME Server object dependencies (connections, topics, etc) is something we'd like to be able to do.

 

You can add this as an idea here.
Badge

Hi @2dews

 

 

We don't actually support publishing a workspace via the REST API at the moment, however this is something we'd like to do in the future. Also determining all of the workspace dependencies, and FME Server object dependencies (connections, topics, etc) is something we'd like to be able to do.

 

You can add this as an idea here.
I am confused. What does this call do if not publish a workspace and the like?

 

 

Badge +2
I am confused. What does this call do if not publish a workspace and the like?

 

 

 

Hi @2dews

 

In the call it has "Note: This version of the FME Server REST API does not replicate all of the functionality available in the Publish to FME Server wizard of FME Workbench. For more information, see "Publishing to FME Server" in the FME Desktop Help (https://docs.safe.com/fme/html/FME_Desktop_Documentation/FME_Workbench/Workbench/publishing_to_fme_server.htm)"

 

It's hard to tell going through the api docs and the desktop docs where the difference is, and what functionality is missing. One of those could well be the dependencies issue that you're referring to.

 

It also looks like things like web or database connections need to already be published to FME Server and referred to during this call, where as during publish from desktop I believe it also publishes the web connections.

 

Also with services it looks like you can specify the service, but no further settings around it.

 

 

However, I have not tested this REST call myself, I would definitely recommend using the desktop wizard to be on the safe side. I've asked my colleagues if any of them have tested this REST call.

 

Aside from dependencies, have you successfully got this call working to publish and run a workspace?
Userlevel 4

Hi @2dews

 

 

We don't actually support publishing a workspace via the REST API at the moment, however this is something we'd like to do in the future. Also determining all of the workspace dependencies, and FME Server object dependencies (connections, topics, etc) is something we'd like to be able to do.

 

You can add this as an idea here.
Hi Jennifer, the API v3 has the method

 

POST /repositories/<repository>/items
which according to the documentation "Publishes a single [file] item to a repository." Does that not include uploading .fmw files? Or is that method meant for republishing only?

 

 

The documentation does say that This version of the FME Server REST API does not replicate all of the functionality available in the Publish to FME Server wizard of FME Workbench, which is great, but it would be ideal if the differences were desceribed in a bit more detail.
Badge

Hi @2dews

 

 

We don't actually support publishing a workspace via the REST API at the moment, however this is something we'd like to do in the future. Also determining all of the workspace dependencies, and FME Server object dependencies (connections, topics, etc) is something we'd like to be able to do.

 

You can add this as an idea here.
I don't know what is missing but we are publishing complete projects (Users, Roles, Resources, Repositories, Workspaces including registration to services and failure and success topics, Topics, Subscriptions, Schedules and all the related permissions) to FME Server using the REST API v2. The only thing we are unable to do is create shared connections thus they have to exists on FME Server before we publish.

 

 

Badge

Hi @2dews

 

 

We don't actually support publishing a workspace via the REST API at the moment, however this is something we'd like to do in the future. Also determining all of the workspace dependencies, and FME Server object dependencies (connections, topics, etc) is something we'd like to be able to do.

 

You can add this as an idea here.
I can successfully publish the workspaces with POST /repositories//items, I then just have to make another call to register the workspace with the desired services. It works great so long as I have the resources on the server first. As we move to a Docker environment I need a way to autonomously rebuild our Server from scratch. If there is no programmatic way to determine the workspace dependencies I will have to require our workspace authors to maintain a known directory of resources that will get pushed every time we publish workspaces.

 

 

Badge

Hi @2dews,

in the FME Server REST API v3 we added an additional endpoint for Projects. This still doesn't let you query dependencies of workspaces, but it allows you to bundle a workspace with all parts the workspace needs to run, and move it as a Project. Here's the REST endpoint:

 

https://docs.safe.com/fme/html/FME_REST/apidoc/v3/#!/projects

Not sure if this solves your problem, but I think it's worth to take a look at.

@larry this might be interesting for your workflow too. Maybe it makes a few things easier compared to REST API v2.

Badge

Hi @2dews,

in the FME Server REST API v3 we added an additional endpoint for Projects. This still doesn't let you query dependencies of workspaces, but it allows you to bundle a workspace with all parts the workspace needs to run, and move it as a Project. Here's the REST endpoint:

 

https://docs.safe.com/fme/html/FME_REST/apidoc/v3/#!/projects

Not sure if this solves your problem, but I think it's worth to take a look at.

@larry this might be interesting for your workflow too. Maybe it makes a few things easier compared to REST API v2.

@GerhardAtSafe This is nice but we still have the need to create the first publication on the first server before being able to create/migrate a project.

 

Badge +2
So, @2dews and @larry I've got the scoop on what the differences are between REST API and FME Publishing Wizard (I will edit my previous answer)

 

 

You can publish to FME Server using the REST API, however this isn't something that we recommend, as the publishing wizard is doing additional stuff to the REST API.

 

 

What the FME Desktop Publishing Wizard can do, that the REST API can't:

 

  • Source dataset paths are updated if files are uploaded to FME Server
  • Named connections may be renamed when uploaded to FME Server, due to character restrictions. This also includes named connections used in web-as-a-filesystem workflows
  • Service registration options are cached within the workspace so they can be used in a future publish. They don't affect the workspace behaviour itself
  • Disable destination redirects (to visualizer or to FFS)
  • Cache last published repository name and workspace name

So the publishing wizard has methods to search the workspace for all of the workspace dependencies that are part of the publishing process, however this is not available through an API call. So if you use the REST API to publish a workspace, you will need to tell the API everything it will need to know to run the workspace as you want. The API calls out to FME Server, where in this case the API wouldn't be valid as it needs to look at a workspace before it's published.

 

 

Because of this, we do not recommend using the REST API to publish workspaces to FME Server, as it isn't guaranteed that it will behave as expected compared to publishing through the Publishing Wizard. So I wouldn't be surprised if you get situations where it doesn't work correctly.
Badge
So, @2dews and @larry I've got the scoop on what the differences are between REST API and FME Publishing Wizard (I will edit my previous answer)

 

 

You can publish to FME Server using the REST API, however this isn't something that we recommend, as the publishing wizard is doing additional stuff to the REST API.

 

 

What the FME Desktop Publishing Wizard can do, that the REST API can't:

 

  • Source dataset paths are updated if files are uploaded to FME Server
  • Named connections may be renamed when uploaded to FME Server, due to character restrictions. This also includes named connections used in web-as-a-filesystem workflows
  • Service registration options are cached within the workspace so they can be used in a future publish. They don't affect the workspace behaviour itself
  • Disable destination redirects (to visualizer or to FFS)
  • Cache last published repository name and workspace name

So the publishing wizard has methods to search the workspace for all of the workspace dependencies that are part of the publishing process, however this is not available through an API call. So if you use the REST API to publish a workspace, you will need to tell the API everything it will need to know to run the workspace as you want. The API calls out to FME Server, where in this case the API wouldn't be valid as it needs to look at a workspace before it's published.

 

 

Because of this, we do not recommend using the REST API to publish workspaces to FME Server, as it isn't guaranteed that it will behave as expected compared to publishing through the Publishing Wizard. So I wouldn't be surprised if you get situations where it doesn't work correctly.
That's unfortunate. We will continue on the path of publishing with the REST api though. We are looking to move towards a continuous integration setup where our workspaces are published to server and tested when checked into Git. Not to mention we want to be able to teardown and rebuild our server completely from code. I would rather write more code to make sure we push everything to the server than require someone to sit down and publish 25 workspaces when we make updates.

 

 

Badge +2
That's unfortunate. We will continue on the path of publishing with the REST api though. We are looking to move towards a continuous integration setup where our workspaces are published to server and tested when checked into Git. Not to mention we want to be able to teardown and rebuild our server completely from code. I would rather write more code to make sure we push everything to the server than require someone to sit down and publish 25 workspaces when we make updates.

 

 

You can also add any ideas you have to our ideas exchange which is a good way to see what new users want in the product, and how many users want this.
Badge
That's unfortunate. We will continue on the path of publishing with the REST api though. We are looking to move towards a continuous integration setup where our workspaces are published to server and tested when checked into Git. Not to mention we want to be able to teardown and rebuild our server completely from code. I would rather write more code to make sure we push everything to the server than require someone to sit down and publish 25 workspaces when we make updates.

 

 

Same here. We are able to rebuild our FME Server from scratch (9 roles, 9 users, 5 resources folders containing 3766 resources files, 9 custom transformers and python files, 9 repositories and permissions, 17 topics and permissions, 17 email subscriptions with HTML templates, 21 workspaces and 1 schedule) in a few minutes using REST while it was taking hours in the past. This allowed us to implement continuous integration and automatically rebuild our server and run a test suite every night.

 

 

Badge
Same here. We are able to rebuild our FME Server from scratch (9 roles, 9 users, 5 resources folders containing 3766 resources files, 9 custom transformers and python files, 9 repositories and permissions, 17 topics and permissions, 17 email subscriptions with HTML templates, 21 workspaces and 1 schedule) in a few minutes using REST while it was taking hours in the past. This allowed us to implement continuous integration and automatically rebuild our server and run a test suite every night.

 

 

@larry If you have any blog posts, GitHub repos, or other material highlighting your CI or tests it would be great to see. Sounds like you are already at where we are hoping to get to.

 

Badge
@larry If you have any blog posts, GitHub repos, or other material highlighting your CI or tests it would be great to see. Sounds like you are already at where we are hoping to get to.

 

Nothing public so far. We are using Subversion for source control.
Userlevel 4
@larry If you have any blog posts, GitHub repos, or other material highlighting your CI or tests it would be great to see. Sounds like you are already at where we are hoping to get to.

 

I agree, this sounds to me like it would make a killer blog post. Maybe get in touch with @Mark2AtSafe to see if there's interest?
Badge
Same here. We are able to rebuild our FME Server from scratch (9 roles, 9 users, 5 resources folders containing 3766 resources files, 9 custom transformers and python files, 9 repositories and permissions, 17 topics and permissions, 17 email subscriptions with HTML templates, 21 workspaces and 1 schedule) in a few minutes using REST while it was taking hours in the past. This allowed us to implement continuous integration and automatically rebuild our server and run a test suite every night.

 

 

@larry do you use REST to set up database connections? I am having trouble figuring that out.

 

Badge
@larry do you use REST to set up database connections? I am having trouble figuring that out.

 

No. We are setting them up manually on each server prior to deploy.

 

 

Reply