Skip to main content
  • EN

Hey FME'ers. I've not been online for a bit recently (because of updating the FME training materials for 2017) but I wanted to throw out a new idea that is part challenge, but part crowdsourcing.

I came up with the idea of an FME-driven service for assessing a workspace for best practices. Just now I found I wasn't the first to think of this, which shows there aren't any new ideas any more!

Anyway, in my spare time I've been putting together a workspace using the FMW reader to test other workspaces for best practice. I've built about 20 different tests and can think of quite a few more. But I thought it would be great if we - the FME community - could work together to take this idea to completion.

So, this is an open invite to take part in this project. I think there are a number of different ways you could contribute:

  • Develop a best practices test and add it to the project workspace
  • Do some testing on the project workspace to look for faults or enhancements
  • Improve the output style (it's currently fairly basic HTML)
  • Suggest some ideas for tests that we haven't yet thought of or implemented
  • Use the project to assess your own workspaces - and let us know what you think

I've shared all the files in a folder on Dropbox (now find it on GitHub instead). Anyone in the FME community is welcome to access this and use the contents for whatever you like.

If you want to contribute a test, then try to get a feel for the workspace style, pick a test that isn't done, and go for it. I haven't done any tests around transformers yet, so there is a lot still to do. And there may be reader/writer tests I haven't thought of. Preferably make a copy of the workspace, since we don't have proper revision control (yet). I'd like to keep it in 2016.1 or earlier for the moment, so no-one has to install a beta version.

I'm also open to any and all other ideas about how to go about this, and how to collaborate on a project like this. As far as I know, there's never been a crowdsourced FME project before!

My end goal is to get this online and hosted in FME Cloud, so we can make a proper web service out of it. My idea is that everyone who contributes would get recognition on the web page (and a custom KnowledgeCentre badge of course)!

So, let me know what you think - and if you want to contribute then please do so.

Mark

Fantastic. I like that. I wish I could be there to see the presentation.

 

@jdh This tool is a command line application and does not require a specific version control. It can be integrated in any source control tool accepting external compare program to compare two version of a file as we did with subversion.

 

Fantastic. I like that. I wish I could be there to see the presentation.

 

Ah, when I asked I was told it required svn.

 

 

Fantastic. I like that. I wish I could be there to see the presentation.

 

As the developer of the tool, I can assure you that SVN is not required. I'll make sure that the presenter is aware of this for the future.

 

 

The attachment seems to have gone missing. Do you have a copy of it still?

 

 

I hate that i dont get notifications when someone answers my posts (answerhub-issue) - just saw this. However I could successfully download the *.zip file .... is it ok now? And I cant seem to (a)mark2atsafe.

 

 

Fantastic. I like that. I wish I could be there to see the presentation.

 

Hi @KenAtSafe,

 

I'll try to see if I can find the presentation. It may be in french. No the code is not available but we are planning to open this as a web service for the beginning. I'll post the URL here as soon as it is done.

 

Fantastic. I like that. I wish I could be there to see the presentation.

 

Hi @KenAtSafe,

 

The presentation is strictly based on a demo, I'll try to make the web service live ASAP. In the mean time, if you want to have a little private demo or two workspaces compared, just email them to me.

 

Thanks @larry ! I'll be in touch

I just filed PR#76897 - a request for a Notification service that is a Repository Watcher. It would be like a Directory Watcher but would issue a notification if a workspace is published to a particular repository.

 

 

That way you could run a testing workspace in response - but you also get the ability to be notified in general when a workspace is published or updated.

 

 

Validating before publishing to Server - well that would need us to incorporate this whole idea into Desktop functionality (and I just don't see that happening - at least not as easily as a Repository Watcher might).

 

You can kind of do this with a directory watch to the repositories directory
Hi @KenAtSafe,

 

The presentation is strictly based on a demo, I'll try to make the web service live ASAP. In the mean time, if you want to have a little private demo or two workspaces compared, just email them to me.

 

Hi @larry,

We are currently looking into possible solutions for comparing workspace versions. Would you be available for a demonstration of this tool? Please let me know if you are happy to do this and I can reach out via email to pin down a date.

Hi @larry,

We are currently looking into possible solutions for comparing workspace versions. Would you be available for a demonstration of this tool? Please let me know if you are happy to do this and I can reach out via email to pin down a date.

Hello @hollyatsafe,

I'm no longer working with Consortech but you can contact them as they are a SAFE partner.

Ping me back if you can't reach them or if they do not get back to you.

Regards,

Larry

 

 

Hi @larry,

We are currently looking into possible solutions for comparing workspace versions. Would you be available for a demonstration of this tool? Please let me know if you are happy to do this and I can reach out via email to pin down a date.

Hello @hollyatsafe,

I'd gladly demo the Workspace Comparator for you.

Contact me by email to set a date.

C.

@mark2atsafe I left some issues on the Github page, but what is the preferred place for it? I am working on an updated version, with more checks and would like to get involveld.

@mark2atsafe I left some issues on the Github page, but what is the preferred place for it? I am working on an updated version, with more checks and would like to get involveld.

Yes, sorry about not responding before. It's just a busy time at the moment. I really would like to get this project back up and running. It's an important one to me. I think Github is the right place for it right now. Eventually it might live on the FME Hub (which is getting some great new updates for 2019 that will really help). I think for now if you can post changes back to Github that would be great. Once the FME 2019 release is done and I have all my training updates and blog posts complete, I'll take the time to dive back into this and see where we're at.

Thanks so much for getting involved. I do think this is a useful project and between us we'll get it fixed up and enhanced for sure.

Yes, sorry about not responding before. It's just a busy time at the moment. I really would like to get this project back up and running. It's an important one to me. I think Github is the right place for it right now. Eventually it might live on the FME Hub (which is getting some great new updates for 2019 that will really help). I think for now if you can post changes back to Github that would be great. Once the FME 2019 release is done and I have all my training updates and blog posts complete, I'll take the time to dive back into this and see where we're at.

Thanks so much for getting involved. I do think this is a useful project and between us we'll get it fixed up and enhanced for sure.

Thanks, I will do that soon.

Working on creating a formalized document with best practices. Stuck on Web/Database Connections and using them on server. I think ideally there would only be 1 connection per account that it's for (ie. 1 connection to 1 user on ArcGIS Online). BUT how do you manage access to them if that's the case? Or is it easier and better to just let anyone create their own connections and automatically get access since they own it? Is there any pros/cons to doing it the former vs latter?

Working on creating a formalized document with best practices. Stuck on Web/Database Connections and using them on server. I think ideally there would only be 1 connection per account that it's for (ie. 1 connection to 1 user on ArcGIS Online). BUT how do you manage access to them if that's the case? Or is it easier and better to just let anyone create their own connections and automatically get access since they own it? Is there any pros/cons to doing it the former vs latter?

Hi @runneals, In my experience, it's easier to let anyone create their own connections when working on FME Server. That ensures that only the account that created the connection can use or modify it by default. The owner of a connection does have the ability to share it with other users/roles as well, so if they wanted to allow others to have access to it, they can do so.

Having one connection per account could be helpful if you don't want your FME Server users to have their own credentials for the service (like AGOL) being used. For that, the owner of the connection can share it with a specific role to let any users part of that role have access to the connection. The drawback to this is that we currently don't have 'read only' access for connections, so sharing it would mean that any user could edit it.

Would be great if this was run right after you hit "Publish to FME Server" on the File menu of FME Workbench - so you could validate it BEFORE you publish it to an FME Server. Or more in generall - add an option to run a custom workspace before "Publish to FME Server" to validate something in that particular fmw workspace - for instance that all your database-connections are connected to staging/dev/prod and with the correct user / paths etc. When uploading to multiple FME Servers dev/stage/prod this can easily be forgotten!

You could probably set up an automation, and then send an email to the user that uploaded it.

Thanks to all you fine folks who provided comments here. I have put together a formal best practices guide.

Check out the updated version 1.1 (thanks to @mmccart for some additional feedback/ideas that I originally overlooked and merged with my original): https://github.com/runneals/GISProjects/tree/master/fme/FME%20Best%20Practices

Bumping this post up, because this can be useful to many also in 2019!

@sigtill @redgeographics and others - I'm just wondering about the best way to set this up on GitHub. If there are commit conflicts then it's very hard to assess what the differences are using GitHub itself. You'd need to open them in FME and do a visual inspection to see any differences (writing a DIFF tool for FME workspaces - that sounds like another challenge!)

 

 

Anyway, I think one way to help minimize that - and to make the project more manageable - is to split off each of the sections into exported custom transformers. It becomes a little harder to 'install' into FME, but I think the other advantages outweigh that.

 

 

So I'm going to do that and see if I can get it into GitHub sometime today.

 

@larry​ - Can you please share which tool you're describing here for FMW DIFF? Since this post is quite old, I'm not sure if there's a new tool available to diff purposes. I tried GIT but it's a great deal with FMW file format.

@sigtill @redgeographics and others - I'm just wondering about the best way to set this up on GitHub. If there are commit conflicts then it's very hard to assess what the differences are using GitHub itself. You'd need to open them in FME and do a visual inspection to see any differences (writing a DIFF tool for FME workspaces - that sounds like another challenge!)

 

 

Anyway, I think one way to help minimize that - and to make the project more manageable - is to split off each of the sections into exported custom transformers. It becomes a little harder to 'install' into FME, but I think the other advantages outweigh that.

 

 

So I'm going to do that and see if I can get it into GitHub sometime today.

 

Hello Ravin,

 

This is an old internal tool Consortech was developing while I was working there.

My best suggestion is for you to try to contact them and see if this is maintained and still available.

And please et me know if you can't have an answer, I can give you some hints if you want to develop your own tool.

 

Regards,

Larry

@sigtill @redgeographics and others - I'm just wondering about the best way to set this up on GitHub. If there are commit conflicts then it's very hard to assess what the differences are using GitHub itself. You'd need to open them in FME and do a visual inspection to see any differences (writing a DIFF tool for FME workspaces - that sounds like another challenge!)

 

 

Anyway, I think one way to help minimize that - and to make the project more manageable - is to split off each of the sections into exported custom transformers. It becomes a little harder to 'install' into FME, but I think the other advantages outweigh that.

 

 

So I'm going to do that and see if I can get it into GitHub sometime today.

 

Check out FME2022 (still in beta, but will be released shortly). It has a visual comparison tool that sounds like it would help.

@sigtill @redgeographics and others - I'm just wondering about the best way to set this up on GitHub. If there are commit conflicts then it's very hard to assess what the differences are using GitHub itself. You'd need to open them in FME and do a visual inspection to see any differences (writing a DIFF tool for FME workspaces - that sounds like another challenge!)

 

 

Anyway, I think one way to help minimize that - and to make the project more manageable - is to split off each of the sections into exported custom transformers. It becomes a little harder to 'install' into FME, but I think the other advantages outweigh that.

 

 

So I'm going to do that and see if I can get it into GitHub sometime today.

 

That would be great! Thanks. I'll try that out and post results.

@sigtill @redgeographics and others - I'm just wondering about the best way to set this up on GitHub. If there are commit conflicts then it's very hard to assess what the differences are using GitHub itself. You'd need to open them in FME and do a visual inspection to see any differences (writing a DIFF tool for FME workspaces - that sounds like another challenge!)

 

 

Anyway, I think one way to help minimize that - and to make the project more manageable - is to split off each of the sections into exported custom transformers. It becomes a little harder to 'install' into FME, but I think the other advantages outweigh that.

 

 

So I'm going to do that and see if I can get it into GitHub sometime today.

 

Thanks Larry

@sigtill @redgeographics and others - I'm just wondering about the best way to set this up on GitHub. If there are commit conflicts then it's very hard to assess what the differences are using GitHub itself. You'd need to open them in FME and do a visual inspection to see any differences (writing a DIFF tool for FME workspaces - that sounds like another challenge!)

 

 

Anyway, I think one way to help minimize that - and to make the project more manageable - is to split off each of the sections into exported custom transformers. It becomes a little harder to 'install' into FME, but I think the other advantages outweigh that.

 

 

So I'm going to do that and see if I can get it into GitHub sometime today.

 

I'm glad to hear that. Not using FME these days but I'll give it a try as soon as I get back to it!

Reply


Community Stats

30,176
Posts
114,538
Replies
37,835
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.