Hello FME'ers,

I have a question-of-the-week today that's really more philosophical than technical; I'll lay out my opinion on the matter and bring in other thoughts from the answers to the question.

Question of the Week

Q) Can a person who has never worked on FME, be tasked with migrating data from GE SmallWorld to PostGIS? Should we engage FME Certified Partners or can we learn this along the way?

Asked here by @navraj28

 

A) Well, that's a fairly loaded question to answer when my role includes providing materials to bring new users up to speed. Plus the question was short on details making the quick answer... it depends!

Let's look at some of the factors I'd use to make a decision...

Resources

Personally I do think it's possible to dive right into a project when you first start FME. We have a lot of resources in the form of documentation, tutorials, knowledge-base articles, training, webinars, technical support, blogs, events, and conferences (did I miss anything?!)

Safe will help; the community will help. It can be done. The tools are there for you.

However, there's going to be a learning curve. The quicker the project needs to be completed, the steeper that curve is going to be. So time is a definite factor.

And there are other factors that affect how much and how fast you need to learn...

Formats

If the task were converting CSV files to Excel, I think we'd all agree that it's possible for a new user to learn that fairly quickly. But more complex formats require extra effort. Here the two formats are GE Smallworld and PostGIS. Let's just say that I wouldn't consider these two formats simple ones.

Plus the fact that the data comes from GE Smallworld suggests another factor...

Data Model

Smallworld is a product for utility companies to use; for example gas, electricity, and telecommunications. In my experience that sort of data tends to be more complex. There is usually lots of topology and interconnectivity included in the data, plus a lot more attributes to handle.

Preserving all of that information could be fairly involved, and that involves transformation...

Data Transformation

When you migrate data between two systems, how much of the source schema do you want to replicate?

If you're just trying to get one piece of data (say you want to extract all lighting columns to show on a web map) then that's simple. If you're trying to replicate the entire data model - all the tables, all the attributes, all the topology - then it's obviously harder for a new user to set up.

Another question might be, how well does the user know the two systems?

User Experience

The best foundation assumes your user has a lot of experience using Smallworld and PostGIS. Then they'll know how the two systems work and just have to learn how to connect them the right way in FME. But if the user has to learn one or both of those systems at the same time, it obviously makes for a steeper learning curve.

A final factor: what is the overall migration plan?

Migration Plans

The term "migration" implies you are moving all business operations from one system to another. Timing then becomes a factor. Will you turn off one system and go straight to the other? Or will you run both of them simultaneously until you are confident in the new setup? During the changeover period, what happens to data updated on the live system? Will you update it manually on both, or would you like FME to move the data, perhaps using Change Detection to find what is different?

There are a lot of questions there and some scenarios will need more FME involvement than others.

Summary

Sorry for the wall of text, but there aren't many images I can think to add to break it up!

To summarize, we know there's a learning curve to FME, but that's why we provide so many learning resources and opportunities. I encourage you to use them because you will learn a lot and make your FME experience better and easier.

However, if you look at the above factors and feel that they all go against you, in that scenario I think it would be wise to get a certified FME expert involved. You can find a list of official partners on our web site and can even filter that list by location, industry, and areas of expertise.

I'll finish by quoting a couple of answers from the actual question:

"The FME documentation is pretty thorough and easy-to-read. You can find plenty of examples and the community is very newbie-friendly. So I think the process could totally be done by a person with null experience in FME if they are given enough time.""While it is certainly possible for somebody without any FME experience to do this and learn as they go... working with an experienced partner can cut the total time down by quite a lot."";An] FME Partner should be able to bring their experience to not only help with the technical aspects of the migration, but also to explain to your internal resources how and why things are done in a particular way... This is a great way to train your internal users in FME and to let them take ownership of the finished result."Other Notable Questions

Let me just highlight a few other interesting questions from this week:

• Extracting a Dynamic HTML Table
  • @kasparlov asked the question and @takashi and @bruceharold helped with answers. The issue is that many websites don't serve up data as static HTML, but generate it dynamically with Javascript. When you scrape the page content with the HTML reader you get the Javascript, not the table it is generating. I haven't yet seen a good way of getting the actual data automatically.
  • So I tried the WebPageOpener transformer to open the page from within FME. File > Save As > HTML Complete gives me what I want, so I looked for an extension to save the page automatically. I found one for Firefox but it saves the page as "HTML Only". The Mozilla API doc tells me that there is no option to choose between "HTML Complete" and "HTML Only" so I can't edit the script. And there I am stuck. I still figure there must be an automatic method though.
• How to Use the StringSearcher Function in the AttributeManagers
  • @edhere asks the question and @markatsafe provides the definitive answer, which is that you can search for and extract a string in the AttributeManager Text Editor dialog (use the FindRegEx and Substring functions) but it's just as easy to use StringSearcher transformers. Of course, general opinion is not to parse HTML with regular expressions, but realistically, we all do it!
• Can you choose to write all your features based on testing just one of them?
  • i.e. If one feature passes a test, they all do. If that feature fails the test, they all do. @billybob asks the question and gets a number of responses. Overall the solution seems to be to aggregate the features so that one test handles them all. The question is how do you get your original features back if they do pass? A ListExploder might help. My usual standby is a combo of the NoFeaturesTester and a FeatureMerger.
• Any Geospatial Podcasts you can recommend?
  • @sigtill wants to know! If you know of a good podcast, feel free to list one. During our webinar this week someone suggested we start an FME one. I have to say it's appealing!