Skip to main content
Question

FME Challenge: Choosing the Right Transformer (Webinar Tie-In)

  • May 24, 2018
  • 3 replies
  • 23 views

mark2atsafe
Safer
Forum|alt.badge.img+58

Hi FME'ers,

In an hour or so we're starting the webinar on Choosing the Right Transformer (24-May, 10am PDT). Several users already submitted scenarios where they are facing a challenge finding the right transformer, and Dale, Iris, and I will try and live-build a workspace to solve their issue.

At the same time, we had a few scenarios that we won't have time to cover in full, and wanted to throw them out for your suggestions. So, if you think you know what transformers to use in the following scenarios, or if you have a brief scenario that we can try and demonstrate online, please let us know in the comments below.

We'll do our best to work this content into the webinar to make it really live (and quite exciting for us presenters!) - and if you have any comments on the webinar or suggestions for the challenges we are showing, do also let us know.

Cheers

Mark

Scenario 1: Given this feed of live cycle data from TFL (Transport for London), which transformers would you suggest to read the data, to create point features, to remove extra attributes, and to add/update XML metadata using a specific schema? Is there a specific technique that you can recommend?

Scenario 2: Given a webpage (no demo available unfortunately) with a list of video files, which transformers would you suggest to download each file, compare it to a list of existing files, and (if new) publish it to a YouTube channel? What other files could be published using a Google API?

This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.

3 replies

  • May 24, 2018

I submitted #2 so I thought I add my solution here for reference.

First off I build a web scraper using http caller and html extractor.

I ended up not using the this method and switching to an API to get this data.

I can post more of this solution if anyone wants but I'm still finishing off the the youtube upload due to an issue outside of FME.


Forum|alt.badge.img
  • May 24, 2018

Scenario 1: (I know its a bit late)

To read the data, to create point features and choose attributes I would use a Python Creator with following code:

import fmeobjects
import requests
from xml.etree import ElementTree as ET

# Template Class
class FeatureCreator(object):
    def __init__(self):
        pass
    def input(self,feature):
        Path = r"https://tfl.gov.uk/tfl/syndication/feeds/cycle-hire/livecyclehireupdates.xml"
colList = ["name", "terminalName","lat","long","nbBikes","nbEmptyDocks","nbDocks"]
        response = requests.get(Path)
        root = ET.fromstring(response.content)

        for child in root:
            newFeature = fmeobjects.FMEFeature()
            for grandchild in child:
if grandchild.tag in colList: 
                 newFeature.setAttribute(grandchild.tag, grandchild.text)
            self.pyoutput(newFeature)
    def close(self):
        pass 

mark2atsafe
Safer
Forum|alt.badge.img+58
  • Author
  • Safer
  • May 24, 2018

I thought I'd just upload the first bit of XML reading that I created during the webinar. It doesn't address the metadata, but it reads the feed and splits the data into attributes.

I used a Textfile reader with the parameter set to read the entire feed into a single attribute, and then fragmented the data with an XMLFragmenter transformer.

The XMLFragmenter will read directly from the XML feed, but I used a text reader because then I could do a partial run on the XMLFragmenter without having to reread the feed.

Workspace - uses the latest 2018.0 version (but should be OK with any 2018.0)