Scraping dynamic web page data

Thanks a lot @revesz. My Python skills are...uhm..copy & paste mostly :) So I would definitely appreciate you sharing any of your content and perhaps use it to learn more. My goal for this project is to scrape the text to know when a new version of the software is available. I have done this on other sites but they either have some table exposed and I can get that directly in FME, or there is an API available which I was able to figure out from the page source itself.

Ok so I was able to get to the data without additional tools. Lucky? Maybe :) Took some sleuthing, but here's how I got to what I needed for the example above:

1. I saved the web page but selected the option that I just noticed in Chrome "Webpage, Complete"
2. I opened this in Notepad++ and tried reading through to see if I could find anything helpful. I noticed that the tag I wanted was "h2", but most importantly, I noticed this around the middle of the saved page:
3. I tested by reconstructing the URL as "https://help.autodesk.com/cloudhelp/ENU/CONNECT/files/GUID-03D59AAD-65B0-45E3-84F2-A12AAA5BB267.htm" and the page loaded (URL re-directed to the original one)
4. In FME I used an HTMLExtractor on this new URL:
5. And BAAM! As happy as can be :)

   

I guess when there's a will, there's a way!

I have ran into the same problem when attempting to scrape some data from a site that uses JavaScript to load the data into the page.  When the HTTPCaller transformer completes the page is not fully loaded because the JavaScript function associated with the page onLoad event has not been fired.

The only way that I could find to address this was to use the PythonCaller, together with the Selenium browser automation module (https://www.selenium.dev/documentation/en/).  For a useful guide on using Selenium with Python to scrape data from web pages, see https://www.scrapingbee.com/blog/selenium-python/

The Python Selenium module needs to be installed on your machine - see https://docs.safe.com/fme/html/FME_Desktop_Documentation/FME_Workbench/Workbench/Installing-Python-Packages.htm, and https://pypi.org/project/selenium/

You will also need to decide which web browser you wish to use and what version of the browser you have, then install the matching webdriver application.  For example, for the Chrome web browser, refer to this site: https://chromedriver.chromium.org/getting-started

I set my PythonCaller up as follows:

import fme
import fmeobjects
from seleniumwire import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.expected_conditions import presence_of_element_located
from selenium.webdriver.support.select import Select
from selenium.common.exceptions import TimeoutException
import time
 
 
class FeatureProcessor(object):
    def __init__(self):
        
        print("PythonCaller: initialising web browser")
        
        access_token =  fme.macroValues['OAuth2_token']
        
        chrome_options = Options()
        chrome_options.add_argument('--headless')
        chrome_options.page_load_strategy = 'normal'
 
        webdriver.header_overrides = {'Authorization': 'Bearer ' + access_token}
 
        # Luanch headless web browser, need to to specify webdriver executable disk location
        self.web_browser = webdriver.Chrome(options=chrome_options, executable_path=r"C:\Apps\chromedriver_win32\chromedriver.exe")  
 
        
    def input(self,feature):
 
        calendar = feature.getAttribute("userPrincipalName")
        booking_url = "https://outlook.office365.com/owa/calendar/"+calendar+"/bookings/"
        id = feature.getAttribute("id")
        out_html = "C:/Temp/"+id+".html"
        
        print("Getting data for: " + calendar)
        
        self.web_browser.get(booking_url)
        delay = 5
        try:
            myElem = WebDriverWait(self.web_browser, delay).until(EC.presence_of_element_located((By.CSS_SELECTOR, "div.focusable.dates")))
            print('Page is fully loaded')
            
            html = self.web_browser.page_source
 
            with open(out_html, "w+", encoding="utf-8") as f:
                f.write(html)
                
            feature.setAttribute("html_file",out_html)
            print('html file is ready')
                        
        except TimeoutException:
            print('page ' + calendar + ' is taking too long')
            feature.setAttribute("html_file","Error")   
 
        self.pyoutput(feature)
 
    def close(self):
        print("PythonCaller - Closing")
        self.web_browser.quit()
    

The PythonCaller init() routine setups the headless web browser.

The input() method processes every input feature - it calls the corresponding web page and then waits for the onload JavaScript to complete (waits for a specific div to load) and then saves the resultant html to a temporary file - which can be accessed and processed by the FME HTMLExtractor transformer.

The page I was calling also requires token authentication so I obtain the token as a Python Scripted parameter at startup and then read it into the init() method.  I also needed to include the seleniumwire module to include the token in the call to the web server.

This is working fine on my desktop but unfortunately I cannot publish this workbench to FME Cloud as there is no way to install the webdriver application onto FME Cloud.  One step at a time...

import fme
import fmeobjects
from seleniumwire import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.expected_conditions import presence_of_element_located
from selenium.webdriver.support.select import Select
from selenium.common.exceptions import TimeoutException
import time
 
 
class FeatureProcessor(object):
    def __init__(self):
        
        print("PythonCaller: initialising web browser")
        
        access_token =  fme.macroValues['OAuth2_token']
        
        chrome_options = Options()
        chrome_options.add_argument('--headless')
        chrome_options.page_load_strategy = 'normal'
 
        webdriver.header_overrides = {'Authorization': 'Bearer ' + access_token}
 
        # Luanch headless web browser, need to to specify webdriver executable disk location
        self.web_browser = webdriver.Chrome(options=chrome_options, executable_path=r"C:\Apps\chromedriver_win32\chromedriver.exe")  
 
        
    def input(self,feature):
 
        calendar = feature.getAttribute("userPrincipalName")
        booking_url = "https://outlook.office365.com/owa/calendar/"+calendar+"/bookings/"
        id = feature.getAttribute("id")
        out_html = "C:/Temp/"+id+".html"
        
        print("Getting data for: " + calendar)
        
        self.web_browser.get(booking_url)
        delay = 5
        try:
            myElem = WebDriverWait(self.web_browser, delay).until(EC.presence_of_element_located((By.CSS_SELECTOR, "div.focusable.dates")))
            print('Page is fully loaded')
            
            html = self.web_browser.page_source
 
            with open(out_html, "w+", encoding="utf-8") as f:
                f.write(html)
                
            feature.setAttribute("html_file",out_html)
            print('html file is ready')
                        
        except TimeoutException:
            print('page ' + calendar + ' is taking too long')
            feature.setAttribute("html_file","Error")   
 
        self.pyoutput(feature)
 
    def close(self):
        print("PythonCaller - Closing")
        self.web_browser.quit()
    

Thanks for the detailed reply @spatialcase​