Question

Scripted parameters - using python script to extract values from JSON

  • 3 October 2013
  • 4 replies
  • 19 views

Badge
Hi all,

 

I'm trying to define some user parameters using scripted parameters.  I've written a python script that loads a JSON from a url into a dictionary then extracts the value for defined key. My Private Parameter is called classField.

 

 

It works fine when I run it from my Python IDE but not in workbench.  What do I need to add/remove to make it work in workbench?  I'm using FME 2012 and this is my code:

 

 

import simplejson

import urllib2

from pprint import pprint

import sys

import __main__

 

 

def URLjson2dictionary(url):

'''Reads json from url and loads it into a python dictionary'''

try:

# open the url (json string) and load into python dictionary

json_dict = simplejson.load(urllib2.urlopen(url))

except Exception as e:

print 'Could not load json from URL. Error: {0}'.format(e)

raise # reraises the exception

return json_dict

 

 

def dictValueFetcher(dict, key):

'''Returns the value for a specified key from the python dictionary'''

if key in dict:

return dict.get(key)

else:

print 'Key not in dictionary'

sys.exit()

 

 

if __name__ == '__main__':

 

# define input parameters

url = "http://www.ga.gov.au/gis/rest/services/topography/Australian_Topography/MapServer?f=pjson"

# load the json into a dictionary object

json_data = URLjson2dictionary(url)

# print dictionary derived from json in nice readable format

print 'json_data:'

pprint(json_data)

 

# Get the parameter value from the dictionary

FME_parameter_value = dictValueFetcher(json_data, "mapName")

print 'KeyFME_parameter_value: {0}'.format(FME_parameter_value)

 

This is the Log:

 

      Windows command-line to run this workspace:               fme.exe CIAP_SBAGDdataValidatorVerson3.fmw           --SourceDataset_GEODATABASE_SDE sde           --DestDataset_GEODATABASE_FILE C:\\CIAP\\FME_Server\\CIAPStagingDB.gdb           --DestDataset_JSON C:\\CIAP\\FME_Server\\DataLoadValidation_working\\InvalidGeomFeatures.json           Starting Translation ...   INCLUDE -- failed to evaluate Python script `def ParamFunc():   import simplejson   import urllib2   from pprint import pprint   import sys   import __main__         def URLjson2dictionary(url):       '''Reads json from url and loads it into a python dictionary'''       try:           # open the url (json string) and load into python dictionary           json_dict = simplejson.load(urllib2.urlopen(url))       except Exception as e:           print 'Could not load json from URL. Error: {0}'.format(e)           # send error message somewhere else   #        send_somewhere(traceback.format_exception(*sys.exc_info()))           raise # reraises the exception            return json_dict                         def dictValueFetcher(dict, key):       '''Returns the value for a specified key from the python dictionary'''       if key in dict:           return dict.get(key)       else:                 print 'Key not in dictionary'           sys.exit()         if __name__ == '__main__':          # define input parameters       url = "http://www.ga.gov.au/gis/rest/services/topography/Australian_Topography/MapServer?f=pjson"       # load the json into a dictionary object       json_data = URLjson2dictionary(url)       # print dictionary derived from json in nice readable format       print 'json_data:'       pprint(json_data)              # Get the parameter value from the dictionary           FME_parameter_value = dictValueFetcher(json_data, "mapName")       return print 'KeyFME_parameter_value: {0}'.format(FME_parameter_value) value = ParamFunc() macroName = 'classField' if value == None:   return { macroName : '' } else:   return { macroName : str(value) } ' Program Terminating   Translation FAILED.

 

 

If the scripting path is not the best method, any suggested workarounds would be greatly appreciated too.

 

 

Thanks

 

Rob

 


4 replies

Userlevel 2
Badge +17
Hi Rob,

 

 

FME Python interpreter is independent from any other Python interpreter in the same machine. I think you should install "simplejson" module into your FME Python environment.

 

See this article:

 

Install Python modules into FME

 

http://fmepedia.safe.com/articles/How_To/Install-Python-modules-into-FME

 

 

Takashi
Userlevel 2
Badge +17
In my personal opinion, defining the script in Startup Python Script seems to be more preferable, if possible. Because the script may raise an error but no messages will be logged when the error occurred in a scripted parameter.
Badge
Thanks for your replies Takashi.

 

After much experimentation I've come up with some code that actually works. I've kept it as a scripted parameter for now but will look into your startup script suggestion when I have time.

 

 

import json import urllib2 import pprint import sys   # Reads json from url and loads it into a python dictionary try:     # define input parameters     url = "http://rhe-ciap-dev01.dev.lan/explorer/resources/config/test.json"     # open the url (json string) and load into python dictionary     json_dict = json.load(urllib2.urlopen(url))     # print dictionary in a nice readable format     print "json_dict: "     pprint.pprint(json_dict)          except Exception as e:     print 'Could not load json from URL. Error: {0}'.format(e)     raise # reraises the exception                      # Returns the value for a specified key from the python dictionary key = "Classification_Field"   if key in json_dict:     parameterValue = json_dict.get(key)     print parameterValue     return parameterValue      else:           print 'Key not in dictionary'     sys.exit()

 

 

 

I'm still not sure why the original script didn't work but I've removed the defined functions to make it a bit simpler.  I didn't have to install any module into the FME python environment but I did swap the simplejson module for the json module (which is comes installed in the FME environment).  simplejson and json modules have pretty much the same functionality from what I've read.

 

 

Hope this helps anyone that comes across the same problem.

 

Rob
Userlevel 2
Badge +17
Glad to know you got a solution; and thank you for sharing your experiences.   I think your original script has at least two problems. 1. Importing "simplejson" module. 2. There is not "return" statement at the end, but there is a "print" statement instead.   I mentioned about 1st problem in the previous post, I'd like to describe about 2nd one here.  Note: The following  description is based on my trial-and-error, there are no documentations describing about this as far as I know. If I'm wrong, please correct!   Script defined for a Scripted Parameter seems to be interpreted as a function definition at run-time. For example, if you define a script like this for a Scripted Parameter named "PARAM": ----- value = 1 return value -----   FME replaces this with: ----- def ParamFunc():   value = 1   return value value = ParamFunc() macroName = 'PARAM' if value == None:   return { macroName : '' } else:   return { macroName : str(value) } -----   then, FME passes this script to Python interpreter to get a pair of parameter name and value. Therefore, the last line of the script should be a "return" statement.   Here, if the last line is not a "return" statement, FME seems to append "return" to head of the last line automatically.

 

The last line of your original script is: ----- print 'KeyFME_parameter_value: {0}'.format(FME_parameter_value) -----   but FME replaces it with this line, as you can see in the log you posted first. ----- return print 'KeyFME_parameter_value: {0}'.format(FME_parameter_value) -----   This will cause a syntax error. I think this is one of reasons that your original script didn't work.

 

Takashi

Reply