Scripted parameter approach to defining the output raster cell size using existing raster?

Hi again,

 

 

In order to extract raster properties (x, y spacing), you will have to read the raster with Python script. And a scripted parameter returns only one value, so you will have to define additional parameters to return x and y separately.

 

For example:

 

-----

 

# Scripted Python Parameter: SPACING_XY

 

# Extracts spacing x and y from an ECW raster data

 

# and returns them as a string value with CSV format.

 

import fmeobjects

 

ecw_path = 'C:\\\\tmp\\\\data.ecw' # your source ECW file path

 

reader = fmeobjects.FMEUniversalReader('ECW', False)

 

reader.open(ecw_path)

 

raster = reader.read()

 

raster.performFunction('@RasterProperties(RASTER)')

 

spacing_x = raster.getAttribute('_spacing_x')

 

spacing_y = raster.getAttribute('_spacing_y')

 

return '%f,%f' % (spacing_x, spacing_y)

 

-----

 

# Scripted Python Parameter: SPACING_X

 

# Returns x spacing value

 

return FME_MacroValues['SPACING_XY'].split(',')[0]

 

-----

 

# Scripted Python Parameter: SPACING_Y

 

# Returns y spacing value

 

return FME_MacroValues['SPACING_XY'].split(',')[1]

 

-----

 

 

It's possible and interesting, but is troublesome a little. I would recommend you to adopt the WorkspaceRunner approach.

 

 

Takashi

I find it an elegant solution,  particularily negating the need for workspacerunning. (wich i consider a "last ditch" solution..

 

 

Also i think if u use gdal, u might extract rasterproperties without actually reading the entire  raster.

 

 

 

:)

Takashi,

 

 

there are Python bindings for GDAL. I haven't had the time/occasion to try them out, but I think it's possible to install them and set FME to use that particular Python interpreter to enable GDAL functionality from within Python scripts in FME.

 

 

That being said, I think I prefer your solution where you use the existing raster support in FME, it's probably way easier than messing about with GDAL and a custom Python interpreter.

 

 

David

Takashi - thanks for the Python example. The 3 scripts themselves mostly make sense, as I have worked with Python before, but never in FME - so I'm not sure where to put them in a workspace. The part of the script that I don't think I fully understand is when the two small scripts are refering to "FME_MacroValues". Is this a different name for public parameters?

 

 

Based on the readings I've done, it seems that I would need to implement the first (long) script using a "startup Python Script" in the Navigator window since I want the values it returns before running the workspace, and then I would need to call the two other scripts (also as "startup scripts"?) and have them somehow have them return the value to a public parameter - unless that is what "FME_MarcroValues" does?

 

 

For now, I just tried a blank workspace using a "startup Python Script" - I copied into it the 1st (long script), changed the reference to the ecw file, and ran the model. This threw an error:

 

 

"Python Exception <SyntaxError>: 'return' outside function (<string>, line 12)"

 

 

not really sure what this means.

 

 

On a broader note,  Gio, I tend to agree with you that having one workspace seems like a more elegant solution (but maybe I'm speaking too soon, as I obviously don't know all the tricks that go into using the python method!) 

General sequence for running a workspace is:

 

1. Configure a mapping file based on fmw file and parameter settings.

 

2. Open a log file.

 

3. Execute startup script if it was defined.

 

4. Perform the translation (Readers, Transformers, Writers)

 

5. Close the log file.

 

6. Execute shutdown script if it was defined.

 

All the user parameters including scripted ones will be configured in the first step. So you cannot define or modify any parameter afterward.

 

Parameters are defined as macros in the mapping file, and FME_MacroValues is a dictionary which is a map of macro (parameter) name to value.

 

Therefore, in this case, SPACING_XY should be defined before SPACING_X and SPACING_Y.

 

 

Regarding the error, since startup script is not a function, it cannot return any value. A script defined as a scripted parameter will be interpreted as a function by Workbench at run-time.

 

When you test the script in Startup, comment out the return statement and log the result instead. For example:

 

-----

 

# return '%f,%f' % (spacing_x, spacing_y)

 

logger = fmeobjects.FMELogFile()

 

logger.logMessageString('%f,%f' % (spacing_x, spacing_y))

 

-----

 

 

@David, thanks for the information. I found Python bindings for GDAL and was able to install in my environment. As Gio mentioned, indeed it's possible to extract pixel sizes without reading whole data, but I failed to open an ECW file with the gdal module. Some other formats e.g. geotiff can be opened.

 

 

I like scripting. But in creating a practical workspace, my general priority is:

 

1. FME Transformers.

 

2. Scripting with libraries contained in FME standard installation.

 

3. Scripting with extension libraries.

 

The WorkspaceRunner is not "last ditch" solution in my thought ;)

Takashi,

 

 

for what it's worth, I completely agree with your priorities and your opinion about the WorkspaceRunner :-)

 

 

David

Maybe so.

 

 

But workspacerunners are almost alwyas used when single workbench solution seems not possible.

 

One of the big ones is dynamic schema's, or dynamic attribute creation. (fme should do something about that)

 

 

One very usefulll use for a workspacerunner we found to be on a pdf workbench wich outputs large amounts of pdf's.

 

Normaly the system would collect all data for al pdf's generated. When finished collecting data it then would serialy output (write) the pdf,s.

 

This can generate high memory usage. Also you cannot see, for instance the 1st pdf, because you have to wait for the last. When a workspace runner is set to call the writer on the other hand, you can force the system to write out a pdf as soon as it has collected the data needed for 1.

 

That way it will collect data and write the pdf's one by one. Now you can view the first as soon as it's ready and the script continue's building the rest.

 

This was imo the only one use of the ws that was not due to fails. (of course i have not seen all creativity possible with the transformer collection).

 

 

But yeah, i consider it a last ditch resort... ;)

 

 

...well it's pretty usefull for staging of course..now u mention it. :)

 

 

Scripting might get complicated very fast, but it's usually very elegant.

 

Compressing a 100 transformer workbench to like less then 10. Though one can argue why use fme at all then?! ehehehe

 

 

alas i can't play with gdal at work, hard to convince itc of its usage. Though we have a single user, "freezone" server available. It's just no match for my workhorse power pc.

 

 

anyway, i was basically more intereseted in extracting meta-data without reading the entire raster, is why i referred to gdal. ECW suppoprt is not standard in gdal:

 

 

http://wiki.openstreetmap.org/wiki/ECW

 

 

 

http://trac.osgeo.org/gdal/wiki/ECW

 

Takashi - thanks for your patience in explaining. I have managed to get an example working now all with scripted python parameters!

After going through the excercise, I am going to stick with Gio's option - using a python script rather than two workbenches seems more elegant (and more intuitive) to me.

Gio, I can partially agree with you.

 

My priority is a general one, is not an absolute rule. There are exceptions depending on actual conditions. For example:

 

If 100 transformers can be replaced with a 10 lines script, I would adopt the script. If 2 minutes processing with transformers can be performed in 0.5 seconds with a script, I would use the script (2 minutes to 0.5 seconds is an actual case which I came across a few months ago).

 

and so on.

 

However, I would avoid 10000 lines script for 100 transformers, and 1.9 minutes script for 2 minutes transformers.

 

I think all should be decided based on the balance of several viewpoints of understandability, maintainability, portability, efficiency etc.. In my experiences, using FME transformers could have better balance in many cases, so my general priority has become like that.

 

 

In this case, I originally had thought of a solution with another published parameter which indicates the source ecw file path. Like this.

 

-----

 

# Python

 

reader.open(FME_MacroValues['ECW_SOURCE_PATH'])

 

-----

 

# Tcl

 

reader open $FME_MacroValues(ECW_SOURCE_PATH)

 

-----

 

It looks better especially from viewpoint of portability, but there was a "trap".

 

It works fine if it was written in Startup script. However, in a script for Scripted Parameter, "FME_MacroValues['ECW_SOURCE_PATH']" would be interpreted like "$(FME_MF_DIR)data.ecw" at run-time if specified ecw file was saved in the FME_MF_DIR. Although there is a workaround with adding several statements to the script, it would be no longer elegant.

 

There always could be an unknown trap when scripting, and you would have to consume long time if you came across it. In fact, I had consumed about 30 minutes for recognizing the trap.

 

It is also one of reasons for having recommended you to adopt the WorkspaceRunner approach in this case.

 

 

Good luck!

Aah Takashi, but i admire youre tenacity...:)

 

 

 

I might collect youre musings and make a little book.  :)

 

 

And i as well agree on various parts with you.