Skip to main content
  • EN
Question

FME can't read SpeedZones shapefile

  • 2 September 2024
  • 7 replies
  • 58 views

I have a shapefile that can’t be read by FME Workbench 2024.1.1 or 2022.2.3.  The copied log is attached.

The “Speed Zones” shapefile comes from https://opendata.transport.nsw.gov.au/dataset/speed-zones where you have to login and download the "Speed Zones data" as a ZIP file.  Once unzipped you get the standard 4 files.

When I load the shapefile into QGIS, it shows the following properties:

    434MB
    MultiLineString
    442,943 features
    EPSG:3308 - GDA94 / NSW Lambert
    Fields:
        Type    Text (string) 200 length
        Status    Text (string) 200 length
        Direction    Text (string) 200 length
        Speed    Text (string) 200 length

The only issue I can see is that there is no unique identifier field but I didn’t think that was an issue for FME.  Any suggestions on how to read this file into FME?

Thanks

Doug

7 replies

bwn
Rank
Userlevel 4
Badge +12

Works OK for me Doug,

I suspect it could be the path used in the Reader that is the problem.   It appears this is using a File Web API where these APIs struggle with multi-file type formats like SHP, as it won’t automatically GET the other needed files like the DBF
 


With a lot of FME Readers, you don’t have to actually Unzip the files and can just point it to the ZIP container and it will assume that the contents to be read are in the ZIP container.

So, I would just leave it in its original ZIP packaging and Read that.  This is what will work with File Access Web API type interfaces as well.

 



Note, you can also read it directly from the from the Transport NSW website without having to manually download it.

In the reader, use the ZIP file URL for the SHP file



Go into the Reader Parameters and set Network Authentication

 

 

...and set your user name and password to use for the Web API

 

 

 

When Workspace executes, this automatically downloads the ZIP for you to a temporary store, unzips it, and reads the Features in

 

 

 

In fact, any Web based API that supports a file get operation can be used this way to get the binary file requested, and I use the same approach for any REST type interface or even our own internal EDRMS that has its own file accessor web commands.

doug_herrick
Rank
Badge +1

Thanks bwn,

I had been using a HTTP Caller to download the ZIP file from the Transport for NSW website but it is great to know that you can do the same thing with a shapefile reader.  So I’m not having any trouble downloading the ZIP file but I can’t open the shapefile within it, which is why I have manually unzipped it.

The fact that you can open the shapefile suggests that it is either a permissions issue, an FME setting or my FME installation.

It doesn’t appear to be a permissions issue as I can open other shapefiles from the same location.  I even moved the shapefile onto the server where we have FME installed and tried to get a shapefile reader to open it from there but I still got the following message:

 

 

2024-09-03 08:58:39|   0.1|  0.0|INFORM|SHAPEFILE reader: Opening dataset 'D:\SpeedZones\SpeedZones.shp'

2024-09-03 08:58:43|      |     |ERROR |Error running translation.

2024-09-03 08:58:43|      |     |ERROR |Workspace generation failed.  Parameter file was C:\Users\dherrick\AppData\Local\Temp\33\FME_1725317916559_7504.

I’ve attached the “Parameter file” but can’t see anything obviously wrong with it.

Bye

Doug

1 Attachment
bwn
Rank
Userlevel 4
Badge +12

Hmmm, I guess start with some basics to rule some things out.

What happens if manually download the ZIP file through the web browser (rather than HTTPCaller),  and then what happens if point the SHP Reader to just read the manually downloaded ZIP?

Also, leave the Coordinate System to Null/Blank for the moment on the Reader.   Initial test is can just use plain, manually downloaded ZIP file and read it in.

doug_herrick
Rank
Badge +1

Thanks for your help bmw.  I manually downloaded the speedzones.zip from the website and then in a new FME workspace, tried to create a shapefile reader that pointed to the zip file.  I immediately got the following errors.

2024-09-04 13:38:29|   0.1|  0.1|INFORM|Creating reader for format: Esri Shapefile
2024-09-04 13:38:29|   0.2|  0.1|INFORM|Trying to find a DYNAMIC plugin for reader named `SHAPEFILE'
2024-09-04 13:38:29|   0.2|  0.0|INFORM|Loaded module 'SHAPEFILE' from file 'C:\Program Files\FME2024_0_1\plugins/shapefile_dbase.dll'
2024-09-04 13:38:29|   0.2|  0.0|INFORM|FME API version of module 'shapefile_dbase' matches current internal version (4.0 20240212)
2024-09-04 13:38:29|   0.2|  0.0|INFORM|SHAPEFILE reader: Opening dataset 'C:\Users\dherrick\AppData\Local\Temp\37\wbrun_1725412642298_604\fmetmp_14\TempFS_1725421106690_7340\ds_0\speedzones.zip_0\SpeedZones.shp'
2024-09-04 13:38:32|      |     |ERROR |Error running translation.
2024-09-04 13:38:32|      |     |ERROR |Workspace generation failed.  Parameter file was C:\Users\dherrick\AppData\Local\Temp\37\FME_1725421105579_604.
 

Bye

Doug

bwn
Rank
Userlevel 4
Badge +12

Rules many things out Doug, but still leaves it in a very odd situation!  Much as I don’t like SHP for a variety of reasons, never had too many issues reading them.

The Translation Log is saying all the right things, can see that it temporarily caches the ZIP fill into your User %Temp% Folder, and then it goes to extract the files from the ZIP and read them.   Further up, FME confirms it got an appropriate SHP API from its plugins folder.  All of this is normal...but yet the translation fails.


Just some real guessing games here, but this could fail if there wasn’t enough space of C:\ to unzip\cache the data.

There could also be some bizarre anti-virus triggering a read access block.

Anyway, to keep checking the basics though

  1. Did the Reader when configured manage to read the Schema?

    ie.  When expand the Reader Attributes, are these visible on the Workspace like this?  If so then confirms the Reader CAN actually recognise it as a SHP and do and intial scan of the schema
     

     

  1.  Check the Reader Parameters match the “default” setings

Here are what “should” show as your Reader Parameters.  Do these match your Reader parameter values?

 

|

 

doug_herrick
Rank
Badge +1

I've checked C:\ drive on the server where FME is installed and there is plenty of space.  I was able to create a shapefile reader for a much larger ZIP file (164MB) and it read the dozen shapefiles inside it ok, though it took nearly a minute to do so.

For SpeedZones (57MB), the shapefile reader does not get created so there is nothing about it in the Navigator panel.  However here are the default parameters that I use:

When I click OK in the "Add Reader" dialog box, entries appear in the Translation Log and then I get the error message shown below.

 

bwn
Rank
Userlevel 4
Badge +12

Hi Doug,

I’m afraid I can’t think of much else to test.  Everything you’ve shown looks fine on setup.

I only have some lingering doubt about perhaps the server machine is configured, but if can create other SHP Readers on the same machine for SHP files in the same folder this tends to rule it out.

At this stage would be thinking about a separate test machine install to check if it is Windows Server machine specific, and/or raising a ticket with Safe.  The SHP file itself is valid….it reads fine in FME 2021.

I assume that the temp folder shown in the Translation log it is C:\ on the Windows Server machine and is your own account remotely logging in?  Can’t see any reason it isn’t, again random thought is that for whatever reason FME is caching/unzipping the SHP file on initial load from the source into a Temp folder different from what the Reader when it is loaded to what FME tries to point it to and throws an error message that the SHP file “isn’t there” when it tries to read from a different location.  I have run into similar problems if any of these User Folders (more typically Documents rather than AppData however) have been configured on the machine as a redirected/symbolic link type folder where the Windows file copy/paste command into the User caches gets intercepted and redirected to another Eg. Network Folder but something like FME then tries to find those files on the local C:\ and they aren’t there.   Again though, if not the same problem with other SHP for the same config…..that tends to rule it out 😕

Reply


Community Stats

29,620
Posts
112,572
Replies
37,269
Members
Terms & ConditionsCookie settings