Question

Data upload via REST makes the dbf of my shapefile corrupt

jeroen
Rank
Badge +7

Using FME Server build 18305

I am testing the Upload multiple files in the FME Server REST API V3 on my server.

The goal is to upload a shape file using the POST /resources/connections/< resource >/filesys/< path > (Upload multiple files) method but getting stuck on using the file due to corruption.

These are the settings:

0684Q00000ArFPiQAN.png

The return looks like this:

Response Code:

200

Response Body

[
  {
    "date": "2018-10-22T11:09:43Z",
    "path": "/Testwerk/",
    "size": 8360,
    "name": "waterschapsgrenzen.dbf",
    "type": "FILE"
  },
  {
    "date": "2018-10-22T11:09:43Z",
    "path": "/Testwerk/",
    "size": 428,
    "name": "waterschapsgrenzen.prj",
    "type": "FILE"
  },
  {
    "date": "2018-10-22T11:09:43Z",
    "path": "/Testwerk/",
    "size": 170,
    "name": "waterschapsgrenzen.shx",
    "type": "FILE"
  },
  {
    "date": "2018-10-22T11:09:43Z",
    "path": "/Testwerk/",
    "size": 960243,
    "name": "waterschapsgrenzen.shp",
    "type": "FILE"
  }
]

Everything looks OK till now, but when downloading and inspecting the shapes with the Data Inspector or useing it in flow the following error with the dbf pops up:

Log File: C:\Users\jeroen\AppData\Local\Temp\FMEDataInspector\logs\fmedatainspector_2018-10-22T13-21-08.log

Loaded module 'fmeprompter' from file 'fmeprompter.dll'

'ESRISHAPE' reader was created successfully

Creating reader for format: Esri Shapefile

The dataset 'C:\Users\jeroen\Downloads\waterschapsgrenzen.shp' was opened successfully

The feature cache has not been loaded while a new setConstraints was performed. Now loading...

Using Shape Reader to read shape files from folder `C:\Users\jeroen\Downloads/'

R_1 Reader: Using enhanced geometry

Shape Reader: Using simple donut geometry detection

Shape Reader: Handling numeric types with standard integer and floating point types

Shape Reader: Allowing attribute names with mixed case

Opened Shape File 'C:\Users\jeroen\Downloads\waterschapsgrenzen.shp' for input

Opened DBF File 'C:\Users\jeroen\Downloads\waterschapsgrenzen.dbf' for input

DBF Reader: Could not detect encoding. Using default system encoding

DBF File 'C:\Users\jeroen\Downloads\waterschapsgrenzen.dbf' has fields: A DBF Internal Type Designation '

The OGC definition of the FME coordinate system '_Netherlands-RDNew-2008_0' is 'PROJCS["RD_New",GEOGCS["GCS_Amersfoort",DATUM["D_Amersfoort",SPHEROID["Bessel_1841",6377397.155,299.1528128]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Double_Stereographic"],PARAMETER["False_Easting",155000.0],PARAMETER["False_Northing",463000.0],PARAMETER["Central_Meridian",5.38763888888889],PARAMETER["Scale_Factor",0.9999079],PARAMETER["Latitude_Of_Origin",52.15616055555555],UNIT["Meter",1.0]]'

FME Configuration: Source coordinate system for reader R_1[ESRISHAPE] set to `_Netherlands-RDNew-2008_0' as read from input data

Coordinate System `_Netherlands-RDNew-2008_0' parameters: CS_NAME=`_Netherlands-RDNew-2008_0' DESC_NM=`Netherlands, Amersfoort RD 2008 datum, New System' DT_NAME=`Amersfoort-RD-2008' ESRI_WKT=`PROJCS["RD_New",GEOGCS["GCS_Amersfoort",DATUM["D_Amersfoort",SPHEROID["Bessel_1841",6377397.155,299.1528128]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]],PROJECTION["Double_Stereographic"],PARAMETER["False_Easting",155000.0],PARAMETER["False_Northing",463000.0],PARAMETER["Central_Meridian",5.38763888888889],PARAMETER["Scale_Factor",0.9999079],PARAMETER["Latitude_Of_Origin",52.15616055555555],UNIT["Meter",1.0]]' GROUP=`EUROPE' MAP_SCL=`1' ORG_LAT=`52.15616055555556' ORG_LNG=`5.387638888888889' PROJ=`OSTERO' QUAD=`1' SCL_RED=`0.9999079' SOURCE=`Kadaster' UNIT=`METER' X_OFF=`155000' Y_OFF=`463000' 

No shape record found when reading record number '1'.  Shape file is either corrupt or end of file has reached with no meaningful data

Unable to input data from file 'C:\Users\jeroen\Downloads\waterschapsgrenzen.dbf'

Failed to specify the feature index as constraints on the reader

The feature cache has not been loaded while a new setConstraints was performed. Now loading...

No shape record found when reading record number '2'.  Shape file is either corrupt or end of file has reached with no meaningful data

Unable to input data from file 'C:\Users\jeroen\Downloads\waterschapsgrenzen.dbf'

Removing dataset 'waterschapsgrenzen [ESRISHAPE]' from View 1

The dataset 'C:\Users\jeroen\Downloads\waterschapsgrenzen.shp' was closed successfully

The 'ESRISHAPE' reader was destroyed successfully

Is there a way to fix this? 

I also saw that the .dbf file has a file size of 8.36KB when uploaded by REST and 8.34KB when i upload it myself using the upload functionality in the Resources folder. Also the other files are larger after uploading.

2 Attachments

10 replies

david_r
Rank
Userlevel 4

I quickly tried using the same endpoint to upload your "temp.zip" file to FME Server 2018.1 and could reproduce the exact same behavior.

Since uploading through the server GUI works without a problem, I'm wondering if Swagger (the tool that's used to generate the interactive API documentation) hasn't been properly set up for this endpoint.

One main difference is that the Content-Type header differs when using the API doc and the GUI:

API documentationFME Server GUIContent-Type

multipart/form-data

application/octet-stream

Have you tried doing resource upload with some other tool, e.g. Postman?

jeroen
Rank
Badge +7

I quickly tried using the same endpoint to upload your "temp.zip" file to FME Server 2018.1 and could reproduce the exact same behavior.

Since uploading through the server GUI works without a problem, I'm wondering if Swagger (the tool that's used to generate the interactive API documentation) hasn't been properly set up for this endpoint. 

One main difference is that the Content-Type header differs when using the API doc and the GUI:

API documentationFME Server GUIContent-Type

multipart/form-data

application/octet-stream

Have you tried doing resource upload with some other tool, e.g. Postman?

 

I have tried to do it with postman but getting stuck on Status 500 Internal Server Error

 

 

    "message": "An unhandled exception has occurred due to some unexpectedly omitted information, consult the web service logs for further details"}
Or  Status: 415 Unsupported Media Type when using another Content-Type.

 

 

I cannot find the log they mention so i do not have any information with that

 

david_r
Rank
Userlevel 4

I tested uploading a file using Postman with the following configuration:

for the body I specified the follwing:

Downloading the file from the FME Server GUI i got the exact same file and could open it in Data Inspector with no issues.

I think this confirms my suspicion about the API docs having an issue with this endpoint.

jeroen
Rank
Badge +7

I quickly tried using the same endpoint to upload your "temp.zip" file to FME Server 2018.1 and could reproduce the exact same behavior.

Since uploading through the server GUI works without a problem, I'm wondering if Swagger (the tool that's used to generate the interactive API documentation) hasn't been properly set up for this endpoint.

One main difference is that the Content-Type header differs when using the API doc and the GUI:

API documentationFME Server GUIContent-Type

multipart/form-data

application/octet-stream

Have you tried doing resource upload with some other tool, e.g. Postman?

 

Using the application/octet-stream status 422 Unprocessable Entity (WebDAV) (RFC 4918) is returned

jeroen
Rank
Badge +7

I tested uploading a file using Postman with the following configuration:

for the body I specified the follwing:

Downloading the file from the FME Server GUI i got the exact same file and could open it in Data Inspector with no issues.

I think this confirms my suspicion about the API docs having an issue with this endpoint.

I'm getting the following result using a Temp.zip file:

 

 

 

david_r
Rank
Userlevel 4
I'm getting the following result using a Temp.zip file:

 

 

 

Seems like you forgot to specify the body contents as well as the headers?
jeroen
Rank
Badge +7

I tested uploading a file using Postman with the following configuration:

for the body I specified the follwing:

Downloading the file from the FME Server GUI i got the exact same file and could open it in Data Inspector with no issues.

I think this confirms my suspicion about the API docs having an issue with this endpoint.

This works, but how can i use it for multiple files?
david_r
Rank
Userlevel 4

This works, but how can i use it for multiple files?
Any particular reason for not simply iterating over each file and upload them individually? Personally, I would recommend that.
jeroen
Rank
Badge +7
Any particular reason for not simply iterating over each file and upload them individually? Personally, I would recommend that.
I prefer to upload all files at once, per job and keep unnecessary loops out of the code.

 

 

 

 

david_r
Rank
Userlevel 4
I prefer to upload all files at once, per job and keep unnecessary loops out of the code.

 

 

 

 

Unfortunately I don't have an example for uploading multiple files, but maybe Google (or someone else here) can help... For example, this might help: https://stackoverflow.com/a/26791188

Reply


Community Stats

28,710
Posts
109,486
Replies
36,559
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.