Skip to main content
Question

cannot upload zip using REST API POST, response 422 - unable to open shape file ... for reading

C

I am using the REST api to POST to transformations/transactdata. The Workspace that processes shapefiles works fine on Desktop and also runs in the cloud using Run Workspace, by dragging in a zipped shapefile. However, the job will not work through the API using any of these three methods:

- when using the REST API interface, (POST /transformations/transactdata/< repository >/< workspace >)

--- though I am able to see zip file data when choosing application/octet-stream as the Content Type, so I would think this would work

- when using Postman

- when submitting with POST from my own code

In all cases I receive error 422, and a response like this:

{

 

...

 

"priority":-1,

 

"statusMessage":"Unable to open shape file `/data/fmeserver/resources/system/temp/tomcat/tempstore/fmerest/sourcedata/pgworkflows-shp2geojson_cloud.fmw/requestdata-ESRISHAPE_1.shp' for read",

 

"status":"FME_FAILURE"

 

}

 

Can anyone provide insight? The full logfile is here. I am POSTing zipped shapefiles to the API which do work, as I said, when running the workspace from the cloud.

 

The successful jobs using "Run Workspace" have logfiles that do not mention the directory /data/.../tomcat/tempstore.... so I do believe there is something that can be determined from the failure at that step. Instead, they contain log messages like this:

 

44Using Shape Reader to read shape files from folder `/fme_temp/engine/fmeengines/localhost_Engine1/FME_TempFileStore_1567009985695_29006/dsESRISHAPE_1_1567009985695_29006/155910-1_NTM_Points_Points_windo_1567009985695_29006/'

 

 

 

 

 

 

The failed job logfile, which results from any of these three methods of submitting a zip file via POST - API REST page on the server, from Postman, or from our code:

 

 

1FME 2019.0.0.0 (20190328 - Build 19238 - linux-x64)2FME_HOME is '/opt/fme-engine-2019/'3FME Engine (node locked-crc)4Serial Number: SAFE5Machine host name is: ip-10-0-0-2426START - ProcessID: 4045, peak process memory usage: 18877 kB, current process memory usage: 18877 kB7FME Configuration: Command line arguments are `fme' `pgworkflows/shp2geojson_cloud/shp2geojson_cloud.fmw' `--FME_SHAREDRESOURCE_SYSTEM' `/data/fmeserver/resources/system/' `-ESRISHAPE_1_DATASET' `/data/fmeserver/resources/system/temp/tomcat/tempstore/fmerest/sourcedata/pgworkflows-shp2geojson_cloud[1].fmw/requestdata-ESRISHAPE_1' `--FME_SERVER_PORT' `7071' `-LOG_FILENAME' `/data/fmeserver/resources/logs/engine/current/jobs/0/job_640.log' `--FME_JOB_ID' `640' `--FME_MF_NAME' `shp2geojson_cloud.fmw' `--FME_SERVER_ROOT' `/data/fmeserver' `--FME_SECURITY_USER' `superuser' `--FME_SHAREDRESOURCE_BACKUP' `/data/fmeserver/resources/backups/' `--FME_SERVER_HOST' `localhost' `--FME_SHAREDRESOURCE_TEMP' `/data/fmeserver/resources/temp/' `--FME_SERVER_WEB_URL' `https://production-*******.fmecloud.com' `--FME_SHAREDRESOURCE_ENGINE' `/data/fmeserver/resources/engine/' `--FME_SHAREDRESOURCE_LOG' `/data/fmeserver/resources/logs/' `--FME_SECURITY_ROLES' `fmeadmin fmesuperuser user:superuser' `--FME_ENGINE' `localhost_Engine1' `--FME_SHAREDRESOURCE_DASHBOARD' `/data/fmeserver/resources/dashboards/' `-FME_ENGINE_MEMORY_REDLINE' `0.5' `--FME_SHAREDRESOURCE_DATA' `/data/fmeserver/resources/data/' `--FME_DATA_REPOSITORY' `/data/fmeserver/resources/system/temp/upload/' `--FME_JOB_CONTEXT' `context-id-9857a6d2-0e88-4912-a7f4-eb8ea8930134'8FME Configuration: Connection Storage: 'localhost'9Shared folders for formats are : /opt/fme-engine-2019/datasources;/data/fmeserver/resources/engine/Formats;/home/fmeserver/.fme/Formats10Shared folders for transformers are : /home/fmeserver/.Safe Software/FME/Packages/19238-linux-x64/transformers;/opt/fme-engine-2019/transformers;/data/fmeserver/resources/engine/Transformers11Shared folders for coordinate systems are : /data/fmeserver/resources/engine/CoordinateSystems;/home/fmeserver/.fme/CoordinateSystems12Shared folders for coordinate system exceptions are : /data/fmeserver/resources/engine/CoordinateSystemExceptions;/home/fmeserver/.fme/CoordinateSystemExceptions13Shared folders for coordinate system grid overrides are : /data/fmeserver/resources/engine/CoordinateSystemGridOverrides;/home/fmeserver/.fme/CoordinateSystemGridOverrides14Shared folders for CS-MAP transformation exceptions are : /data/fmeserver/resources/engine/CsmapTransformationExceptions;/home/fmeserver/.fme/CsmapTransformationExceptions15Shared folders for transformer categories are : /home/fmeserver/.fme/TransformerCategories16FME Configuration: Reader Keyword is `MULTI_READER'17FME Configuration: Writer Keyword is `MULTI_WRITER'18FME Configuration: Writer Group Definition Keyword is `MULTI_WRITER_DEF'19FME Configuration: Reader type is `MULTI_READER'20FME Configuration: Writer type is `MULTI_WRITER'21FME Configuration: No destination coordinate system set22FME Configuration: Current working folder is `/data/fmeserver/repositories'23FME Configuration: Temporary folder is in the system location `/fme_temp/engine/fmeengines/localhost_Engine1'24FME Configuration: FME_HOME is `/opt/fme-engine-2019/'25FME Configuration: FME_BASE is 'no'26FME Configuration: FME_MF_DIR is '/data/fmeserver/repositories/pgworkflows/shp2geojson_cloud/'27FME Configuration: FME_MF_NAME is 'shp2geojson_cloud.fmw'28FME Configuration: FME_PRODUCT_NAME is 'FME(R) 2019.0.0.0'29System Status: 9.18 GB of disk space available in the FME temporary folder (/fme_temp/engine/fmeengines/localhost_Engine1)30System Status: 3.28 GB of virtual memory available31Operating System: Ubuntu 16.04.6 LTS (Xenial Xerus)32FME Platform: linux-x6433The memory allocator being used by FME is : `/usr/lib/libtcmalloc_minimal.so.4'34Locale: en_US35FME Configuration: Process limit is 3.86 GB of physical memory36FME Configuration: Start freeing memory when process usage exceeds 2.11 GB of virtual memory37FME Configuration: Stop freeing memory when process usage is below 1.58 GB of virtual memory38Creating writer for format:39Creating reader for format:40MULTI_READER(MULTI_READER): Will fail with first member reader failure41MULTI_READER(MULTI_READER): Adding ESRISHAPE Reader with keyword ESRISHAPE_142Using Multi Reader with keyword `MULTI_READER' to read multiple datasets43Creating reader for format: Esri Shapefile44Using Shape Reader to read shape files from folder `/data/fmeserver/resources/system/temp/tomcat/tempstore/fmerest/sourcedata/pgworkflows-shp2geojson_cloud.fmw/'45ESRISHAPE_1 Reader: Using enhanced geometry46Shape Reader: Using simple donut geometry detection47Shape Reader: Handling numeric types with standard integer and floating point types48Shape Reader: Allowing attribute names with mixed case49Unable to open shape file `/data/fmeserver/resources/system/temp/tomcat/tempstore/fmerest/sourcedata/pgworkflows-shp2geojson_cloud.fmw/requestdata-ESRISHAPE_1.shp' for read50Translation FAILED with 1 error(s) and 0 warning(s) (0 feature(s) output)51FME Session Duration: 0.1 seconds. (CPU: 0.0s user, 0.0s system)52END - ProcessID: 4045, peak process memory usage: 18973 kB, current process memory usage: 18790 kB

3 replies

C
  • csplanit
  • Author
  • August 28, 2019

I also tried copying the shapefile (zipped and unzipped) to a server of ours, then added an opt_geturl parameter in the REST API form for this endpoint on the server. If I put in an incorrect value, I do get a 404 for fetching the external resource, and if I put in the right name for either the shp or zipfile, I get the same error as above back in JSON:

 

{

....,

"statusMessage": "Unable to open shape file `/data/fmeserver/resources/system/temp/tomcat/tempstore/fmerest/sourcedata/pgworkflows-shp2geojson_cloud.fmw/requestdata-ESRISHAPE_1.shp' for read",

"status": "FME_FAILURE"

}

H
Forum|alt.badge.img+2

Hi @csplanit,

I believe this is a limitation of the transactdata API endpoint. The issue appears to be that when the file is uploaded we do not send the filename up to Server so it is stripped of the file extension. Therefore when the file path is input into the Reader it will add the extension of that specific format, so in your case it is trying to read the entire zip file as a single .shp entity and thus failing. I have filed FMESERVER-13396 to see if we can have this addressed.

Unfortunately I am not aware of any workarounds, but please share on this post if you are able to find one.

C
  • csplanit
  • Author
  • August 28, 2019
hollyatsafe wrote:

Hi @csplanit,

I believe this is a limitation of the transactdata API endpoint. The issue appears to be that when the file is uploaded we do not send the filename up to Server so it is stripped of the file extension. Therefore when the file path is input into the Reader it will add the extension of that specific format, so in your case it is trying to read the entire zip file as a single .shp entity and thus failing. I have filed FMESERVER-13396 to see if we can have this addressed.

Unfortunately I am not aware of any workarounds, but please share on this post if you are able to find one.

Thanks Holly - thanks for looking into a fix. There is reason to doubt that is the cause - please see my comment, that I also tried uploading an .shp (by itself) and using the opt_geturl to point to the .shp on a server. In those cases I get the same error. (in other words, submitting a single shape file doesn't result in a different error as I would suspect). Please know two other working endpoints I use presently submit .json and .dwg files without issue, so I have been using this very technique in previous weeks and have working enpoints to which I submit files in this same manner.

cad_import_1_cloud.fmw is one of the workflows using this exact technique, in which I stream a file to an endpoint. I think I'm missing some small detail on my end, and if I catch it of course I'll let you know - but if not, there must be something different about the job.

The other is cad_export_1_cloud.fmw, which takes a json file in the POST body.

UPDATE - I did try again, uploading both a zipped shape file collection and an individual shp file to the endpoint, and as with many other attempts using files with different names, I got the same error message:

statusMessage: 'Unable to open shape file `/data/fmeserver/resources/system/temp/tomcat/tempstore/fmerest/sourcedata/pgworkflows-shp2geojson_cloud.fmw/requestdata-ESRISHAPE_1.shp\\' for read',

So, I see where the file is probably being renamed on the server in that temp folder, but again cannot see yet why this isn't working using this endpoint, while it is with the other workspaces mentioned,

Reply


Helpful Members This Week

Community Stats

31,340
Posts
118,850
Replies
38,929
Members

Latest FME

Terms & ConditionsCookie settings

Cookie policy

We use cookies to enhance and personalize your experience. If you accept you agree to our full cookie policy. Learn more about our cookies.

 
Cookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.