Question

HttpCaller: send POST and then problems writing download stream to zip file

peteralstorp
Rank
Badge

Hello FME-community. 


I'm having trouble downloading data with the HttpCaller to a zip file. 


1. First I upload text/xml below with a file key to say which file I want to download. See image . 

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <GetOrder xmlns="http://Vagverket.VIS.Lastkajen.LastkajenLogistics">
      <orderKey>@Value(FileKey)</orderKey>
    </GetOrder>
  </soap:Body>
</soap:Envelope>

2. I then receive (according to the data download specification) a download stream in base64Binary, see example below. 

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <GetOrderResponse xmlns="http://Vagverket.VIS.Lastkajen.LastkajenLogistics">
      <GetOrderResult>base64Binary</GetOrderResult>
    </GetOrderResponse>
  </soap:Body>
</soap:Envelope>

3. I use the XMLFlattener to filter out GetOrderResult, BUT I can't seem to get the download stream into a proper functional zip file. I have tried numerous ways (writing to a file directly from the httpcaller, using the datafile-writer and the attributewriter but I just can't make it work). 

GetOrderResult (base64Binary) follows. This data contains, if I'm correct, a zip file containing a shape file TESTNVDB_DKC_Cykelled.shp (the file contains zero objects), a dbf file TESTNVDB_DKC_Cykelled_domainmap.dbf and a text file.

UEsDBBQAAAAIAG+HuUiDBir8bQAAACIBAAAZACQAVEVTVE5WREJfREtDX0N5a2VsbGVkLmRiZgoAIAAAAAAAAQAYAMlbqgqWttEBNsrpEJa20QE2yukQlrbRAWMuYZVkAAJFxhUMGCCcgcHP0dcPynMGEUZoSgKCPH0d/cJATLA6FjT5IB9PF2T9amjywSGOQSGOYcEhDAxuIL4wN5q8TyhUGoe8h2OAa7yPqx8OebcgR794F8eQUF+I+zjR7A/x9PHBJ88rBQBQSwMEFAAAAAgAb4e5SPQgcxb8AAAAhwEAABkAJABURVNUTlZEQl9ES0NfQ3lrZWxsZWQucHJqCgAgAAAAAAABABgAyVuqCpa20QE2yukQlrbRATbK6RCWttEBZY9Bb4MgGIb/C2diAGuVo6nobKYYtOnBGEIssyTGJsi2vz80WbJ2HDg87/O9HzSCn09tD9orEyynVHYVgAXjxQb9JX8DALO0u1Q9yP6gtnljgpeZV0UrMU0QgMcwTnAYBwgSmgQkigkhGOFhgI0oK+YbCqv18m3GO4AoQAO81GXne/Xk+Y5wfIhCQkmEKT2E4T7Kz+zUlbzuQWfVsn5pu2pZaTsq97DAG6lIK9Yx0YNczT5janVmmQCM0Ha2Rf+c+mHdfZde85NenFXztsHcjFoAxNGL0o5q1jJX4/YAX0ApPT4J78oZ93nTkn9Ibs1klqcPV9ppP4g9GX4AUEsDBBQAAAAIAG+HuUhKJdNaHQAAAGQAAAAZACQAVEVTVE5WREJfREtDX0N5a2VsbGVkLnNocAoAIAAAAAAAAQAYACe+rAqWttEBNsrpEJa20QE2yukQlrbRAWNgUOdiwA6MXjAzMPDikGRg+LAfRv8Hg/f/cdEAUEsDBBQAAAAIAG+HuUhKJdNaHQAAAGQAAAAZACQAVEVTVE5WREJfREtDX0N5a2VsbGVkLnNoeAoAIAAAAAAAAQAYACe+rAqWttEBbyzsEJa20QFvLOwQlrbRAWNgUOdiwA6MXjAzMPDikGRg+LAfRv8Hg/f/cdEAUEsDBBQAAAAIAHGHuUjkPO7Z0wAAALoQAAAjACQAVEVTVE5WREJfREtDX0N5a2VsbGVkX2RvbWFpbm1hcC5kYmYKACAAAAAAAAEAGACDUoYNlrbRAW8s7BCWttEBbyzsEJa20QFjLmGV5GBgYGhkYGdiQAfhDAyOJSVFmUmlJakMDM4gIS40Jc75KakwJoj4hyYflphTmopHnlchIMjT19EvTEFBwVBhJIOC1LyUnMy89OKc1JSBdssAAEQyMBpopwwoSK7MTs0pKS3KLM4diQkBkQyMB9opAwpy80sy8/NGaFmAnAxMBtopAwpQKoX85IwRVjwgkoHpQDtlQAFGMhhZxQMiGZgNtFMGFKBl/pGWEBDJwHygnTKgALk00MGWKIZ3qkAkA8uBdsqAgvzsJ3nDMX6JA1IAUEsDBBQAAAAIAHGHuUixR8oV9QAAAKkBAAAmACQAVEVTVE5WREJfREtDX0N5a2VsbGVkX2ZpZWxkYWxpYXNlcy50eHQKACAAAAAAAAEAGADHjYENlrbRAW8s7BCWttEBbyzsEJa20QGVjz1uwzAMhXcBugPHdOjSGwhwghqwXcM/WQM1oh3BslLQsk/kXCEX0MUqIW7RJUO5kHji9/R40GhUIUcEEEbLKY6cFSIv4FFBsJyVVZqL4hiFkvToVwJpF79aZbXtOauyNNmACjsktJMJr0Po8JEmnNWNqBpxrJu4YKTTiwO5TM7frILO3wkmJ8nBl7+Fifx6Hmb3Sbj511m70U9xMz+n30W5P2X7cFP7K8doPV5HdKSnQULM2yvQ8KY4O1SiOCWiaXNIpJtH6Ch8BdfzBUZUsGgzoIP5jxn0fjUGibMmzbJ/sto6fAQg/DGCndPGvKpo8cLZN1BLAQItABQAAAAIAG+HuUiDBir8bQAAACIBAAAZACQAAAAAAAAAIAAAAAAAAABURVNUTlZEQl9ES0NfQ3lrZWxsZWQuZGJmCgAgAAAAAAABABgAyVuqCpa20QE2yukQlrbRATbK6RCWttEBUEsBAi0AFAAAAAgAb4e5SPQgcxb8AAAAhwEAABkAJAAAAAAAAAAgAAAAyAAAAFRFU1ROVkRCX0RLQ19DeWtlbGxlZC5wcmoKACAAAAAAAAEAGADJW6oKlrbRATbK6RCWttEBNsrpEJa20QFQSwECLQAUAAAACABvh7lISiXTWh0AAABkAAAAGQAkAAAAAAAAACAAAAAfAgAAVEVTVE5WREJfREtDX0N5a2VsbGVkLnNocAoAIAAAAAAAAQAYACe+rAqWttEBNsrpEJa20QE2yukQlrbRAVBLAQItABQAAAAIAG+HuUhKJdNaHQAAAGQAAAAZACQAAAAAAAAAIAAAAJcCAABURVNUTlZEQl9ES0NfQ3lrZWxsZWQuc2h4CgAgAAAAAAABABgAJ76sCpa20QFvLOwQlrbRAW8s7BCWttEBUEsBAi0AFAAAAAgAcYe5SOQ87tnTAAAAuhAAACMAJAAAAAAAAAAgAAAADwMAAFRFU1ROVkRCX0RLQ19DeWtlbGxlZF9kb21haW5tYXAuZGJmCgAgAAAAAAABABgAg1KGDZa20QFvLOwQlrbRAW8s7BCWttEBUEsBAi0AFAAAAAgAcYe5SLFHyhX1AAAAqQEAACYAJAAAAAAAAAAgAAAARwQAAFRFU1ROVkRCX0RLQ19DeWtlbGxlZF9maWVsZGFsaWFzZXMudHh0CgAgAAAAAAABABgAx42BDZa20QFvLOwQlrbRAW8s7BCWttEBUEsFBgAAAAAGAAYAmQIAAKQFAAAAAA==

Any suggestions helping me out here are welcome. 

Kind regards, 
Peter

10 replies

takashi
Rank
Userlevel 2
Badge +17

Hi @peteralstorp, did you try saving the response into a file by the HTTPCaller with setting File to the 'Save Response Body To' parameter?

peteralstorp
Rank
Badge

Hi @peteralstorp, did you try saving the response into a file by the HTTPCaller with setting File to the 'Save Response Body To' parameter?

Thanks, takashi. Yes I did. I figure it must be a character encoding problem or maybe I need to understand the zip format better.

takashi
Rank
Userlevel 2
Badge +17

Thanks, takashi. Yes I did. I figure it must be a character encoding problem or maybe I need to understand the zip format better.

I think you can read the zip file with a FeatureReader following to the HTTPCaller, once you have saved it into the disk. Have you tried this?

peteralstorp
Rank
Badge

I think you can read the zip file with a FeatureReader following to the HTTPCaller, once you have saved it into the disk. Have you tried this?

I think the zip file is the problem. I tried your suggestion (although I need to pass an xml-transformer to get rid of some xml-code. See image. takashi.png. The featurereader will not accept the zipfile.

takashi
Rank
Userlevel 2
Badge +17

I think the zip file is the problem. I tried your suggestion (although I need to pass an xml-transformer to get rid of some xml-code. See image. takashi.png. The featurereader will not accept the zipfile.

How about setting fme-binary to the 'Target File Character Encoding' parameter of the AttributeFileWriter?

peteralstorp
Rank
Badge

I think the zip file is the problem. I tried your suggestion (although I need to pass an xml-transformer to get rid of some xml-code. See image. takashi.png. The featurereader will not accept the zipfile.

Tried it but no. The zip file can't be read properly. Error msg: Error status STK_FAILURE pending -- message is: Error while processing folders

takashi
Rank
Userlevel 2
Badge +17

I think the zip file is the problem. I tried your suggestion (although I need to pass an xml-transformer to get rid of some xml-code. See image. takashi.png. The featurereader will not accept the zipfile.

Well, the TextDecoder might help you to decode the Base64 string to an fme-binary string before writing.

  • Encoding Type: Base64
  • Character Encoding for Binary Data: Binary (fme-binary)
peteralstorp
Rank
Badge

I think you can read the zip file with a FeatureReader following to the HTTPCaller, once you have saved it into the disk. Have you tried this?

Hm, still no readable zip file. Thanks for your efforts so far, Takashi!

takashi
Rank
Userlevel 2
Badge +17

Thanks, takashi. Yes I did. I figure it must be a character encoding problem or maybe I need to understand the zip format better.

I was able to read your sample Base64 string as a zipped Shepfile dataset. See the attachment: base64-to-fme-binary.fmw (FME 2016.1)

peteralstorp
Rank
Badge

I was able to read your sample Base64 string as a zipped Shepfile dataset. See the attachment: base64-to-fme-binary.fmw (FME 2016.1)

Takashi, you are the king! Thank you kindly. I can't yet see what you did different than me, but there must be something!

Reply


Community Stats

28,707
Posts
109,479
Replies
36,552
Members
Terms & ConditionsCookie settings
A product of
Safe Software respectfully acknowledges that we live, learn and work on the traditional and unceded territories of the Kwantlen, Katzie, and Semiahmoo First Nations.