How to extract PDF attachment from ArcGIS Online and upload to online location with HTTPCaller?

Hi @aaron​ , I did some testing with this today and it did take a fair bit of trial and error to sort out how to get the results from the ArcGIS Online Reader into a useable encoding, but I did manage it. Here's what I did:

• Used a BinaryEncoder first to convert from the original Binary format to Base64
• Then, I connected a BinaryDecoder to that to go from Base64 to System Default (fme-system) encoding (I tested on a Mac, so I suspect 'system default' might mean something different on different operating systems)
• From there, I was able to write the PDF directly to my file system using an AttributeFileWriter

I haven't tried using the output from the BinaryDecoder directly in an HTTPCaller though, so I'm not completely sure if that will work. If it doesn't, one option might be to try writing the pdf to a temp folder first, then upload it from there via the HTTPCaller.

Hope this helps!

Hi @aaron​, Ah! I had a feeling I might have overcomplicated things in my solution! The AttributeFileWriter option you used is definitely a more elegant solution for this.

As for the bug in the BinaryEncoder/Decoder, I'm seeing the same behaviour in build 20806 (2020.2). I'll report that internally to get it fixed. The destination attribute should be exposed automatically. Thanks for reporting this issue!

UPDATE: This issue is being tracked internally as FMEENGINE-59756

Hi @aaron​ , Unfortunately, we don't have a way to set the encoding to Binary (fme-binary) directly in the HTTPCaller. But, I had some success with using my BinaryEncoder/BinaryDecoder trick from my first response to change the attribute containing the PDF binary from fme-binary to something compatible with Binary (application/octet-stream) in the HTTPCaller.

For this, I just fed my arcgisonline_attachment{0}.data into a BinaryEncoder with Encoding Type = HEX and Encode to Different Destination Attributes unchecked (to avoid the bug where the new attributes aren't exposed). Then I passed that newly encoded attribute into a BinaryDecoder with Encoding Type = HEX, Decode to Different Attribute unchecked and the Character Encoding for Output Data set to System Default. In the HTTPCaller, I then set the Content Type for that data to Binary (application/octet-stream) and the API I tested against was then able to recognize the PDF file.

Hi @aaron​ , ah, yes, Sorry! I actually see the same problem too. My original testing consisted of just making sure that the POST request was successful when uploading the PDF, then checking that the file was there. When I download and open that PDF it is corrupt and the file size isn't right (same as what you see). I'll do a bit more digging to see if there's a way to get this to work, but I'm starting to suspect that saving the file first, then uploading it might be the best option here.

Old thread I know, but I came across this issue too (writing corrupt pdf's from an attribute retrieved using HTTP caller). I ended up using python to create the file directly from the attribute:

        # Your Base64 encoded PDF string
        data = feature.getAttribute('Stream')
        outputFilePath = feature.getAttribute('OutputFilePath')
 
        # Decode the Base64 string
        decoded_data = base64.b64decode(data)
 
        # Create directory if it does not exist
        directory = os.path.dirname(outputFilePath)
        if not os.path.exists(directory):
            os.makedirs(directory)
 
        # Write the binary data to a PDF file
        with open(outputFilePath, 'wb') as f:
            f.write(decoded_data)
        self.pyoutput(feature)