Downloading multiple files using PythonCaller | Community

Trying to download multiple files using multiple URLs in PythonCaller. I couldn't use HTTPCaller due to security issues. Please suggest.

Thanks

Page 1 / 1

Curious about the security issues, whats the issue with the HTTPCaller.

Curious about the security issues, whats the issue with the HTTPCaller.

The issue is with the TLS Cipher suites that aren't allowed in Windows servers. So I'm thinking Python is the only alternative here or may be curl ?

The issue is with the TLS Cipher suites that aren't allowed in Windows servers. So I'm thinking Python is the only alternative here or may be curl ?

Oh, I've never heard of that issue. Is it new? Do you just get an error when trying the HTTPCaller or is there some issue when installing. How did you find out about it?

The HTTPCaller is curl based so maybe curl isn't an option (although I'm sure there would be a way to change the cipher?)

For python here is some information, https://stackoverflow.com/questions/46264056/python-download-multiple-files

But in the end it all comes down to the how the input is structured. You best bet it to have a bit of a google to get some snippets to play around with I guess.

The issue is with the TLS Cipher suites that aren't allowed in Windows servers. So I'm thinking Python is the only alternative here or may be curl ?

Ahh I see: https://docs.microsoft.com/en-us/security/engineering/disable-legacy-tls

The issue is with the TLS Cipher suites that aren't allowed in Windows servers. So I'm thinking Python is the only alternative here or may be curl ?

Hmm, HTTPCaller (at least in FME 2020) seems to be ok. when pinging: https://www.howsmyssl.com/

They have an API which you can use to request against. I've attached a sample workspace.

The issue is with the TLS Cipher suites that aren't allowed in Windows servers. So I'm thinking Python is the only alternative here or may be curl ?

Thanks @virtualcitymatt , but HTTPCaller doesn't work either. It has to be PythonCaller though.

The issue is with the TLS Cipher suites that aren't allowed in Windows servers. So I'm thinking Python is the only alternative here or may be curl ?

Oh sorry, howsmyssl.com is a website which checks the client (either browser or in this case the HTTPCaller) to see if there are any security flaws. The HTTPCaller in FME 2020 passed all the tests with no warnings. To me this suggests that the TLS Cipher isn't the issue with the HTTPCaller, but I'm no expert. I'm still trying to understand why the HTTPCaller isn't working for you. Do you have a log file or anything from the HTTPCaller you can share with us?

You can try to turn on Debug logging in FME to see more Information about the request and why it's being rejected.

The issue is with the TLS Cipher suites that aren't allowed in Windows servers. So I'm thinking Python is the only alternative here or may be curl ?

@virtualcitymatt it looks like the workspace isn't compatible with 2019.2 due to HTTPCaller version difference. But nonetheless I think howsmyssl.com is a URL which checks for TLS Ciphers that the URL has. But in this case we know the Ciphers that are no longer present in our windows servers.

So probably making a call in Python should work as it doesn't make use of Ciphers.

The issue is with the TLS Cipher suites that aren't allowed in Windows servers. So I'm thinking Python is the only alternative here or may be curl ?

Ahh I see, you can turn off Certificate validation in the HTTPCaller, maybe that would help. If you need to connect to a website that uses HTTPS then even the python library would need to use some kind of cipher (I would think).

Reply