Trying to download multiple files using multiple URLs in PythonCaller. I couldn't use HTTPCaller due to security issues. Please suggest.
Thanks
Question
Downloading multiple files using PythonCaller
This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
+11- Contributor
Curious about the security issues, whats the issue with the HTTPCaller.
The issue is with the TLS Cipher suites that aren't allowed in Windows servers. So I'm thinking Python is the only alternative here or may be curl ?
+48- Celebrity
The issue is with the TLS Cipher suites that aren't allowed in Windows servers. So I'm thinking Python is the only alternative here or may be curl ?
Oh, I've never heard of that issue. Is it new? Do you just get an error when trying the HTTPCaller or is there some issue when installing. How did you find out about it?
The HTTPCaller is curl based so maybe curl isn't an option (although I'm sure there would be a way to change the cipher?)
For python here is some information, https://stackoverflow.com/questions/46264056/python-download-multiple-files
But in the end it all comes down to the how the input is structured. You best bet it to have a bit of a google to get some snippets to play around with I guess.
+48- Celebrity
The issue is with the TLS Cipher suites that aren't allowed in Windows servers. So I'm thinking Python is the only alternative here or may be curl ?
Ahh I see: https://docs.microsoft.com/en-us/security/engineering/disable-legacy-tls
+48- Celebrity
The issue is with the TLS Cipher suites that aren't allowed in Windows servers. So I'm thinking Python is the only alternative here or may be curl ?
Hmm, HTTPCaller (at least in FME 2020) seems to be ok. when pinging: https://www.howsmyssl.com/
They have an API which you can use to request against. I've attached a sample workspace.
+11- Contributor
The issue is with the TLS Cipher suites that aren't allowed in Windows servers. So I'm thinking Python is the only alternative here or may be curl ?
Thanks @virtualcitymatt , but HTTPCaller doesn't work either. It has to be PythonCaller though.
+48- Celebrity
The issue is with the TLS Cipher suites that aren't allowed in Windows servers. So I'm thinking Python is the only alternative here or may be curl ?
Oh sorry, howsmyssl.com is a website which checks the client (either browser or in this case the HTTPCaller) to see if there are any security flaws. The HTTPCaller in FME 2020 passed all the tests with no warnings. To me this suggests that the TLS Cipher isn't the issue with the HTTPCaller, but I'm no expert. I'm still trying to understand why the HTTPCaller isn't working for you. Do you have a log file or anything from the HTTPCaller you can share with us?
You can try to turn on Debug logging in FME to see more Information about the request and why it's being rejected.
+11- Contributor
The issue is with the TLS Cipher suites that aren't allowed in Windows servers. So I'm thinking Python is the only alternative here or may be curl ?
@virtualcitymatt it looks like the workspace isn't compatible with 2019.2 due to HTTPCaller version difference. But nonetheless I think howsmyssl.com is a URL which checks for TLS Ciphers that the URL has. But in this case we know the Ciphers that are no longer present in our windows servers.
So probably making a call in Python should work as it doesn't make use of Ciphers.
+48- Celebrity
The issue is with the TLS Cipher suites that aren't allowed in Windows servers. So I'm thinking Python is the only alternative here or may be curl ?
Ahh I see, you can turn off Certificate validation in the HTTPCaller, maybe that would help. If you need to connect to a website that uses HTTPS then even the python library would need to use some kind of cipher (I would think).