Hello All,
Is it possible to perform any sort of web crawling via FME? I have a button the website which downloads documents for me. I wish to automate it by some sort of crawling from FME which reaches the site and downloads the data.
Kindly help.
Userlevel 5
+25
- Influencer
- 3345 replies
-
16 October 2019
You can use an HTTPCaller to download the web page and then parse through it. Depending on how it's built this may or may not give you access to the data. Plain HTML usually works ok but dynamic stuff can be a hassle.
Userlevel 4
+25
- Safer
- 2399 replies
-
16 October 2019
A link/URL is fairly simple, but to replicate the action of a button is trickier. Take a look at the source code for the page, then see if the HTTPCaller can be used to replicate it. Like @redgeographics says, you want to hope that they are using HTML and the button has a formaction and formmethod attribute. That you can easily mimic in a HTTPCaller. But if the site uses Javascript or similar then I don't know what you would do.
+16
- Contributor
- 1425 replies
-
16 October 2019
Just as @mark2atsafe says replicating the action of a button can be tricky. Using developers tools of your browser of choice might provide some insights on what is happening under the hood.
Reply
Enter your username or e-mail address. We'll send you an e-mail with instructions to reset your password.