I'm trying to get imdb movie reviews , for example the movie "The Matrix" has 4K reviews and I need to retrieve all of it.
Question
How to scrape data from websites with a “Load More” button?
This post is closed to further activity.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
It may be an old question, an answered question, an implemented idea, or a notification-only post.
Please check post dates before relying on any information in a question or answer.
For follow-up or related questions, please post a new question or idea.
If there is a genuine update to be made, please contact us and request that the post is reopened.
Hi @dani_kelevra,
It looks like IMDb has a number of unofficial APIs which are free for a certain number of requests. It may be worth exploring this as another option to work with instead of scraping data directly from the web page, since there are some examples and documentation it may be easier to work with pagination.
E.g. https://rapidapi.com/apidojo/api/imdb8?endpoint=apiendpoint_a4ba0a9f-9314-45d1-b50f-3a6b7cb34c69
