How to automatically turn to the next page to collect data?

In the previous chapters, we have learned how to extract data from pages.

In this tutorial, we will learn how to automatically turn pages to collect data?

Now open our practice page: Best Seller Books

Yes, we need to collect data page by page.

Task Analysis

Before scraping the data, we need to break down the task. In this case, the steps are as follows:

1. Open link Best Seller Books
2. Scroll page to bottom
3. Extract data
4. Click the "Next" button
5. Wait page loaded
6. Scroll page to bottom
7. Extract data
8. Click the "Next" button
9. Wait page loaded
10. ...

Repeat steps 2 ~ 5 until the list scraping is complete.

Create Recipe

OK, through the task analysis above, we clear what to do in each step?

Let's get started!

In order to get to the point quickly, we chose to develop secondary recipes based on first-recipe.

First, we find the first-recipe recipe and copy it as auto-next-page.

Then, edit the auto-next-page recipe.

Step 1. Add the node "Enter Loop" under the "Open Page" node, the configuration is as follows.

1. Loop Type chose Load Times
2. Source Type）chose From User Input
3. Arg Name fill in Next Page（You can enter anything you like）
4. Default Value change to 3（other values can be also）

Step 2. Add a "Scroll Page" node, use the default value, do not need to change.

1. Add a "Scroll Page" node under the "Load More" node
2. Drag the "Scroll Page" node to the first position in the loop

Step 3. Drag the "Extract Data" node below the "Scroll Page" node.

Step 4. Edit the "Load More" node, which is the key to turning the page. The configuration is as follows:

1. Fill out the target element selector: button[aria-label="Go\ to\ next\ page"]（you can generate it via Advanced Finder）

Step 5. Add a "Wait Moment" node under the "Load More" node.

1. Change "Page Loaded" to "Page Changed"

At this point, all nodes are configured, click the "Save" button to save it.

Summarize

The recipe for automatically turning pages to collect data has been developed, let us see what it looks like?

all right! This is the end of this tutorial, thank you for reading, go and try it now! 👉