Hi everyone, having worked in the web scraping industry for a few years I know how easily troublesome it can be to write, maintain and even begin web scraping. One year ago, I wrote a web-scraping guide that was really loved by the community. Reddit post, article. It was actually my first and only gilded post here 😊. Reddit blocks web crawler's especially web scraping and data crawling web crawlers. You can see 80legs (web-crawling company) has been blocked explicitly in their Robots.txt. You will end up in a.
Wednesday, December 04, 2019The latest version for this tutorial is available here. Go to have a check now!
In this tutorial, we are going to show you how to scrape posts from a Reddit group.
- I made a Python web scraping guide for beginners I've been web scraping professionally for a few years and decided to make a series of web scraping tutorials that I wish I had when I started. The series will follow a large project I'm building that analyzes political rhetoric in the news.
- Browse other questions tagged web-scraping reddit or ask your own question. The Overflow Blog Podcast 330: How to build and maintain online communities, from gaming to Level Up: Creative Coding with p5.js – part 6.
- This is the first video of Python Scripts which will be a collection of scripts accomplishing a collection of tasks.This video will introduce you to Reddit A.
To follow through, you may want to use this URL in the tutorial:
We will open every post and scrape the data including the group name, author, title, article, the number of the upvote and that of the comments.
This tutorial will also cover:
· Handle pagination empowered by scrolling down in Octoparse
· Deal with AJAX for opening every Reddit post
· Locate all the posts by modifying the loop mode and XPath in Octoparse
Here are the main steps in this tutorial: [Download task file here ]
1) Go To Web Page - to open the targeted web page
· Click '+ Task' to start a task using Advanced Mode
Advanced Mode is a highly flexible and powerful web scraping mode. For people who want to scrape from websites with complex structures, like Airbnb.com, we strongly recommend Advanced Mode to start your data extraction project.
· Paste the URL into the 'Extraction URL' box and click 'Save URL' to move on
2) Set Scroll Down - to load all items from one page
· Turn on the 'Workflow Mode' by switching the 'Workflow' button in the top-right corner in Octoparse
We strongly suggest you turn on the 'Workflow Mode' to get a better picture of what you are doing with your task, just in case you mess up with the steps.
· Set up Scroll Down
For some websites like Reddit.com, clicking the next page button to paginate is not an option for loading content. To fully load the posts, we need to scroll the page down to the bottom continuously.
· Check the box for 'Scroll down to bottom of the page when finished loading'
· Set up 'Scroll times', 'Interval', and 'Scroll way'
By inputting value X into the 'Scroll times' box, Octoparse will automatically scroll the page down to the bottom for X times. In this tutorial, 1 is inputted for demonstration purposes. When setting up 'Scroll times', you’ll often need to test running the task to see if you have assigned enough times.
'Interval' is the time interval between every two scrolls. In this case, we are going to set 'Interval' as 3 seconds.
For 'Scroll way', select 'Scroll down to the bottom of the page'
· Click 'OK' to save
Tips! To learn more about how to deal with infinite scrolling in Octoparse, please refer to： · Dealing with Infinite Scrolling/Load More |
3) Create a 'Loop Item' - to loop click into each item on each list
· Select the first three posts on the current page
· Click 'Loop click each element' to create a 'Loop Item'
Octoparse will automatically select all the posts on the current page. The selected posts will be highlighted in green with other posts highlighted in red.
· Set up AJAX Load for the 'Click Item' action
Reddit applies the AJAX technique to display the post content and comments thread. Therefore, we need to set up AJAX Load for the 'Click Item' step.
· Uncheck the box for 'Retry when page remains unchanged (use discreetly for AJAX loading)' and 'Open the link in new tab'
· Check the box for 'Load the page with AJAX' and set up AJAX Timeout (2-4 seconds will work usually)
· Click 'OK' to save
Tips! For more about dealing with AJAX in Octoparse: · Deal with AJAX |
4) Extract data - to select the data for extraction
After you click 'Loop click each element', Octoparse will open the first post.
· Click on the data you need on the page
· Select 'Extract text of the selected element' from 'Action Tips'
· Rename the fields by selecting from the pre-defined list or inputting on your own
5) Customize data field by modifying XPath - to improve the accuracy of the item list (Optional)
Once we click “Loop click each element”, Octoparse will generate a loop item using Fixed list loop mode by default. Fixed list is a loop mode used for dealing with a fixed amount of elements. However, the number of posts on Reddit.com is not fixed but increases with scrolling down. In order to enable Octoparse to capture all the posts, including those to be loaded later, we need to swift the loop mode to Variable list and enter the proper XPath to have all the posts to be located.
Reddit Web Scraping Services
· Select 'Loop Item' box
· Select 'Variable list' and enter '//div[contains(@class, 'scrollerItem') and not(contains(@class, 'promote'))]'
· Click 'OK' to save
Tips! 1. 'Fixed list' and 'Variable list' are loop modes in Octoparse. For more about loop modes in Octoparse: ·5 Loop Modes in Octoparse 2. If you want to learn more about XPath and how to generate it, here is a related tutorial you might need： ·Locate elements with XPath |
6) Start extraction - to run the task and get data
· Click “Start Extraction” on the upper left side
· Select “Local Extraction” to run the task on your computer, or select “Cloud Extraction” to run the task in the Cloud (for premium users only)
Here is the sample output.
Reddit Web Scraping Software
Was this article helpful? Feel free to let us know if you have any question or need our assistance.
Reddit Web Scraping Rules
Contact ushere !