Python Programming Web Scraping With Requests And Beautifulsoup Complete Guide

1. Introduction to Web Scraping

Web scraping involves retrieving information from a website automatically. In Python, you can use the requests library to fetch the HTML content of a webpage and BeautifulSoup from the bs4 package to parse and extract data from this HTML.

2. Install Required Libraries

Before you start, ensure you have the required libraries installed. You can install them using pip:

pip install requests
pip install beautifulsoup4

3. Basic Example: Fetching a Webpage

Let's begin by fetching the HTML content of a webpage.

import requests

# URL of the webpage you want to scrape
url = 'https://example.com'

# Send an HTTP request to the server
response = requests.get(url)

# Check if the request was successful
if response.status_code == 200:
    print("Webpage fetched successfully!")
else:
    print(f"Failed to fetch webpage. Status code: {response.status_code}")

# Print the fetched HTML content
print(response.text)

4. Parsing HTML with BeautifulSoup

Now that we have the HTML content, let's parse it using BeautifulSoup.

from bs4 import BeautifulSoup
import requests

# URL of the webpage you want to scrape
url = 'https://example.com'

# Send an HTTP request to the server
response = requests.get(url)

# Check if the request was successful
if response.status_code == 200:
    print("Webpage fetched successfully!")
    
    # Parse the HTML content using BeautifulSoup
    soup = BeautifulSoup(response.text, 'html.parser')
    
    # Print the prettified version of the parsed HTML
    print(soup.prettify())
else:
    print(f"Failed to fetch webpage. Status code: {response.status_code}")

5. Extracting Specific Data

Suppose we wanted to extract all the links from the webpage (<a> tags).

from bs4 import BeautifulSoup
import requests

# URL of the webpage you want to scrape
url = 'https://example.com'

# Send an HTTP request to the server
response = requests.get(url)

# Check if the request was successful
if response.status_code == 200:
    print("Webpage fetched successfully!")
    
    # Parse the HTML content using BeautifulSoup
    soup = BeautifulSoup(response.text, 'html.parser')
    
    # Extract all links on the webpage
    links = soup.find_all('a')
    
    # Print each link
    for link in links:
        print(link.get('href'))
else:
    print(f"Failed to fetch webpage. Status code: {response.status_code}")

6. Extracting Headings

Now, let’s extract all headings (e.g., <h1>, <h2>, etc.).

from bs4 import BeautifulSoup
import requests

# URL of the webpage you want to scrape
url = 'https://example.com'

# Send an HTTP request to the server
response = requests.get(url)

# Check if the request was successful
if response.status_code == 200:
    print("Webpage fetched successfully!")
    
    # Parse the HTML content using BeautifulSoup
    soup = BeautifulSoup(response.text, 'html.parser')
    
    # List of heading tags to extract
    heading_tags = ['h1', 'h2', 'h3', 'h4', 'h5', 'h6']
    
    # Extract and print all headings
    for tag in heading_tags:
        headings = soup.find_all(tag)
        for heading in headings:
            print(tag, ":", heading.text.strip())
else:
    print(f"Failed to fetch webpage. Status code: {response.status_code}")

7. Working with Attributes

Sometimes, you might need to work with attributes within your tags. For example, extracting the title attribute of images.

from bs4 import BeautifulSoup
import requests

# URL of the webpage you want to scrape
url = 'https://example.com'

# Send an HTTP request to the server
response = requests.get(url)

# Check if the request was successful
if response.status_code == 200:
    print("Webpage fetched successfully!")
    
    # Parse the HTML content using BeautifulSoup
    soup = BeautifulSoup(response.text, 'html.parser')
    
    # Extract all image tags
    images = soup.find_all('img')
    
    # Print the source and title attributes of each image
    for img in images:
        print(img.get('src'), ":", img.get('title', '--No Title--'))
else:
    print(f"Failed to fetch webpage. Status code: {response.status_code}")

8. Scraping Data Within a Specific Class

To extract data from elements of a specific class, you can do this as follows:

from bs4 import BeautifulSoup
import requests

# URL of the webpage you want to scrape
url = 'https://example.com'

# Send an HTTP request to the server
response = requests.get(url)

# Check if the request was successful
if response.status_code == 200:
    print("Webpage fetched successfully!")
    
    # Parse the HTML content using BeautifulSoup
    soup = BeautifulSoup(response.text, 'html.parser')
    
    # Find all div elements with the class name "content"
    contents = soup.find_all('div', {'class': 'content'})
    
    # Loop through found elements and print their text
    for content in contents:
        print(content.text.strip())
else:
    print(f"Failed to fetch webpage. Status code: {response.status_code}")

9. Complete Example: Scrape Titles and Links From News Articles

Let’s combine all the above into a more practical example. Suppose we want to scrape news titles and their corresponding links from a news website like BBC News.

Here are the steps:

• Inspect the website to find out where the titles and links are located.
• Use the appropriate tags and classes in your script to locate the data.
• Extract and print the data.

Note: The actual tags and classes will vary depending on the structure of the website. Here, I'll assume certain tags and classes for demonstration.

from bs4 import BeautifulSoup
import requests

# URL of the news page you want to scrape
url = 'https://www.bbc.com/news'

# Send an HTTP request to the server
response = requests.get(url)

# Check if the request was successful
if response.status_code == 200:
    print("News page fetched successfully!")

    # Parse the HTML content using BeautifulSoup
    soup = BeautifulSoup(response.text, 'html.parser')

    # Assume all news titles are inside <a> tags under <h3> with class="gs-c-promo-heading__title"
    headlines = soup.find_all('h3', {'class': 'gs-c-promo-heading__title'})

    for headline in headlines:
        title = headline.text.strip()
        link = headline.find('a')['href']
        
        # Append the base URL to get the full link
        if not link.startswith('http'):
            link = 'https://www.bbc.com' + link
        
        print(title)
        print(link)
        print()
else:
    print(f"Failed to fetch news page. Status code: {response.status_code}")

10. Handling Dynamic Content

Many websites now use JavaScript to load content dynamically after the initial HTML has been loaded. For such cases, you might need to use a tool like Selenium or Playwright.

However, many sites serve the dynamic content using Ajax requests, and you can still get the data using requests by finding these Ajax endpoints.

11. Respect Website’s Terms and Conditions

Always check the website's robots.txt file and terms of service before scraping. Some websites prohibit scraping activities or have guidelines for robots/web crawlers.

robots.txt URL Example:
https://example.com/robots.txt

12. Conclusion

You have now learned how to fetch a webpage using requests, parse HTML with BeautifulSoup, and extract specific pieces of data. With these foundational skills, you can start building more complex scrapers to gather the data you need from various websites.

Additional Resources

• Requests Documentation: https://docs.python-requests.org/en/latest/
• BeautifulSoup Documentation: https://www.crummy.com/software/BeautifulSoup/bs4/doc/
• Selenium Documentation: https://selenium-python.readthedocs.io/
• Playwright Documentation: https://playwright.dev/python/docs/intro