← Back to Blog

How to Extract Tables from HTML with Python and Pandas

As a data scientist or software engineer, you’ve probably encountered the challenge of extracting data from HTML files. HTML tables can be a valuable source of data, but extracting them can be a time-consuming process. Luckily, Python and Pandas can make this process much easier. In this article, we will explain how to extract tables from HTML files using Python and Pandas.
By Saturn Cloud | | Miscellaneous | Updated:

As a data scientist or software engineer, you’ve probably encountered the challenge of extracting data from HTML files. HTML tables can be a valuable source of data, but extracting them can be a time-consuming process. Luckily, Python and Pandas can make this process much easier. In this article, we will explain how to extract tables from HTML files using Python and Pandas.

Table of Contents

  1. Why Extract Tables from HTML?
  2. Extracting Tables from HTML with Pandas
  3. Best Practices
  4. Conclusion

Why Extract Tables from HTML?

HTML is a markup language used to create web pages. HTML files contain various elements such as text, images, and tables. Tables are often used to display data in a structured way, making them a valuable source of information. However, extracting data from HTML tables can be difficult, especially if the table is large or contains complex formatting. That’s where Python and Pandas come in.

Python is a powerful programming language used in data science and software development. It has a variety of libraries and tools that make it easy to work with data. Pandas is one such library. It is a data manipulation library that provides easy-to-use data structures and data analysis tools. Pandas can read data from a wide range of sources, including HTML files.

Extracting Tables from HTML with Pandas

To extract tables from HTML files using Pandas, you need to follow a few simple steps:

  1. Install the necessary libraries
  2. Read the HTML file into a Pandas dataframe
  3. Extract the table from the dataframe

1. Install the Necessary Libraries

Before you can extract tables from HTML files with Pandas, you need to make sure you have the necessary libraries installed. You will need to install Pandas and Beautiful Soup.

You can install Pandas using pip:

pip install pandas

You can install Beautiful Soup using pip as well:

pip install beautifulsoup4

2. Read the HTML File into a Pandas Dataframe

Once you have installed the necessary libraries, you can read the HTML file into a Pandas dataframe. To do this, you will use the read_html() function from Pandas. This function takes an HTML file and returns a list of dataframes, one for each table in the HTML file.

Let’s consider the following html content:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Example HTML Table</title>
</head>
<body>

<h2>Sample HTML Table</h2>

<table border="1">
    <tr>
        <th>Header 1</th>
        <th>Header 2</th>
        <th>Header 3</th>
    </tr>
    <tr>
        <td>Data 1-1</td>
        <td>Data 1-2</td>
        <td>Data 1-3</td>
    </tr>
    <tr>
        <td>Data 2-1</td>
        <td>Data 2-2</td>
        <td>Data 2-3</td>
    </tr>
</table>

</body>
</html>

import pandas as pd
from bs4 import BeautifulSoup

# Read the HTML file into a Pandas dataframe
with open('example.html') as file:
    soup = BeautifulSoup(file, 'html.parser')
tables = pd.read_html(str(soup))

In this example, we first open the HTML file and parse it using Beautiful Soup. We then use the read_html() function to read the HTML file into a Pandas dataframe.

3. Extract the Table from the Dataframe

Once you have read the HTML file into a Pandas dataframe, you can extract the table you are interested in. To do this, you will need to know the index of the table in the list of dataframes returned by read_html().

# Extract the table from the dataframe
table = tables[0]
print(table)

In this example, we extract the first table from the list of dataframes returned by read_html().

Output:

   Header 1  Header 2  Header 3
0  Data 1-1  Data 1-2  Data 1-3
1  Data 2-1  Data 2-2  Data 2-3

Best Practices

Practice 1: Check HTML Structure

Before attempting to extract tables, ensure that your HTML file is well-structured. Use tools like online HTML validators to identify and fix any structural issues.

Practice 2: Handle Multiple Tables

If your HTML file contains multiple tables, iterate through the list of dataframes returned by read_html() to identify and extract the specific table you need.

# Extract all tables from the dataframe
for i, table in enumerate(tables):
    print(f"Table {i + 1}:\n{table}\n")

Practice 3: Handle Complex Formatting

For HTML files with complex formatting, consider using additional libraries like lxml as it might handle certain cases better.

Install lxml:

pip install lxml

Use lxml with read_html:

tables = pd.read_html(str(soup), flavor='lxml')

Practice 4: Error Handling

Implement robust error handling to catch potential issues during the extraction process.

try:
    tables = pd.read_html(str(soup))
    if not tables:
        raise ValueError("No tables found in the HTML file.")
except Exception as e:
    print(f"Error: {e}")

Conclusion

In conclusion, extracting tables from HTML files with Python and Pandas is a straightforward process. Once you have installed the necessary libraries, you can use the read_html() function from Pandas to read the HTML file into a dataframe, and then extract the table you are interested in. This process can save you time and effort when working with data from HTML files.

However, it is important to note that the read_html() function may not work for all HTML files. If the HTML file contains complex formatting or multiple tables, you may need to use other libraries or tools to extract the data you need. Nonetheless, Python and Pandas provide a powerful set of tools for working with data, and they are a valuable addition to any data scientist or software engineer’s toolkit.

About Saturn Cloud

Saturn Cloud is your all-in-one solution for data science & ML development, deployment, and data pipelines in the cloud. Spin up a notebook with 4TB of RAM, add a GPU, connect to a distributed cluster of workers, and more. Request a demo today to learn more.

Get a Technical Demo