How to Check if a Pandas DataFrame Contains Only Numeric Columns

In the world of data science, Pandas is a powerful tool that allows us to manipulate and analyze data in Python. One common task is to check if a DataFrame contains only numeric columns. This blog post will guide you through the process, step by step.

Table of Contents

1. Introduction
2. Prerequisites
3. Step 1: Create a DataFrame
4. Step 2: Check Column Data Types
5. Step 3: Check if All Columns are Numeric
   • Pros
   • Cons
6. Alternative Method
   • Pros
   • Cons
7. Conclusion

Introduction

Pandas is a Python library that provides flexible data structures, designed to make working with structured data fast, easy, and expressive. It is fundamental for data manipulation and analysis in Python.

In this tutorial, we will focus on a specific task: checking if a DataFrame contains only numeric columns. This is a common requirement when preparing data for machine learning algorithms, as they often require numeric input.

Prerequisites

Before we start, make sure you have the following:

• Python 3.6 or later installed.
• Pandas library installed. You can install it using pip:

pip install pandas

Step 1: Create a DataFrame

First, let’s create a DataFrame with both numeric and non-numeric columns for demonstration purposes.

import pandas as pd

data = {
    'Name': ['John', 'Anna', 'Peter', 'Linda'],
    'Age': [28, 24, 35, 32],
    'Salary': [3000, 3200, 4500, 3800]
}

df = pd.DataFrame(data)

Step 2: Check Column Data Types

Pandas provides the dtypes attribute for DataFrame objects, which returns a Series with the data type of each column.

print(df.dtypes)

The output will be:

Name      object
Age        int64
Salary     int64
dtype: object

Step 3: Check if All Columns are Numeric

To check if all columns are numeric, we can use the apply() function with the pd.to_numeric() function, which attempts to convert a pandas object to a numeric dtype.

numeric_df = df.apply(pd.to_numeric, errors='coerce')

The errors='coerce' argument will replace all non-numeric values with NaN.

Then, we can check if there are any NaN values in the DataFrame. If there are, it means that the original DataFrame had non-numeric values.

is_all_numeric = not numeric_df.isnull().values.any()
print(is_all_numeric)

The output will be False, indicating that the DataFrame contains non-numeric columns.

Pros

• Comprehensive Handling of Non-Numeric Values: The use of apply(pd.to_numeric, errors='coerce') followed by checking for NaN values provides a comprehensive way to handle non-numeric values. It clearly indicates which columns have non-numeric data.

• Granular Control: The approach allows for fine-grained control over the conversion process using the errors parameter in pd.to_numeric(). This can be useful when dealing with specific data cleaning scenarios.

Cons

• More Code: This method involves more lines of code, potentially making it less concise and more prone to errors.

Alternative

To check if all columns are numeric, we can use an alternative method involving the select_dtypes method along with np.number. This provides a concise way to filter columns based on their data types:

import pandas as pd
import numpy as np

# Step 1: Create a DataFrame
data = {
    'Name': ['John', 'Anna', 'Peter', 'Linda'],
    'Age': [28, 24, 35, 32],
    'Salary': [3000, 3200, 4500, 3800]
}

df = pd.DataFrame(data)

# Step 2: Check if All Columns are Numeric
numeric_columns = df.select_dtypes(include=np.number).columns
is_all_numeric = len(numeric_columns) == len(df.columns)

print(is_all_numeric)

In this alternative approach, select_dtypes is used to filter columns based on their data types, and np.number is employed to specify numeric data types. The resulting numeric_columns will contain only the columns with numeric data types. The check len(numeric_columns) == len(df.columns) ensures that all columns in the DataFrame are numeric.

Pros

• Conciseness: The use of select_dtypes along with np.number is more concise, making the code easier to read and understand. It achieves the same result with fewer lines of code.

• Readability: The method reads like a natural language sentence – selecting types that are numbers. This enhances code readability, especially for those familiar with Pandas.

Cons

• Less Granular Control: The method does not provide the same level of granular control over the conversion. This might be a limitation in scenarios where specific handling of non-numeric values is required.

• Dependency on Specific Numeric Types: The method relies on np.number, which encompasses various numeric types. If a more specific numeric type check is needed, additional filtering or checks may be required.

Conclusion

In this tutorial, we’ve learned how to check if a DataFrame contains only numeric columns using Pandas. This is a crucial step in data preprocessing for machine learning algorithms, as they often require numeric input.

Remember, data science is all about understanding and manipulating your data, and Pandas provides a powerful toolset to do just that. Keep exploring, keep learning, and keep pushing the boundaries of what you can do with your data.

SHARE:

About Saturn Cloud

Saturn Cloud is your all-in-one solution for data science & ML development, deployment, and data pipelines in the cloud. Spin up a notebook with 4TB of RAM, add a GPU, connect to a distributed cluster of workers, and more. Request a demo today to learn more.