Festival Season OfferUp to 25% off on all our programmes — claim it before you enrol
← All Career InsightsData Science

How do I clean data with Pandas step by step?

To clean data with Pandas, first inspect it (head, info, describe), then handle missing values (isna, fillna, dropna), remove duplicates (drop_duplicates), fix data types (astype, to_datetime), tidy text columns (str.strip, str.lower), deal with outliers, and finally re-check the cleaned table. Most of a data scientist's preparation time goes into exactly these steps.

Why cleaning comes before every model

Real data arrives with gaps, typos, duplicate rows, numbers stored as text and dates in three formats. A model trained on that data learns the mess along with the signal, and a chart built on it can simply be wrong.

Pandas, the Python library for working with tables, has a small set of methods that fix almost all of these problems. Learning them as a repeatable workflow is one of the fastest ways to become useful on a data team.

A repeatable Pandas cleaning workflow

Run these steps in order on any new dataset, and keep the code in a notebook so the cleaning can be repeated.

  1. Load and look

    Read the file with pd.read_csv, then use df.head(), df.info() and df.describe() to see the columns, types, row count and value ranges.

  2. Count missing values

    df.isna().sum() shows the gaps per column. Decide column by column whether to drop rows, fill with a sensible value, or keep the gap as information.

  3. Remove duplicates

    df.duplicated().sum() counts repeated rows and df.drop_duplicates() removes them. Check duplicates on the key column too, such as a customer ID.

  4. Fix data types

    Convert numbers stored as text with pd.to_numeric, dates with pd.to_datetime, and repeated labels with astype("category").

  5. Tidy text columns

    Use str.strip(), str.lower() and replace() so "Hyderabad ", "hyderabad" and "HYD" become one value. Check with value_counts().

  6. Handle outliers

    Find extreme values with describe() or the interquartile range rule, then decide whether they are errors to fix or real cases to keep.

  7. Re-check and save

    Run info() and describe() again, confirm the row count makes sense, and save the clean data to a new file without overwriting the raw one.

Choosing how to fill missing values

There is no single right answer. These are the common options and when they fit.

Drop the rows

Reasonable when very few rows are missing and they are missing at random. Risky when it removes a whole group of customers.

Fill with median or mode

A simple choice for numbers (median) and categories (mode). The median is safer than the mean when there are outliers.

Fill with a flag

Fill text with "Unknown", or add a column that marks which rows were missing, so the model can learn from the gap itself.

Forward fill for time series

For readings over time, carrying the last known value forward with ffill() is often more realistic than an average.

Mistakes that quietly ruin cleaned data

Watch for these common slips:

  • Filling missing values using statistics from the whole dataset before splitting into train and test, which leaks information into the model.
  • Overwriting the raw file, so the cleaning can never be checked or repeated.
  • Dropping outliers that are real, such as genuinely large orders, and hiding the cases the business cares about most.
  • Merging two tables on a key with duplicates, which multiplies rows without any error message.
  • Cleaning by hand in a spreadsheet, so the same steps cannot be rerun when new data arrives.

Quick answers about Pandas data cleaning

Short answers to the questions learners search for most.

What is data cleaning in Pandas?

It is using Pandas methods to fix problems in a table, such as missing values, duplicates, wrong data types, inconsistent text and outliers, so the data is ready for analysis or modelling.

How do I find missing values in Pandas?

Use df.isna().sum() to count missing values in each column, or df.isna().mean() to see the share of missing values per column.

What is the difference between dropna and fillna?

dropna removes rows or columns that contain missing values. fillna keeps them and replaces the gaps with a value you choose, such as the median or "Unknown".

How do I convert a column to a date in Pandas?

Use pd.to_datetime(df["column"]). Add errors="coerce" to turn values that cannot be read into missing values instead of stopping with an error.

Is SQL or Pandas better for data cleaning?

Both are used. SQL is efficient for cleaning large tables inside a database; Pandas is more flexible for exploration and complex fixes in Python. Data scientists usually know both.

Go further with Python for data science

See where data cleaning sits in the syllabus, or read the guides that build on it.

See the Data Science programmeSee the Data Wrangling, SQL & Cleaning moduleSee the Python Programming moduleRead: learn statistics and Python step by stepRead: build a data science resume, GitHub and LinkedInRead: detect and prevent data leakageBrowse all Career Insights

Clean one messy dataset end to end

Download a public dataset with obvious problems, write every cleaning step in a notebook with a short note on why, and publish it on GitHub. A clear, repeatable cleaning notebook is a strong first portfolio project.

Train for a Data Science role

The same programme, duration and fees, with the learning path built around one job role.

Data ScientistML EngineerData AnalystBI AnalystData EngineerAnalytics Consultant

Learn Python data cleaning with a mentor

Ask our team how the Data Science programme in Madhapur, Hyderabad teaches Pandas, SQL and data wrangling.

Our admissions team will call you back within 90 minutes.
AddressHouse No. 3-535, 3rd Floor A Section, LR Towers, 100 Feet Road, Ayyappa Society, Madhapur, Hyderabad, Telangana 500081