Why cleaning comes before every model
Real data arrives with gaps, typos, duplicate rows, numbers stored as text and dates in three formats. A model trained on that data learns the mess along with the signal, and a chart built on it can simply be wrong.
Pandas, the Python library for working with tables, has a small set of methods that fix almost all of these problems. Learning them as a repeatable workflow is one of the fastest ways to become useful on a data team.
A repeatable Pandas cleaning workflow
Run these steps in order on any new dataset, and keep the code in a notebook so the cleaning can be repeated.
Load and look
Read the file with pd.read_csv, then use df.head(), df.info() and df.describe() to see the columns, types, row count and value ranges.
Count missing values
df.isna().sum() shows the gaps per column. Decide column by column whether to drop rows, fill with a sensible value, or keep the gap as information.
Remove duplicates
df.duplicated().sum() counts repeated rows and df.drop_duplicates() removes them. Check duplicates on the key column too, such as a customer ID.
Fix data types
Convert numbers stored as text with pd.to_numeric, dates with pd.to_datetime, and repeated labels with astype("category").
Tidy text columns
Use str.strip(), str.lower() and replace() so "Hyderabad ", "hyderabad" and "HYD" become one value. Check with value_counts().
Handle outliers
Find extreme values with describe() or the interquartile range rule, then decide whether they are errors to fix or real cases to keep.
Re-check and save
Run info() and describe() again, confirm the row count makes sense, and save the clean data to a new file without overwriting the raw one.
Choosing how to fill missing values
There is no single right answer. These are the common options and when they fit.
Drop the rows
Reasonable when very few rows are missing and they are missing at random. Risky when it removes a whole group of customers.
Fill with median or mode
A simple choice for numbers (median) and categories (mode). The median is safer than the mean when there are outliers.
Fill with a flag
Fill text with "Unknown", or add a column that marks which rows were missing, so the model can learn from the gap itself.
Forward fill for time series
For readings over time, carrying the last known value forward with ffill() is often more realistic than an average.
Mistakes that quietly ruin cleaned data
Watch for these common slips:
- Filling missing values using statistics from the whole dataset before splitting into train and test, which leaks information into the model.
- Overwriting the raw file, so the cleaning can never be checked or repeated.
- Dropping outliers that are real, such as genuinely large orders, and hiding the cases the business cares about most.
- Merging two tables on a key with duplicates, which multiplies rows without any error message.
- Cleaning by hand in a spreadsheet, so the same steps cannot be rerun when new data arrives.
Quick answers about Pandas data cleaning
Short answers to the questions learners search for most.
What is data cleaning in Pandas?
It is using Pandas methods to fix problems in a table, such as missing values, duplicates, wrong data types, inconsistent text and outliers, so the data is ready for analysis or modelling.
How do I find missing values in Pandas?
Use df.isna().sum() to count missing values in each column, or df.isna().mean() to see the share of missing values per column.
What is the difference between dropna and fillna?
dropna removes rows or columns that contain missing values. fillna keeps them and replaces the gaps with a value you choose, such as the median or "Unknown".
How do I convert a column to a date in Pandas?
Use pd.to_datetime(df["column"]). Add errors="coerce" to turn values that cannot be read into missing values instead of stopping with an error.
Is SQL or Pandas better for data cleaning?
Both are used. SQL is efficient for cleaning large tables inside a database; Pandas is more flexible for exploration and complex fixes in Python. Data scientists usually know both.
Go further with Python for data science
See where data cleaning sits in the syllabus, or read the guides that build on it.
Clean one messy dataset end to end
Download a public dataset with obvious problems, write every cleaning step in a notebook with a short note on why, and publish it on GitHub. A clear, repeatable cleaning notebook is a strong first portfolio project.

