DataFrame Operations

2. Selecting and Indexing Data

2.1 🎯 Selecting Columns
1. We can select a single column using: df["age"]Example: age_series = df["age"] This returns a Series.
2.2 Selecting Columns with Dot Syntax
1. We can also write: df.age But this only works when the column name is simple and does not contain spaces or special characters. Recommended style: df["age"] 📌 df["column"] is safer than df.column.
2.3 Selecting Multiple Columns
1. To select multiple columns, we pass a list of column names. sample_df = df[["name", "age"]] 📌 Single column ➜ One pair of brackets. 📌 Multiple columns ➜ List inside brackets.

3. loc and iloc

3.1 📍 What is loc?
1. loc selects data using labels. Syntax: df.loc[row_label, column_label]Example: age_loc = df.loc[:, "age"] This selects all rows and the age column.
3.2 🔢 What is iloc?
1. iloc selects data using integer position. Syntax: df.iloc[row_position, column_position]Example: age_iloc = df.iloc[:, 2] This selects all rows and the column at position 2.

3.3 Selecting Columns Examples
1. age_series = df["age"] # Select age column age_loc = df.loc[:, "age"] # Select age column using label age_iloc = df.iloc[:, 2] # Select column at index position 2
3.4 Selecting Multiple Columns Examples
1. sample_df = df[["name", "age"]]sample_loc = df.loc[:, ["name", "age"]]sample_iloc = df.iloc[:, [0, 2]]
3.5 Selecting Rows and Columns Together
1. age_loc_samp = df.loc[0:5, "age"]age_iloc_samp = df.iloc[0:5, 2]age_samp = df["age"][0:5]sample_data = df.loc[0:5, ["name", "age"]] 📌 loc uses labels and includes the ending label in slicing. 📌 iloc uses positions and excludes the ending position.

4. Filtering Data

4.1 🔎 What is Filtering?
1. Filtering means selecting rows based on a condition. Example: Select people whose age is greater than 30.
4.2 Basic Filtering
1. age_filter = df["age"] > 30filtered_df = df[age_filter] Shorter version: filtered_df = df[df["age"] > 30] 📌 Simple idea: First create a condition. Then use it to filter the DataFrame.
4.3 Filtering with Multiple Conditions
1. Sometimes we need to use multiple conditions to get the final result. For this, we combine individual conditions using: & for AND | for OR ~ for NOT Example: data_filter = (df["age"] > 30) & (df["weight"] < 70)filtered_df = df[data_filter] 📌 Always wrap each condition in parentheses.
4.4 Filtering Text Columns
1. name_filter = df["name"].str.startswith("A")filtered_df = df[name_filter] This selects rows where name starts with A.
4.5 Filtering with isin()
1. country_filter = df["country"].isin(["USA", "Canada"])filtered_df = df[country_filter] To exclude those countries: filtered_df = df[~country_filter] 📌 isin() checks if values are inside a list.

5. Broadcasting and Vectorized Operations

5.1 ⚡ What are Vectorized Operations?
1. Pandas operations are usually vectorized. This means operations are applied to the whole column at once instead of using loops. Example: df["age_plus_one"] = df["age"] + 1 This adds 1 to every value in the age column. 📌 Simple idea: Pandas works column-wise, so we can avoid writing loops.
5.2 Broadcasting Example
1. Broadcasting means assigning one value to many rows. df["course"] = "Python" This adds a new column where every row has the value "Python".
5.3 Vectorized Operation Examples
1. df["age_plus_one"] = df["age"] + 1df["bmi"] = df["weight"] / (df["height"] ** 2)df["is_above_30"] = df["age"] > 30

5.4 Adding Random Values
1. import numpy as nprandom_array = np.random.rand(len(df))df["random_value"] = random_array 📌 len(df) gives the number of rows in the DataFrame.

6. Sorting and Ranking

6.1 🔃 Sorting Data
1. We can sort a DataFrame using: sort_values()
6.2 Sort by One Column
1. sorted_df = df.sort_values(by="age", ascending=False) This sorts rows by age from highest to lowest.
6.3 Sort by Multiple Columns
1. sorted_df = df.sort_values(by=["age", "weight"],ascending=[True, False] ) This sorts by: ✅ age in ascending order ✅ weight in descending order
6.4 Ranking
1. We can create ranks using rank(). ranked_df = df.copy()ranked_df["age_rank"] = ranked_df["age"].rank(method="min") 📌 Simple idea: Sorting changes row order. Ranking gives each value a rank number.

7. Drop and Rename

7.1 🗑️ Dropping Columns
1. We can drop columns using drop(). df_dropped = df.drop(columns=["age", "weight"]) Another way: df_dropped = df.drop(["age", "weight"], axis=1) Here, axis=1 means column.
7.2 Dropping Rows
1. We can also drop rows using index labels. df_dropped = df.drop([0, 1]) This drops rows with index labels 0 and 1.
7.3 Renaming Columns
1. We can rename columns using rename(). df_renamed = df.rename(columns={"age": "Age","weight": "Weight"} ) 📌 Simple idea: Use drop() to remove rows or columns. Use rename() to change column names.

8. Concatenating DataFrames

8.1 🧱 What is pd.concat()?
1. pd.concat() is used to stack DataFrames together. It is similar to adding rows from multiple tables together. Example use case: We have daily sales files and want to combine them into one DataFrame.
8.2 Example: Combine Multiple CSV Files
1. import pandas as pddf1 = pd.read_csv("2025-01-02.csv") df2 = pd.read_csv("2025-01-03.csv") df3 = pd.read_csv("2025-01-04.csv")combined_df = pd.concat([df1, df2, df3])print(df1.shape) print(df2.shape) print(df3.shape) print(combined_df.shape)

8.3 Ignore Old Index
1. After concatenation, old indexes may repeat. We can fix that using: combined_df = pd.concat([df1, df2, df3], ignore_index=True) 📌 Simple idea: concat() stacks DataFrames. ignore_index=True creates a fresh index.

9. Merging DataFrames

9.1 🔗 What is pd.merge()?
1. pd.merge() combines DataFrames based on common columns. It is similar to SQL JOIN. Syntax: pd.merge(left_df, right_df, on="key_col", how="left")

9.3 Merge Example
1. import pandas as pdleft_df = pd.DataFrame({"customer_id": [1, 2],"name": ["Hari", "Bob"] })right_df = pd.DataFrame({"id": [2, 3],"age": [25, 30] })merged_df = pd.merge(left_df,right_df,left_on="customer_id",right_on="id",how="left" )print(merged_df)Output: customer_id name id age 1 Hari NaN NaN 2 Bob 2.0 25.0 📌 merge() joins DataFrames using matching key columns. 📌 If both DataFrames have the same key column name, we can use on instead of left_on and right_on.

10. Data Type Conversion

10.1 🔄 Why Convert Data Types?
1. Sometimes data is loaded with the wrong data type. Example: - Age may load as text - Date may load as text - Category may load as normal string We can convert data types using: astype() or specialized functions like: pd.to_datetime()
10.2 Common Type Conversions
1. df["age"] = df["age"].astype(int)df["weight"] = df["weight"].astype(float)df["name"] = df["name"].astype(str)df["category_col"] = df["category_col"].astype("category")df["date_col"] = pd.to_datetime(df["date_col"]) 📌 Simple idea: Correct data types make analysis easier and safer.

11. GroupBy and Aggregation

11.1 📊 What is GroupBy?
1. groupby() is used to group data and calculate summary values. Example: Find total sales for each category.
11.2 Common Aggregation Functions
1. Common functions include: 👉 sum 👉 mean 👉 count 👉 min 👉 max 👉 std 👉 median 👉 quantile 👉 idxmin 👉 idxmax 👉 unique 👉 nunique 👉 cumsum 👉 prod 👉 mode 👉 var
11.3 Basic Aggregation
1. total_sales = df["sales"].sum()print(total_sales)
11.4 GroupBy Example
1. sales_summary = df.groupby(["category", "product"],as_index=False ).agg(total_sales=("sales", "sum"),avg_sales=("sales", "mean") )print(sales_summary) 📌 Simple idea: groupby() creates groups. agg() calculates summary values for each group.

12. Time Series Operations

12.1 🕒 What is Time Series Data?
1. Time series data is data that changes over time. Examples: - Daily sales - Monthly revenue - Hourly temperature - Stock prices Pandas has powerful tools for working with time-based data.
12.2 Convert Date Column to Datetime
1. Before doing time series operations, convert the date column to datetime. df["date"] = pd.to_datetime(df["date"])
12.3 Set Date as Index
1. For resampling, the index usually needs to be datetime. df.set_index("date", inplace=True)
12.4 Resampling
1. Resampling means changing the frequency of time-based data.

12.5 Monthly Sales Example
1. monthly_sales = df.resample("ME").agg({"sales": "sum" })
12.6 Rolling Average
1. Rolling average calculates the average over a moving window. df["rolling_avg"] = df["sales"].rolling(window=7).mean() 📌 Simple idea: Resampling summarizes data by time period. Rolling calculates moving values.

13. String and Datetime Functions

13.1 🔤 String Functions with .str
1. String functions are applied to text columns using .str. df["name"] = df["name"].str.upper()df["name"] = df["name"].str.strip()df["name"] = df["name"].str.replace("old", "new")
13.2 Datetime Functions with .dt
1. Datetime functions are applied to datetime columns using .dt. df["year"] = df["date"].dt.yeardf["day"] = df["date"].dt.daydf["day_of_week"] = df["date"].dt.day_name() 📌 Simple idea: Use .str for text columns. Use .dt for datetime columns.