NumPy: Transforming Data

✕

Filtering and Masking

  • Used to select / modify elements from an array based on conditions.
  • Boolean indexing creates a mask that can be applied to the array.
  • Conditions can be combined using logical operators (and &, or | ,not ~).
    • Examples:
    1. arr = np.array([1, 2, 3, 4, 5]) mask = (arr > 2) filtered_arr = arr[mask] print(filtered_arr) # Output: [3, 4, 5]
    2. mask_2 = (arr > 2) & (arr < 5) filtered_arr_2 = arr[mask_2] print(filtered_arr_2) # Output: [3, 4]
    3. mask_3 = (arr % 2 == 0) filtered_arr_3 = arr[~mask_3] print(filtered_arr_3) # Output: [1, 3, 5]
    4. arr[arr <= 2] = -1 print(arr) # Output: [-1, -1, 3, 4, 5]

where for Conditional Logic

  • Returns indices where condition is True. (filter gave value)
  • if-else can be implemented using np.where.
  • Syntax: np.where(condition, x, y)
    • Examples:
    1. arr = np.array([1, 2, 3, 4, 5]) result = np.where(arr > 3) print(result) # Output: (array([3, 4]),)
    2. cate = np.where(arr > 3, "High", "Low") print(cate) # Output: ["Low", "Low", "Low", "High", "High"]
    3. height = np.array([150, 160, 170, 180, 190]) new_height = np.where(height <= 160, height + 2, height) print(new_height) # Output: [152, 162, 170, 180, 190]

Select

  • if-elif-elif-....-else can be implemented using np.select.
  • Syntax: np.select(condlist, choicelist, default=0) - condlist: List of boolean arrays or conditions. - choicelist: List of values to return for each condition. - default: Value to return when no condition is True.
    • Examples:
    1. arr = np.array([-1, 2, 3, 4, 5]) conditions = [(arr > 0) & (arr < 2), (arr >= 2) & (arr < 4), arr >= 4] choices = ["Low", "Medium", "High"] result = np.select(conditions, choices, "Invalid") print(result) # Output: ["Invalid", "Medium", "Medium", "High", "High"]

Reshape, Resize, Flatten, Ravel, Transpose

  • Changes shape (dimension) of an array without changing data.
    • Reshape
    1. Returns a new array with the specified shape without modifying the original array.
    2. New shape must be compatible with original shape (same number of elements).
    3. Example: arr = np.array([[1, 2], [3, 4], [5, 6]])reshaped_arr = np.reshape(arr, (2, 3))print(reshaped_arr) # Output: [[1, 2, 3], [4, 5, 6]] print(arr) # Original array remains unchanged: [[1, 2], [3, 4], [5, 6]] print(np.reshape(arr, (4, 2))) # Raises error print(np.reshape(arr, (-1,2))) # Automatically infers. Output: [[1, 2], [3, 4], [5, 6]] print(np.reshape(arr, (-1,))) # Flattens to 1D. Output: [1, 2, 3, 4, 5, 6]`
    Resize
    1. Returns a new array with the specified shape, padding or truncating as needed.
    2. Example: arr = np.array([[1, 2], [3, 4], [5, 6]])resized_arr = np.resize(arr, (2, 4))print(resized_arr) # Output: [[1, 2, 3, 4], [5, 6, 1, 2]] resized_arr = np.resize(arr, (2, 3))print(resized_arr) # Output: [[1, 2, 3], [4, 5, 6]] resized_arr = np.resize(arr, (2, 2))print(resized_arr) # Output: [[1, 2], [3, 4]]
    Flatten
    1. Returns a copy of the array collapsed into one dimension.
    2. Example: arr = np.array([[1, 2], [3, 4], [5, 6]])flattened_arr = arr.flatten()print(flattened_arr) # Output: [1, 2, 3, 4, 5, 6] print(arr) # Original array unchanged: [[1, 2], [3, 4], [5, 6]]
    Ravel
    1. Returns a flattened view of the array.
    2. Example: arr = np.array([[1, 2], [3, 4], [5, 6]])raveled_arr = np.ravel(arr)print(raveled_arr) # Output: [1, 2, 3, 4, 5, 6] raveled_arr[0] = 99print(arr) # Original array modified: [[99, 2], [3, 4], [5, 6]]
    Transpose
    1. Swaps rows and columns of the array.
    2. Example: arr = np.array([[1, 2], [3, 4], [5, 6]])transposed_arr = np.transpose(arr) # or arr.Tprint(transposed_arr) # Output: [[1, 3, 5], [2, 4, 6]] print(arr) # Original array unchanged: [[1, 2], [3, 4], [5, 6]]

Merging Arrays: Concatenate, hstack, vstack

  • Used to combine multiple arrays into one.
    • Concatenate
    1. Combine multiple numpy arrays along rows or columns as specified in axis.
    2. Example: arr1 = np.array([[1, 2], [3, 4]])arr2 = np.array([[5, 6], [7, 8]])concatenated_arr = np.concatenate((arr1, arr2), axis=0)print(concatenated_arr) # Output: [[1, 2], [3, 4], [5, 6], [7, 8]] concatenated_arr = np.concatenate((arr1, arr2), axis=1)print(concatenated_arr) # Output: [[1, 2, 5, 6], [3, 4, 7, 8]]
    hstack
    1. Stacks arrays in sequence horizontally (column-wise).
    2. Example: arr1 = np.array([[1, 2], [3, 4]])arr2 = np.array([[5, 6], [7, 8]])hstacked_arr = np.hstack((arr1, arr2))print(hstacked_arr) # Output: [[1, 2, 5, 6], [3, 4, 7, 8]]
    vstack
    1. Stacks arrays in sequence vertically (row-wise).
    2. Example: arr1 = np.array([[1, 2], [3, 4]])arr2 = np.array([[5, 6], [7, 8]])vstacked_arr = np.vstack((arr1, arr2))print(vstacked_arr) # Output: [[1, 2], [3, 4], [5, 6], [7, 8]]

Copying Arrays

  • Assigning one array to another variable creates a reference, not a copy.
  • If you modify the new variable, it will affect the original array.
  • To create an actual copy, use np.copy() or the copy method.
    • Examples:
    1. arr = np.array([1, 2, 3]) arr_ref = arr arr_ref[0] = 99 print(arr) # Output: [99, 2, 3] (original array modified)
    2. arr_copy = np.copy(arr) # or arr.copy() arr_copy[0] = 1 print(arr) # Output: [99, 2, 3] (original array unchanged) print(arr_copy) # Output: [1, 2, 3] (copy modified)

Linear Algebra Operations

  • NumPy provides functions for performing linear algebra operations on arrays.
Common Linear Algebra Operations in NumPy:
OperationExample
Matrix Multiplication (dot product)np.dot(a2, a2) / a2 @ a2
Determinantnp.linalg.det(a2)
Inversenp.linalg.inv(a2)
Solve Linear Equationsnp.linalg.solve(a2, b)
Examples of linear algebra operations in NumPy

Practice QuestionsNot started

  1. Data Cleaning

    Question 1 of 4

    • Write a function clean_outlier that takes 1D array, replaces outliers with median & return clean data. (Outliers: data below Q1 - 1.5 * IQR or above Q3 + 1.5 * IQR).
    • Write a function clean_outlier_std that implements similar task but different outlier definition. (data below mean - 2 * std or above mean + 2 * std).
    • You have sensor recording data over time. It records invalid value as -1. Write a function clean_invalid that takes 1D array, replace -1 with surrounding mean & return clean data.
    • Write function fix_invalid_data that takes 2D NumPy array. Array contains invalid value represented as -999. Function has to substitute invalid value with mean of data. Optionally function should have axis argument. If axis=0, replacement has to be done by column mean. If axis=1, replace by row mean else overall mean. [Hint: np.apply_along_axis(fn, axis, data)]

  2. Conditional (where, select)

    Question 2 of 4

    • Write a function incrment_salary that takes salary_array and returns increased salary. Hike Rule: <1000 => 20%, 1000-5000 => 10%, >5000 => 5%. (use: np.select)
    • Write a function categorize_age that takes age_array and categorizes it as Child, Teen, Adult, Senior depending on age range. (use: np.select)
    • Assign temp_data as [15, 22, 30, 5, 18, 25]. Generate new array temp_category that has elements below_avg and above_avg. (use: np.where)
    • Assign player_name as ["Player1", "Player2", "Player3", "Player4"], player_score as [85, 92, 78, 90]. Find the name of players with score above 90 and store as top_players.

  3. Transforming Arrays

    Question 3 of 4

    • Create 4x4 array with values randomly from 0-255. Reshape it to 3x3 with np.resize.
    • Create 6x6 array with values randomly from 0-255. Reshape it to 4x9 with np.shape.
    • Create 4x8 array from normal distribution (u=40, sd=10). Transpose this and print.
    • Create 3x3 array with random value from 0-255. Convert it to 4x4 such that excess element to left and top are filled with zero.
    • Create 4x4 array with random values from 0-20. Convert it into 1D using reshape, resize, flatten and ravel.
    • Practice the questions from Hacker Rank.

  4. Merging Arrays

    Question 4 of 4

    • Create 4x3 array with random values from 0-60. Another 2x3 array with random values from 10-80. Merge them vertically using concat and vstack.
    • Create 3x4 array with random values from 60-80. Another 3x2 array with random values from 20-90. Merge them horizontally using concat and hstack.
    • Ram and Shyam are doing survey on KTM & POKHARA city for pollution. CO2 & Temp reading recorded by them are [[.5, 22], [.3, 20], [.7, 25]], [[.3, 20], [.2, 23], [.24, 21]]. - What is max & min value of CO2 and Temperature? (considering both cities) - Display CO2 & Temp readings for day - 2? (both city)
    • Store brnd_sz_ar as [['Adidas', 'S'], ['Nike', 'L'], ['Adidas', 'M'], ['Nike', 'S']. Store prc_wrnty_ar as [[150, 1], [120, 0.6], [200, 1.2], [180, 0.9]]. Create array tshirt_info by combining feature from these.