How can I remove outliers using positional indices from np.where without causing a KeyError in pandas?

The error occurs because df2 has a non-default index. When you use np.where on df2, it returns positional indices (i.e., 0, 1, 2, … for the rows in df2) rather than the actual index labels in df2. When you then pass these positional indices directly to df2.drop(index=…), pandas looks for those values as index labels and fails if they don’t exist.

To fix this, select the actual index labels using df2.index. For example, change:

PYTHONCopied
df2.drop(index=upper_array, inplace=True)

to:

PYTHONCopied
df2.drop(index=df2.index[upper_array], inplace=True)

and do the same for lower_array:

PYTHONCopied
df2.drop(index=df2.index[lower_array], inplace=True)

Below is your modified function:

PYTHONCopied
import pandas as pd 
import numpy as np

# Load the dataset
df = pd.read_csv('.xxx.csv')

# Function to remove outliers using IQR
def remove_outlier_using_IQR(df: pd.DataFrame, name_column: str, value: int) -> pd.DataFrame:
    """
    Remove outliers using IQR in the 'name_column'.
    
    Args:
        df (pd.DataFrame): The DataFrame containing the columns for outlier removal.
        name_column (str): The name of the column containing outliers to be removed.
        Value: Value in name column for outlier removal.
        
    Returns:
        pd.DataFrame: The DataFrame with outliers in 'name_column' removed.
    """
    # Filter the subset for the specified value
    df2 = df[df[name_column] == value]
    
    # Compute quartiles and IQR on 'final_test'
    Q1 = df2['final_test'].quantile(0.25)
    Q3 = df2['final_test'].quantile(0.75)
    IQR = Q3 - Q1
    lower = Q1 - 1.5 * IQR
    upper = Q3 + 1.5 * IQR

    # Debug prints
    df2.info()
    print("Shape:", df2.shape)
    print("Q3:", Q3)
    print("Upper threshold:", upper)
    
    # Identify outlier positions
    upper_array = np.where(df2['final_test'] >= upper)[0]
    lower_array = np.where(df2['final_test'] <= lower)[0]
    
    print("Outlier indices (relative positions):", upper_array)
    
    # Drop outliers using actual index labels
    df2.drop(index=df2.index[upper_array], inplace=True)
    df2.drop(index=df2.index[lower_array], inplace=True)
    
    # Combine with the rest of the data that was not filtered by 'name_column'
    df3 = df[df[name_column] != value]
    df_merged = pd.concat([df2, df3], ignore_index=False, sort=False)
    
    return df_merged

# Use the function to remove outliers for x values 0 through 4
df = remove_outlier_using_IQR(df=df, name_column='hours_per_week', value=0)
df = remove_outlier_using_IQR(df=df, name_column='hours_per_week', value=1)
df = remove_outlier_using_IQR(df=df, name_column='hours_per_week', value=2)
df = remove_outlier_using_IQR(df=df, name_column='hours_per_week', value=3)
df = remove_outlier_using_IQR(df=df, name_column='hours_per_week', value=4)

By indexing into df2.index with your upper_array and lower_array, you ensure that the correct rows are dropped by label rather than by relative position.