from sklearn.preprocessing import StandardScaler scaler = StandardScaler() X_scaled = scaler.fit_transform(X_train) X_test_scaled = scaler.transform(X_test) Why is it critical to use transform instead of fit_transform on the test set?

Question

Data Science — Hard

from sklearn.preprocessing import StandardScaler scaler = StandardScaler() X_scaled = scaler.fit_transform(X_train) X_test_scaled = scaler.transform(X_test) Why is it critical to use transform instead of fit_transform on the test set?

Accepted Answer

Using 'fit' on the test set calculates the mean and standard deviation based on test data, which introduces information from the test set into the training process (data leakage). The model must be evaluated on unseen data scaled using the parameters derived strictly from the training set to ensure an unbiased performance estimate.