Which problem occurs when using Target Encoding on a categorical feature with very low frequency categories?

Question

Data Science — Hard

Which problem occurs when using Target Encoding on a categorical feature with very low frequency categories?

Accepted Answer

Target encoding replaces categorical values with the mean of the target variable for that category. For categories with very few observations, the calculated mean is a poor estimate of the true relationship, leading to high variance. This causes the model to overfit to the noise present in those specific categories. Smoothing techniques are typically required to mitigate this variance by shrinking the estimate toward the global mean.