An Empirical Analysis of Overfitting Mitigation Techniques Using Regularized Linear Regression: A Case Study on Noise Data Modeling

Abstract

This study addresses the regularization optimization problem in linear regression models by proposing a simplified implementation of Ridge regression based on the gradient descent method. A synthetic dataset with a linear relationship (y = 5x + 5 + ε) is generated to systematically compare the differences between ordinary linear regression and the L2-regularized model in terms of training efficiency and generalization performance. The experiments employ the normal equation method to solve the standard linear regression parameters and apply a custom gradient descent algorithm to implement regularized regression with a weight decay factor. The results show that when the regularization hyperparameter λ = 0.1, the Ridge model achieves the lowest mean squared error (MSE = 4.31) on the test set, improving prediction accuracy by approximately 7.3% compared to ordinary linear regression (MSE = 4.65). In terms of parameter estimation, the regularized model’s intercept (5.12) and slope coefficient (4.97) are closer to the true parameter values, reducing the error by 21% compared to the baseline model. Visual analysis further confirms that regularization effectively mitigates parameter overfitting. This study provides a reproducible experimental framework for understanding the regularization mechanism, and the proposed methodology can be extended to higher-order polynomial regression scenarios.