rootmain: Overfitting a noisy dataset with a higher order hypothesis

The following example illustrates a scenario where the usage of a complex hypothesis causes overfitting.

The initial target function (blue dotted line) is a 2nd order polynomial with some noise added to it (red points in the plot below).

For the hypothesis, 2 approaches are tested.

With a 15th order polynomial (green line) we get very low in sample error, however the out-of-sample error rises because the learning algorithm was more focussed on catching up with the noisy data, than learning the general trend.

The 2nd order polynomial (magenta line) on the other hand has a higher in-sample error, however performs better in terms of the out-of-sample error

import numpy as np
from matplotlib import pyplot as plt
from sklearn.linear_model import LinearRegression

x = np.arange(0,10,.5)
y = 3*x**2 + 2*x
plt_target = plt.plot(x,y, '--')

noise = (np.random.rand(np.size(x))-np.ones(np.size(x))*.5)*150
ynoisy = y+noise
plt_noisy_pts = plt.scatter(x,ynoisy, color='r')

reg = LinearRegression()
X = np.vstack((x, x**2, x**3, x**4, x**5, x**6, x**7, x**8, x**9, x**10, x**11, x**12, x**13, x**14, x**15)).T
reg.fit(X, ynoisy)
res = reg.predict(X)
plt_hyp_10deg = plt.plot(x,res, color='g')

reg = LinearRegression()
X = np.vstack((x, x**2)).T
reg.fit(X, ynoisy)
res = reg.predict(X)
plt_hyp_2deg = plt.plot(x,res, color='magenta')


plt.legend( (plt_target[0], plt_noisy_pts, plt_hyp_10deg[0], plt_hyp_2deg[0]), 
            ("Target Function", 
             "Noisy Points in the dataset",
             "15th order polynomial",
             "2nd order polynomial"))

plt.show()

Output

References:
Prof. Yaser's lecture #11 on overfitting : http://www.youtube.com/watch?v=EQWr3GGCdzw

May 20, 2013

Overfitting a noisy dataset with a higher order hypothesis