Least Squares Linear Fit

It shows how to use minimal least squares method to fit data. We will generate one-dimensional regression sample set. Tree linear regression models will be constructed, their parameters will be tuned according to the sample set.

Contents

Generate a sample set

Assume a sample set reflects some ‘unknown’ law we must discover.

 x = linspace(0,10,200)'; % independent variable
 y = x + 1.5*sin(x) + randn(size(x,1),1); % dependent variable
%y = 0.5 + 0.7*x + randn(size(x,1),1); % alternative set
% dlmwrite('data4lsf.csv', [x,y]); % we can save the set in a .csv file
D = [y,x]; % usually we store dependent variable in the 1st column, and independent variable(s) behind

Load the file

If an external sample data file is given, load it.

%D = dlmread('data4lsf.csv');
y = D( :,1); % the 1st column is independent variable.
x = D( :,2:end); % the rest are independent variables
x = x(:,1); % below we consider one-dimensional regression

The first model, linear function

The 1st model is just straight line, y = a x + b

A = [x, ones(size(x))]; % linear function
w = inv(A'*A)*(A'*y); % the normal equation
y1 = w(1) .* x + w(2); % the regression
r = y-y1; % calculate residuals
y1txt = sprintf('y_1=w_1x+w_2, SSE=%0.2f', r'*r); % SSE means Sum Square Error

The second model, quadratic function

The regression model is y = a x^2 + b x + c

A = [x.^2, x, ones(size(x))]; % quadratic function
w = inv(A'*A)*(A'*y); % the normal equation
y2 = w(1) .* x.^2 + w(2).* x + w(3); % the regression
r = y-y2; % calculate residuals
y2txt = sprintf('y_2=w_1x^2+w_2x+w_3, SSE=%0.2f', r'*r);

The third model, linear combination of linear and sine

The regression model is y = a x + b sin(x) + c

A = [x, sin(x), ones(size(x))]; % linear and sine function
w = inv(A'*A)*(A'*y); % the normal equation
y3 = w(1) .* x + w(2).* sin(x) + w(3); % the regression
r = y-y3; % calculate residuals
y3txt = sprintf('y_3=w_1x+w_2sin(x)+w_3, SSE=%0.2f', r'*r);

Plot the results

h = figure;
plot(x,y,'k.'); hold on
plot(x,[y1 y2 y3]);%,'LineWidth',2);
hold off
title('Least Squares Linear Fit');
legend('sample data',y1txt,y2txt,y3txt);
xlabel('x');
ylabel('y');
% saveas(h,'img/demo_least_squares_fit.png','png');
close(h);

Note that ‘linear model’ does mean ‘streight line’. It means the model is a linear combination (weighted sum) of some functions.

http://strijov.com

% this file: demo_least_squares_fit