Linear Regression

Matrix Decompositions

Matrix decomposition is the expression of a matrix as a product of matrices. Just as factorizing a number can tell us important properties of a number, so does matrix decomposition can reveal important properties.

Here are five important matrix factorizations

\(A = LU\)

This comes from elimination. Matrix L is lower triangular and U is upper triangular.

\(A = QR\)

Comes from orthogonalizing the columns \(a_1\) to \(a_n\), as in ‘Gram-Schmidt’

\(S = Q{\Lambda}Q^T\)

S is a symmetric matrix. Q has orthonormal eigen vectors as its columns. The corresponding eigen values are the digagonal of \(\Lambda\)

import numpy as np
from scipy import linalg 
import random

# create a symmetric matrix
a = np.random.randint(1, 10, size=(3,3))
s = np.tril(a, -1).T + np.tril(a)
print(f'symmetric matrix: \n\n{s}')

# eigen decomposition
w, q = linalg.eigh(s)
print(f'\neigen values - {w}')

symmetric matrix: 

[[5 6 1]
 [6 5 5]
 [1 5 3]]

eigen values - [-2.78571018  2.86068379 12.92502638]

# lets reconstruct the matrix
np.linalg.multi_dot([q, np.diag(w), np.transpose(q)])

array([[6., 5., 4.],
       [5., 3., 5.],
       [4., 5., 4.]])

# Check that the columns of the matrix are orthogonal
# select two columns randomly and check the dot prodct
col_idxs = random.sample(range(0, s.shape[0]), k=2)
assert np.isclose(np.dot(q[col_idxs[0]], q[col_idxs[1]]), 0)

# check the Euclidean norm of any of columns. It should be 1 as they are orthonormal
col_idx = random.choice(range(0, s.shape[0]))
np.linalg.norm(q[col_idx], ord=2)

0.9999999999999999