Singular value decomposition

I don't know how I have not learned this until today because it is very important.
Given any real square matrix $A$, we can write

where $U$ and $V$ are orthogonal matrices and $\mathrm{\Sigma }$ is a diagonal matrix with non negative entries.
The meaning of this is easy: any linear transformation of a vector space can be obtained by a rigid transformation (rotation or reflection), followed by a scale change in the main axis direction (and different scales could be applied in every axis) and finally followed by another rigid transformation.

Important conclusion: any 2-dimensional linear transformation transform a circle into an ellipse.

It is related to the polar decomposition.
Also related: principal components analysis.

Visualization: see this web or my own web

Non square matrices

This also works for non square matrices. Consider that $A$ is the $n\times m$ matrix of a transformation $T:{\mathbb{R}}^n\to {\mathbb{R}}^m$. Then again $A=U\mathrm{\Sigma }V$ where $U$ and $V$ are square orthogonal matrices of dimension $m$ and $n$ respectively. But now, $\mathrm{\Sigma }$ is a non square diagonal matrix with non-negative entries.
The $m$ columns of $U$ represent a orthonormal basis of ${\mathbb{R}}^m$, $B_U$. And the $n$ columns of $V$ are a orthonormal basis of ${\mathbb{R}}^n$, $B_V$. The transformation $T$ can be understood like the one which sends the $i$th element of $B_V$ to the $i$th element of $B_U$ multiplied by the (non-negative) diagonal element ${\sigma }_{ii}$ of $\mathrm{\Sigma }$. This happens for every linear transformation!

It is related to matrix diagonalization. Indeed they are equal when the matrix is symmetric positive-demidefinite.