Orthogonality

Orthogonality is a critical concept related to inner products and vector geometry in general. Two vectors are orthogonal when their inner product is 0 $$ y^Tx = 0 $$ From the geometric definition of inner products, this implies that the angle in between them is $ \theta = \pm \tfrac{\pi}{2} $ (since $ \cos( \theta )=0 $ ), ie. that the vectors are perpendicular to each other. Intuitivley, orthogonal vectors represent entirely separate directions that do not affect each other. In the context of probability distributions, orthogonality indicates that two random vectors are uncorrelated with each other.

Subspace Definitions

For a vector $x \in \mathbb{R}^n$, the set of vectors orthogonal to $x$ has dimension $n-1$. Subspaces are often defined as being orthogonal to some vector or set of vectors (see the discussion of nullspaces).

Perpendicular vectors in $\mathbb{R}^2$ and $\mathbb{R}^3$ are pictured below.