I had never really heard of this result, sometimes called the Matrix Determinant Lemma, but it came up in the process of answering a relatively simple question. Suppose I have an -dimensional jointly Gaussian vector
with covariance matrix
. The differential entropy of
is
. Suppose now I consider some rank-1 perturbation
. What choice of
maximizes the differential entropy?
On the face of it, this seems intuitively easy — diagonalize and then pick
to be the eigenvector corresponding to the smallest singular value of
. But is there an simple way to see this analytically?
Matrix Determinant Lemma. Let
be an
positive definite matrix and
and
be two
matrices. Then
.
To see this, note that
,
and take determinants on both sides.
So now applying this to our problem,
But the right side is clearly maximized by choosing corresponding to the largest singular value of
, which in this case is the smallest singular value of
. Ta-da!