The first dimension of PCA is doing the same thing. You rotate your coordinate system (hence the convex combo) to be more friendly; PCA just happens to choose the “widest” direction in your data cloud automatically as the 1st axis.

]]>As far as I know, the Total Least Squares minimizes the euclidean distance.

You also say the PCA minimizes it.

Yet the calculation is different (Though both could be calculated using SVD).

Is it the exact solution?

If so, why bother with the TLS?

Actually you can work with non-perpendicular axes – doing so is a pain in the butt, but so long as they are linearly independent (for 2 vectors in 2D space “independent” boils down to “not parallel”, more complicated in 3D+) then the eigenvectors can be used as the basis for a coordinate system.

But perpendicular axes would help! However, Smith’s comment that “all eigenvectors are perpendicular” only applies to what are called normal matrices. Here’s a non-normal matrix which has them 45 degrees apart. But it’s {{1,2},{0,3}} and fortunately there’s no possible data for which this is a covariance matrix (can you spot why not?).

In fact, it’s even worse than Smith implies. Not all real *n x n* matrices even have real eigenvalues: for instance {{0,1},{-1,0}} has imaginary eigenvalues so doesn’t have real eigenvectors. And of those that do have real eigenvalues, not all have *n* independent eigenvectors e.g. {{1,1},{0,1}} has a repeated eigenvalue with only a one dimensional eigenspace, so its eigenvectors can’t be used as a coordinate system for the x,y plane.

So how do we make sure that there is a full set of real eigenvectors, that can form a basis for a coordinate system for our data, and which are conveniently orthogonal (i.e. mutually perpendicular) to boot? The great news is something that Smith pointed out earlier – the covariance matrix is a symmetric matrix. What Smith doesn’t note is that the eigenvectors of a symmetric matrix have all those lovely properties because symmetric matrices have real eigenvalues (as they’re a subset of the Hermitian matrices) and a full set of *n* orthogonal eigenvectors (because Hermitian matrices are in turn a subset of the normal matrices)! We already saw that {{1,2},{0,3}} has inconveniently non-perpendicular eigenvectors, but its more symmetric cousin {{1,2},{2,3}} works just fine.

I know this departs rather from the “what’s the point” tone of this blog, but this quibble apart I recommend the Smith piece – I just know “properties of matrices” is something that causes trouble for my students so can imagine this bit sowing confusion! Hopefully what I’ve managed to explain “under the hood” is why we know the eigenvectors will come out perpendicular, and conversely what the Big Deal is about the covariance matrix being symmetric.

]]>At the moment, y matters more, since the errors over y are larger than the ones over x, so the PCA is closer to y~x. ]]>

I enjoy writing in the tone of “on the couch with a beer.” I find it helps simple things seem simple and complex things seem approachable. Writing formally as if in a journal article tends to make simple things seem obfuscated and complex things impossible… at least that’s my experience. Thanks for reading.

]]>Nice post, interesting stuff and appreciate the “sitting on the couch with a beer” tone of your writing ^_^

]]>