Context. To directly image rocky exoplanets in reflected (polarized) light, future space- and ground-based high-contrast imagers and telescopes aim to reach extreme contrasts at close separations from the star. However, the achievable contrast will be limited by reflection-induced polarization aberrations. While polarization aberrations can be modeled with numerical codes, these computations provide little insight into the full range of effects, their origin and characteristics, and possible ways to mitigate them. Aims: We aim to understand polarization aberrations produced by reflection off flat metallic mirrors at the fundamental level. Methods: We used polarization ray tracing to numerically compute polarization aberrations and interpret the results in terms of the polarization-dependent spatial and angular Goos-Hänchen and Imbert-Federov shifts of the beam of light as described with closed-form mathematical expressions in the physics literature. Results: We find that all four beam shifts are fully reproduced by polarization ray tracing. We study the origin and characteristics of the shifts as well as the dependence of their size and direction on the beam intensity profile, incident polarization state, angle of incidence, mirror material, and wavelength. Of the four beam shifts, only the spatial Goos-Hänchen and Imbert-Federov shifts are relevant for high-contrast imagers and telescopes because these shifts are visible in the focal plane and create a polarization structure in the point-spread function that reduces the performance of coronagraphs and the polarimetric speckle suppression close to the star. Conclusions: Our study provides a fundamental understanding of the polarization aberrations resulting from reflection off flat metallic mirrors in terms of beam shifts and lays out the analytical and numerical tools to describe these shifts. The beam shifts in an optical system can be mitigated by keeping the f-numbers large and angles of incidence small. Most importantly, mirror coatings should not be optimized for maximum reflectivity, but should be designed to have a retardance close to 180°. The insights from our study can be applied to improve the performance of SPHERE-ZIMPOL at the VLT and future telescopes and instruments such as the Roman Space Telescope, the Habitable Worlds Observatory, GMagAO-X at the GMT, PSI at the TMT, and PCS (or EPICS) at the ELT.