Well over 700 exoplanets have been detected to date. Only a handful of these have been observed directly. Direct observation is extremely challenging due to the small separation and very large contrast involved. Imaging polarimetry offers a way to decrease the contrast between the unpolarized starlight and the light that has become linearly polarized after scattering by circumstellar material. This material can be the dust and debris found in circumstellar disks, but also the atmosphere or surface of an exoplanet. We present the design, calibration approach, polarimetric performance and sample observation results of the Extreme Polarimeter, an imaging polarimeter for the study of circumstellar environments in scattered light at visible wavelengths. The polarimeter uses the beam-exchange technique, in which the two orthogonal polarization states are imaged simultaneously and a polarization modulator is swaps the polarization states of the two beams before the next image is taken. The instrument currently operates without the aid of Adaptive Optics. To reduce the effects of atmospheric seeing on the polarimetry, the images are taken at a frame rate of 35 fps, and large numbers of frames are combined to obtain the polarization images. Four successful observing runs have been performed using this instrument at the 4.2 m William Herschel Telescope on La Palma, targeting young stars with protoplanetary disks as well as evolved stars surrounded by dusty envelopes. In terms of fractional polarization, the instrument sensitivity is better than 10$^-4$. The contrast achieved between the central star and the circumstellar source is of the order 10$^-6$. We show that our calibration approach yields absolute polarization errors below 1%.