In Roman times Chavagnes (from the common Latin cabanas, 'huts') was the site of an important Roman villa.

In the thirteenth century the Benedictine monks of Luçon founded a priory dedicated to St Anthony of Egypt on the site now occupied by the College. This monastery had fallen into disuse by the time of the French Revolution.

The future Pope Clement V visited the Priory of Chavagnes shortly before being elected Pope in 1305.