The first castle in Vitré was built of wood on a feudal motte around the year 1000 on the Sainte-Croix hill. The first stone castle was built by the baron Robert I of Vitré at the end of the 11th century. The defensive site chosen, a rocky promontory, dominated the valley of the Vilaine. A Romanesque style doorway still survives from this building. During the first half of the 13th century, baron André III, rebuilt it in its present triangular form, following the contours of the rocks, surrounded with dry moats.
The castle was bought by the town in the 1820 for 8,500 francs. It was one of the first castles in France to be classified as a monument historique (historic monument).