During the early 12th century the original wooden motte and bailey structure was replaced by stone defences and a keep was constructed. This was built in 2 stages and is thought to have been the work of King Stephen. The first stage was really nothing more than a shell that was later strengthened and heightened. Once completed, the keep was used as a private apartment for the ruling monarch. During the second construction stage the curtain walls were crenellated.
Later during the 12th century, the keep became the sheriff’s headquarters and the county jail for the counties of Surrey and Sussex. By this time King Henry II had built private living accommodation in the bailey.
King Henry III made several lavish improvements to the castle in the 13th century and it began to be thought of as a palace rather than a castle. The Great Hall suffered fire damage in 1254 but was soon restored. Henry also bought additional land in order to extend the bailey to construct rooms for his son, Edward.
King Henry died in 1272 and the castle and palace buildings fell into disrepair although the great tower continued to be used as the county jail.
By 1379 virtually the entire castle had fallen down with the exception of the keep and the great chamber.
The county jail was moved to Southwark during the 16th century and John Daborne became the guardian of the castle in 1544. During the remainder of the 16th century, the Daborne family added brick windows and fireplaces to the property.
King James I sold the castle and surrounding estates to Francis Carter in 1611. He attempted to convert the keep into a private residence but this proved unsuccessful and the roof and floors of the keep were removed in 1630. Francis Carter then rented out sections of the castle grounds to farmers. Lord Grantley of Wonersh later became the owner of the castle and sold it to Guildford Borough Council in 1885. The Corporation restored the keep and castle walls and opened up the property as public gardens in 1888.