In 1066 the English were defeated by the invading army of Duke William of Normandy . After his victory at the Battle of Hastings, William strengthened his hold on the defeated English by ordering castles to be built throughout the country. Colchester was chosen for its port and its important military position controlling the southern access to East Anglia. In 1076 work began on Colchester Castle, the first royal stone castle to be built by William in England.

The castle was built around the ruins of the colossal Temple of Claudius using the Roman temple vaults as its base, parts of which can be seen to this day. As a result the castle is the largest ever built by the Normans. It was constructed mainly of building material from Colchester's Roman ruins with some imported stone. Most of the red brick in the castle was taken from Roman buildings.

England, William's newly won possession, was soon under threat from another invader, King Cnut of Denmark. The castle had only been built to first floor level when it had to be hastily strengthened with battlements. The invasion never came and work resumed on the castle which was finally completed to three or four storeys in 1125.

The castle came under attack in 1216 when it was besieged for three months and eventually captured by King John after he broke his agreement with the rebellious nobles (Magne Carta). By 1350, however, its military importance had declined and the bulding was mainly used as a prison. By 1600 it was no longer defensible and by 1637 the hall roof had fallen in. In 1629 the castle was sold by the crown and in 1683 came into the possession of John Wheeley, a local ironmonger. Wheeley partly demolished the castle and sold the stone to local builders. Fortunately this did not prove to be profitable and he stopped before the castle was totally destroyed.

In 1726, after a number of owners, Charles Gray, member of Parliament of Colchester acquired and began to restore and alter the battered castle. He added the tiled roof, the great dome, a study, library, the arches by the well and on the first floor, and enlarged the windows on the south wall facing the High Street.

In 1860 the crypt was opened to the public as a museum and in 1920 the castle was presented to the Borough of Colchester. In 1934/35 the castle keep was roofed over, beginning the development of today's museum.

Interesting facts about Colchester

• Colchester was the first capital of Roman Britain.
• The first elephants in Britain walked through Colchester.
• The Roman town was completely destroyed and all its inhabitants massacred by the Iceni led by Boudicca.
• Matthew Hopkins, the famous Witchfinder General, interrogated "witches" in Colchester Castle Prison.