This site was probably occupied by a Roman building of sorts from the second century with a Roman bastion standing here from about AD400. This was superseded by the Wardrobe Tower in the mid to late twelfth century, when the Roman wall still formed the perimeter of the castle.
The Wardrobe had charge of the king's clothing, armour, equipment and treasure including the Crown Jewels which are said to have been placed in the tower in 1244 while work was in progress on Westminster Abbey.
In the fourteenth century the king's household developed into the Privy Wardrobe of the Tower responsible for arms and war stores. It was probably also at this time that a large stone building was constructed against the east side of the White Tower to accommodate the organisation. As more space became required another Wardrobe building was built from the tower to the Broad Arrow Tower during the reign of Henry VIII. This building was demolished during the reign of Charles II in 1879 leaving the remains of the Wardrobe Tower as they now appear.