From his forebears, from his childhood years, and from his many excursions on horseback into the valleys of Ettrick and Yarrow, Tweed and Teviot, and Liddesdale, Sir Walter was imbued with a deep love of the Border country. His tenancy of Ashiestiel was coming to a close; he therefore decided to become a Tweedside laird by purchasing in -1811 from Dr. Robert Douglas, parish minister of Galashiels, for 4000 guineas a farm on the right bank of the Tweed. It lay six miles down river from Ashiestiel and consisted of a haugh, called Newharthaugh, and a rough bank mainly covered with its native heath, a total of about 110 acres. The little farm house and steading was called Cartleyhole. Sir Walter and his family with the farmhands and domestic servants headed Tom Purdie, and all their livestock and dog flitted to their new home on May 28th, 1812. As the land had once belonged to the monks of Melrose and a ford just below the house had been used by the monks, he decided to change the name to Abbotsford. When they moved in Sir Walter already had plans for enlarging the house, but these were not carried out until 1818. The additional building connected the farm house to the steading and consisted on the ground floor of an armoury, dining-room, study and conservatory, and on the first floor of three bedrooms. There was a tower on the western corner. Before leaving Ashiestiel Sir Walter had completed The Lay of the Last Minstrel, Marmion, and The Lady of the Lake; but it was not until he moved to Abbotsford that he began the series of Waverley Novels, commencing with Waverley in 1814. From the success of his writings there grew a hunger for more land and a desire to increase the size of Abbotsford.

Early in 1822 the old farm house was entirely pulled down and in its place the present main block of Abbotsford was built. Sir Walter and his family were able to occupy the new portion of the house by the autumn of 1824. The architect responsible for the plans of Abbotsford was William Atkinson, although Sir Walter himself and his friends, Daniel Terry, Edward Blore and James Skene of Rubislaw, had many suggestions to make. George Bullock of London also gave useful advice as to the furnishings. The builders of Abbotsford were Sanderson and Paterson of Galashiels (first portion) and John and Thomas Smith of Damick (second portion, which included the gateway and court walls). John Smith was no mean sculptor as well as a master builder. By 1820 Sir Walter had extended the estate to some 1400 acres, including the lands of Mr. Usher of Toftfield. John Smith built a new cottage on Harlawburn for the Lockharts which Sir Walter called Chiefswood. One of the houses at Toftfield Sir Walter renamed Huntlyburn and installed there his great friend, Adam Ferguson, and his sisters, as tenants. Sir Walter was a great planter of trees, and he was one of the first to make his own oil gas for lighting purposes (1823).

After the death early in 1853 of his brother-in-law, Walter Lockhart Scott, Mr. Hope-Scott made many improvements to the house and grounds and to the estate. He added a west wing to the house which included a chapel, kitchen and domestic offices; and he built an entrance lodge. There was a great deal of unemployment in Galashiels at that time and he gave work to a large number of men who helped in the construction of the present terraces and of a bank which screened the house from the main road. He feued from the Scotts of Gala about half a mile of bank on the opposite side of the Tweed and planted trees along it; and he made fresh access to the house from the main road for tourists.

Many interesting people came to stay at Abbotsford as guests of Sir Walter. Amongst them were Maria Edgeworth, Wordsworth, Thomas Moore and Washington Irving. After Sir Walter's death in 1832, tourists came to Abbotsford in increasing numbers year by year. In 1883 about 1500 came, including 20 from the United States. Queen Victoria came to Abbotsford on 22nd Augusts 1867, and had tea with the family. King George V and Queen Mary, and the Duke and Duchess of York (later King George VI and Queen Elizabeth) visited Abbotsford on 12th July, 1923. The Duke and Duchess of Gloucester visited Abbotsford on 26th August,1964.