The castle and manor were granted in 1568 to John Astley, Queen Elizabeth’s Master of the Jewel House, though he did not live there. Around 1600, two farm houses were subsequently built in the grounds of Allington Castle while the rest of the castle gradually fell into ruin.
Most of the Great Hall and the north-east wing were destroyed in a disastrous fire in the second half of the 16th century. An early 17th-century lessee named John Best pulled down the battlements and added a half-timbered gabled second storey to the east and west wings as a replacement for the fire-damaged areas of the castle. The Bests were Catholics and used a room in the east tower as their private chapel. There is still a priest hole in the lodge of the gatehouse, a sign of the persecution that Catholics faced at the time.
Allington Castle was bought in 1720 by Sir Robert Marsham, the 2nd Baron Romney and a descendant of the Wyatts, but he did not live there and let it deteriorate. Its decaying appearance was recorded by JMW Turner in sketches and watercolours made in 1798.
The top of the Long Gallery was destroyed in another fire in the early 19th century and the rest of the castle was nearly demolished a few decades later by Charles Marsham, 5th Earl of Romney. He was dissuaded by opposition from local residents, notably the Rector of the nearby Church of St. Lawrence, but by this time the castle was totally ruined. The Penchester wing was abandoned and used as a quarry for building materials, while the ground-floor remnants of the Long Gallery were converted into a pair of farmhouses.
