Perhaps because it was border country, the Saxon kings kept most of it in their own hands. However, in 904, when times were more peaceful, King Edward made a generous present of all his manors surrounding Taunton to the Bishop of Winchester. Withiel was part of this gift. The Bishop administered these 16,000 acres of estate via the monastery of Taunton, and when nearly 200 years later William the Conqueror confirmed the donation, Winchester continued to own the lands, and Taunton Priory to run them. The Augustinian canons of the Priory were almost certainly responsible for the first stone church here, perhaps with some contribution from their tenants, and also for its 15th century extension, though they evidently saw no need to add an aisle or chapels. Its dedication to St Mary Magdalene is of course the same as that of the principal church of Taunton, a rectory also owned by the canons.

Because the Priory owned the tithes it was responsible for the services. A canon would have visited from time to time to serve as well as to inspect property; there may have been a house for his use and perhaps that of a humbler priest, paid to stand in as necessary. If there was no important permanent resident there would have been the less reason for a village to develop from the original Saxon hamlet on this lonely site. The Priory sub-let its holdings to lay tenants; Ranulph de Flury is recorded as holding the Nynehead Florey and Withiel manors in 1237, and the Wyke family, who inherited from the Flurys, were still the Priory’s tenants for both of these in the time of Henry VIII. However their house was at Nynehead; there is no evidence of a medieval manor house or of a lost village at Withiel, although there are traces of some vanished buildings in the field to the south of the church. There were only seven tax payers in 1332, and the richest of these was Mabilla de Gopeworthy, who looks like a farmer’s widow.

In 1539 the prior and canons of Taunton surrendered all their possessions to Henry VIII’s Royal Commissioner. It seems that the Wykes then bought both Withiel and Nynehead from the Crown, but in 1613 sold them to the Sanford family, who continued to live at Nynehead. The tenant farmers of Withiel had long included the Bryant family, and in 1641 James Bryant bought the Farm, as Withiel Farm was originally known, which his parents had previously rented. His son and grandson went on to acquire the rest of the manor. The Bryants were replaced by the Lethbridges in 1791, substantial local landowners who already owned the neighbouring Luxborough estate. The Lethbridges sold both Withiel and Luxborough in 1875 to James Insole, a successful colliery owner from Cardiff, and in turn the Insoles sold them in 1922 to the Mallets.

Through most of its long history, Withiel Florey has not consisted of much more than a handful of scattered farms and cottages, though there was once also a mill, a poor house and by 1851 a school. At the Protestation Return of 1642 (when all men of 18 and over had to swear an oath of loyalty to the Church, King and Parliament) there was a total of 22 men apart from the curate, which suggests a community of about 70. Life revolved around the six or seven farms, which normally provided almost all employment and from which came the churchwardens and the annual overseer of the poor. There was some division of religious practice between these in the later 17th century when the Lyddons of Swansey became Quakers; two William Lyddons, presumably father and son, were prosecuted and imprisoned in 1670 and 1697 for their refusal to pay tithes, the second of these being committed to Taunton jail where he died several years later. A tradition of a Quaker burial ground at Swansey persists. The Lyddons seem to have returned to the church in the next generation or so.

In the later 19th century there was a brief departure from this unchanging farming life. There had been ancient opencast iron mining in the Brendon Hills, and Sir William Lethbridge attempted to revive it in Luxborough and Withiel. His efforts, made over ten years till his death in 1849, effectively bankrupted him, but were then followed by the Brendon Hills Iron Ore Company in partnership with the Ebbw Vale Iron Works. Ore was mined on a large scale, generally unprofitably, for some 30 years. A temporary village (supplied with a Mission Church and school) grew up around the Gupworthy mine, at the head of the new West Somerset Mineral Railway, which carried the ore 11 miles to Watchet. The population of the parish nearly trebled in these years, to 300. It dropped back, as the mining ended, to its old levels. By 1901 the community and the care of its church had returned to the hands of the farmers of Withiel, Castle Hill, Swansey, Ford, Escott and Gupworthy.

There was, temporarily, a big house, described in the 1780s as “a pretty house with pleasant gardens”, the residence of Thomas Stawell Esq, husband of Amelia Bryant. The Bryants had risen from the ranks of yeomen farmers to own much of the parish and property outside it. They lived at Withiel Farm, also farmed “Parsonage”, the fields below the church, and finally built this much grander house which the 1809 OS map shows as standing to the south of Withiel Farm. Possibly this was in anticipation of the 22 year old Amelia’s marriage in 1776 to the 18 year old Thomas Stawell, heir to property in Winsford and East Anstey and with superior social status. Amelia died only eleven years later, leaving her husband with six small children. Evidently in money difficulties, he retreated with his family to Rhyll in East Anstey. His advertisement of Withiel Florey House for sale (together with the tithes) in 1790 lists 10 bedrooms, a drawing room wainscoted with “the best foreign oak”, also used for the staircase, pleasure grounds planted with shrubs, fishponds, stabling for 12 horses, kennels with a pack of harriers and as a further evidence of its gentility “a good farmhouse… at a proper distance from the above seat.” His parents in law were perhaps fortunately no longer alive to resent this distinction.

The house was pulled down in the early 19th century and no trace remains, other than a kink in the lane showing where its drive once turned in. The Lethbridges built Chargot in Luxborough for use as a shooting lodge in 1827, and the Withiel oak staircase was transferred there. The Insoles also used Chargot Lodge when in residence in Somerset. The Bryants were apparently the only owners of Withiel Florey manor ever to live in the parish, and there is little evidence of any resident clergyman either. No parsonage house appears in the censuses, which start in 1841, and although there is mention of one in a mortgage of 1781 raised by Thomas Stawell on Withiel Farm, the parsonage, glebe and tithes, this may simply be a conventional description of a benefice for sale. The incumbent at the time lived at Old Cleeve, where he was vicar. A vestry note of 1873 mentions the transfer of a meeting to the “parson’s lodgings” at Lower Escott, which seems the most likely arrangement for part-time clergy.