A separate staff of groundskeepers makes sense to me, in part because the family and guests wouldn't have needed to interact with the gardeners much. Peter Wimsey sometimes interviewed gardeners, but that's not a home design consideration: you don't get clients by telling them they'll need a live-in gardener in case of a burglary.