In England, at least, with grand homes the gardeners would not live in the main mansion--the only servants who did were the ones who worked directly in the building. Chauffers, gardeners, all the other "outdoors" staff would have their own cottages somewhere inconspicuous.