I don't have source handy, but I remember reading that the middle and upper classes had sub-replacement fertility in the late Roman Empire. You need to investigate differences within societies too. In pre-modern times there was a much more significant break, both culturally and militarily, between the upper (and middle where they existed) and the lower classes (which were often effectively slaves, even if not called that).