I think you make a very important point. Even if the universe was infinitely large, there might be a point where past that the conditions can't exist that would allow something like our planet to function. For infinity to grant you duplicates of yourself the possibility must exist within that "zone of infinity". Gravity for example might be stronger or weaker.