From Sahih Bukhari (Hadith 7243):
Narrated `Aisha:I asked the Prophet (ﷺ) about the wall (outside the Ka`ba). "Is it regarded as part of the Ka`ba?" He replied, "Yes." I said, "Then why didn't the people include it in the Ka`ba?" He said, "(Because) your people ran short of money." I asked, "Then why is its gate so high?" He replied, ''Your people did so in order to admit to it whom they would and forbid whom they would. Were your people not still close to the period of ignorance, and were I not afraid that their hearts might deny my action, then surely I would include the wall in the Ka`ba and make its gate touch the ground
Reference: Bukhari 7243