From Sahih Bukhari (Hadith 3830):
Narrated `Amr bin Dinar and 'Ubaidullah bin Abi Yazid:In the lifetime of the Prophet (ﷺ) there was no wall around the Ka`ba and the people used to pray around the Ka`ba till `Umar became the Caliph and he built the wall around it. 'Ubaidullah further said, "Its wall was low, so Ibn Az-Zubair built it
Reference: Bukhari 3830