From Sahih Bukhari (Hadith 1585):
Narrated `Aisha:Allah's Messenger (ﷺ) said to me, "Were your people not close to the Pre-Islamic period of ignorance, I would have demolished the Ka`ba and would have rebuilt it on its original foundations laid by Abraham (for Quraish had curtailed its building), and I would have built a back door (too)
Reference: Bukhari 1585