From Sahih Muslim (Hadith 3244):
Abdullah b. Zubair (Allah be pleased with him) reported on the authority of his mother's sister ('A'isha) saying that Allah's Messenger (ﷺ) said:'A'isha, if your people had not been recently polytheists (and new converts to Islam), I would have demolished the Ka'ba, and would have brought it to the level of the ground and would have constructed two doors, one facing the east and the other one to the west, and would have added to it six cubits of area from Hijr, for the Quraish had reduced it when they rebuilt it
Reference: Muslim 3244