From Sahih Muslim (Hadith 3240):
A'isha (Allah be pleased with her) reported:Allah's Messenger may peace be upon him) said to me: Had your people not been unbelievers in the recent past (had they not quite recently accepted Islam), I would have demolished the Ka'ba and would have rebuilt it on the foundation (laid) by Ibrahim; for when the Quraish had built the Ka'ba, they reduced its (area), and I would also have built (a door) in the rear
Reference: Muslim 3240