From Sahih Bukhari (Hadith 1583):
Narrated `Aisha:(the wife of the Prophet) that Allah's Messenger (ﷺ) said to her, "Do you know that when your people (Quraish) rebuilt the Ka`ba, they decreased it from its original foundation laid by Abraham?" I said, "O Allah's Messenger (ﷺ)! Why don't you rebuild it on its original foundation laid by Abraham?" He replied, "Were it not for the fact that your people are close to the Pre-Islamic Period of ignorance (i.e. they have recently become Muslims) I would have done so." The sub-narrator, `Abdullah (bin `Umar ) stated: `Aisha 'must have heard this from Allah's Messenger (ﷺ) for in my opinion Allah's Messenger (ﷺ) had not placed his hand over the two corners of the Ka`ba opposite Al-Hijr only because the Ka`ba was not rebuilt on its original foundations laid by Abraham
Reference: Bukhari 1583