From Sahih Bukhari (Hadith 1601):
Narrated Ibn `Abbas:When Allah's Messenger (ﷺ) came to Mecca, he refused to enter the Ka`ba with idols in it. He ordered (idols to be taken out). So they were taken out. The people took out the pictures of Abraham and Ishmael holding Azlams in their hands. Allah's Messenger (ﷺ) said, "May Allah curse these people. By Allah, both Abraham and Ishmael never did the game of chance with Azlams." Then he entered the Ka`ba and said Takbir at its corners but did not offer the prayer in it
Reference: Bukhari 1601