From Sahih Bukhari (Hadith 1746):
Narrated Wabra:I asked Ibn `Umar, "When should I do the Rami of the Jimar?" He replied, "When your leader does that." I asked him again the same question. He replied, "We used to wait till the sun declined and then we would do the Rami (i.e. on the 11th and 12th of Dhul-Hijja)
Reference: Bukhari 1746