From Sahih Muslim (Hadith 6240):
Sa'd reported:This verse was revealed in relation to six persons and I and Ibn Mas'ud were amongst them. The polytheists said to him (the Holy Prophet): Do not keep such persons near you. It was upon this that (this verse was revealed):" Drive not away those who call upon their Lord morning and evening desiring only His pleasure" (vi)
Reference: Muslim 6240