Why is it racist to throw a banana at a brazilian?

I remember when someone threw one at neymar last year for brazil and just saw it now, but roberto carlos was the victim .... He said he would never play again ? Is it cause brazil have yellow shirts ?