In this, as in other cases of discrimination, it’s probably best to let those affected speak for themselves.
Yep. If a person is in a group that's marginalized and oppressed, and if that group tells you that a word is derogatory, hurtful, stigmatizing, and contributes to a culture that identifies this group as less than human, then thoughtful individuals should pay attention to what that group says. It doesn't matter if someone in the privileged group is like, "Well, I didn't mean it that way!" Doesn't matter what you meant. What matters is the effect the word has, and the people who can tell you about those effects are folks affected by it. Once you've been told, then your job is to change the way you speak.
As my friend George Estreich notes, "Any word can be repurposed for contempt." It's not like eradicating the word "retarded" from our casual speech is going to eradicate all hateful speech aimed at people with intellectual disabilities for all time. But it's a step in the right direction. And when thoughtful people do change their speech, they have the opportunity to see how that changes their thinking. The challenge of saying "a person with an intellectual disability" might help you recognize in some small way that you're talking about a person who happens to have an intellectual disability.
And, hey, the challenge of coming up with an insult that doesn't harm people with intellectual disabilities will force you to be more precise in your insulting language, and that can lead to creativity. You can always use the Kelly Piepmeier classic, "You're about as innocent as a toadstool!"