Charles Oakley: I Don't Know If Dolan Is Racist ... But I've Heard Things (AUDIO)

Charles Oakley says he will NOT definitively label James Dolan as a racist because he's never personally seen him do racist things ... but says he's heard rumblings "over and over."

Oakley says he's "heard stuff" about Dolan over the years -- but when TMZ Sports asked him bluntly if he thinks Dolan is a racist Oakley said, "I don't know because I don't know him. I never met him like that."

"You hear all that stuff but I don't know. I can't say yet ... If it is, it'll come out. But right now, we're still waiting to see."

Oakley also says he's NOT accepting Dolan's proposed truce saying he's still injured -- physically and emotionally -- after the big incident at Madison Square Garden.