Charles Oakley: 'I Don't Know Why I Got Booted from Knicks Game' (VIDEO)

Charles Oakley says he has NO IDEA why he was kicked out of Madison Square Garden before things got physical with security Wednesday night ... swearing he has nothing but LOVE for the Knicks.

After Oakley was released from police custody, he sat down with a friend of his from Jimmy's NYC restaurant and calmly gave his side of the story.

Oakley also told the The NY Daily News he did NOT put his hands on Knicks owner Jimmy Dolan -- saying, "I was there for four minutes" ... I didn't say anything to him. I swear on my mother."

"They came over and wanted to know why I was sitting there. I bought the ticket. I said, 'Why do you guys keep staring at me?' Then they asked me to leave. And I said I'm not leaving."

Oakley was arrested and charged with 3 counts of misdemeanor assault and 1 count of trespassing.

Oakley's rep Akhtar Farzaie tells TMZ Sports, "Charles loves the Knicks, loves the fans, loves the city of NY."

"Oak bought his ticket to support the team and was told he was ordered to leave without reason ... anyone would be furious!"