Ray J -- I Got Kicked Out of the Palms for Punching Fabolous

Ray J didn't take too kindly to Fabolous talking smack about him on Twitter, so when he saw him out in Las Vegas this weekend ... he clocked him!

Kim K's sex tape partner re-told the tale on Power 105.1 this morning, explaining, "When I seen him, I was with Floyd [Mayweather] and 50 [Cent] and he tried to say, 'Don't touch me.' And I touched that n***a!"

Ray J says he got kicked out of the Palms and detained by police, but according to cops in Vegas ... he was not arrested.