Ray J Bails as Celeb Boxing Ref After Promoters Fail to Pay Up

Ray J understands better than anyone why Farrah Abraham's refusing to fight in a celebrity boxing match ... 'cause promoters can't keep their word in the money department.

We're told Ray J was all set to referee Farrah's fight with Nicole "Hoopz" Alexander, slated for November 10 in Atlantic City. Ray's team initially wanted $50k plus travel costs and security but the promoter, Damon Feldman, didn't have the cash, so Ray agreed to do it for $25k.

Problem is ... a few weeks ago Feldman and fight investor Samantha Goldberg came back to Ray and asked if he'd ref for less dough 'cause the event wasn't selling. Ray's manager, David Weintraub, wasn't having it ... saying no dice and pulling out of the event altogether.

Sounds like this was the beginning of the end. As we reported ... Farrah, also repped by Weintraub, pulled the plug on the event because she says the same promoter screwed her on the deal.