Showdown at Billboard Music Awards Between Whitney Houston's Family and Ray J

Ray J was almost physically removed from the Billboard Music Awards last night after angry members of Whitney Houston's family ordered their security to get him away from them ... TMZ has learned.

Pat Houston's blood was boiling because she, Bobbi Kristina and other family members were seated next to Ray J, the person Pat believes was a very bad influence on Whitney. Minutes before the tribute to Whitney, Pat asked her security to remove Ray J from his seat, but he refused to budge.

Our sources say ... Pat's security summoned hotel security, along with several uniformed officers to physically remove Ray J.

Just before push was about to come to shove, security warned Pat that things would likely get physical and it would almost certainly be caught on video, just as Whitney was about to be honored. Sources say Pat then grudgingly recanted and Ray J kept his seat.

For his part, Ray J groused to friends, "Pat ruined what was a great opportunity to honor the woman who I loved."

11:40 AM PDT -- A rep for the Houston family tells TMZ ... Pat's assistant DID ask for Ray J to be moved from his seat ... but only after an official from Billboard told the family they needed to move to the seat that Ray J was already using.

Since Ray J had a ticket with the proper seat number, he refused to budge ... so Pat's assistant called for security to help sort out the situation.