Ex-UFC Champ BJ Penn -- Slammed Head Against Bar ... Before Fight

BJ Penn slammed his head against a solid wooden bar 3 times in anger ... before the ex-UFC champ went outside and got into a fight this weekend ... bar sources tell TMZ Sports.

We're told ... BJ was a "perfect gentleman" when he arrived to the South Shore Tiki Lounge in Maui on Saturday night -- but became agitated and at one point began banging his head into the bar.

"He was definitely aggravated at something," one source tells us who also says Penn did NOT appear to be drunk.

"He only had one drink. At first he was okay, then he was hitting his head on the bar .... and 20 minutes later he was outside fighting."

As we previously reported, Penn allegedly attacked one of his friends in the parking lot -- and reportedly injured the guy so bad, he was hospitalized for a broken eye socket.