Unclear exactly what the guys were verbally hurling toward the QB (there appeared to be some barbs about losing to a 1-14 team) ... but Baker snapped.
He challenged the guys to come down to him ... but they playfully responded, "We’re not allowed to, you know that!"
Unfortunately for Baker, the trash talk toward him didn't end there ... 'cause hours after the Browns ultimately fell to the Bengals and fired head coach Freddie Kitchens, Dez Bryant hopped on social media to trash the QB.
"Had a stud cast around him under performed and got out performed by somebody he talked down on Danny dimes..." the ex-Dallas Cowboys superstar said.
"Big mouth and being a one read QB not going to cut it.. kitchens put the game in his players hands."
To his credit, Baker at least had one supporter in his corner after a rough weekend ... his wife, Emily Mayfield, wrote on her Instagram page, "Disappointing season… but still couldn’t be more proud of this guy, and the class he has shown through it all."