5:34 PM PT -- Brown says his issues with Coach Sarkisian hit a breaking point during a meeting with the coach on Tuesday.
Brown says he walked into Sark's office to talk about where he stood with the team (since he's been injured) and was met with hostility.
"When I walked in the room he said, 'You have the f**king guts to walk in here -- coward."
Brown continues, "The way he was talking to me ... it was like a slave master talking to his slave."
According to Brown, Sarkisian never used the N-word or any other racial slur ... but feels the overall "tone" was racist.
Eventually, Brown says, "He told me to get the f**k out of his office."
We reached out to Sarkisian and USC for comment -- so far, no word back.
1:31 PM PT -- TMZ Sports contacted Brown to find out what the hell happened ... and here's what he told us:
"I called a meeting up with this disrespectful guy!!! He called me a coward, all types of f**k words."
"He stereotype my mom and dad. like he belittle me to another level."
"I just responded Yes Sir and No Sir!"
"Never been treated like that before!! Don't Wish it upon anyone! That's who he is!"
More drama at USC ... where running back Anthony Brown just QUIT THE FOOTBALL TEAM after calling head coach Steve Sarkisian a "racist."
Brown -- a redshirt senior -- just made the announcement on Instagram ... where he said, "Sark treated me like a slave in his Office...Can't play for a racist MAN!!!!! #Fighton."
He also added an image that says, "Couldn't play for a racist man."
It's just the latest drama at the University of Southern California ... where cornerback Josh Shaw was just suspended for lying about heroically rescuing his 7-year-old nephew.
Meantime, several USC players are coming to Sarkisian's defense ... including linebacker Scott Felix who tweeted, "Coach Sark is a great coach and even better person. Accusations like that couldn't be more untrue."
Defensive tackle Antwaun Woods also chimed in ... saying, "Coach Sark is a great person/Coach! Please disregard comments that have been made by someone."