"Hate him now," Portnoy says ... "when he's on the Patriots, I love him. When he's not on the Patriots, I hate him."
But, Portnoy admits he's not really trying to judge AB's character -- he's a fan of anyone who can help his team win. And, it seems obvious Dave thinks Brown can help (let's face it, the guy's a monster on the field).
Brown -- who recently filed a grievance against the Patriots and Raiders -- went on social media last week and suggested the Pats give him a second chance since he believes New England is gonna be on the hook for paying him roughly $10 mil in salary anyway.
Dave ain't opposed to the idea ... but says if Pats management makes it clear they want no part of AB, Portnoy will "fall in line" and swear off the WR too.
Don't miss the part where our photog -- a NY Giants fan -- tries to talk smack to Portnoy and gets owned right there on the street.