The man who claimed he had a sexual relationship with ex-NFL star Kordell Stewart is admitting he made the whole thing up -- but says he isn't worried about Kordell's lawsuit.
As we previously reported, Stewart filed a lawsuit against viral video star Andrew Caldwell -- saying Caldwell lied and damaged his reputation when he went on a radio show and said they hooked up.
TMZ Sports spoke with Caldwell who says ... "It's true, I never dated Kordell. But I'm not worried about the lawsuit. If Kordell wants money, he has money. I don't owe him a dime."
What's weird is despite the "I'm not worried" demeanor -- Caldwell seems to admit he hurt Stewart's reputation.
"[What I said] did damage Kordell. He has a son. For his son to think, 'Is my dad gay?' That is damaging."
And it gets even weirder ... Caldwell says Kordell has since trashed HIM while denying the gay rumors -- and he's considering filing a lawsuit of his own.
"I may go after Kordell. Yes, I hurt his name, but he also hurt mine."
Good luck with that ...