Kordell Stewart Accuser: 'It's True, I Never Dated Kordell'

The man who claimed he had a sexual relationship with ex-NFL star Kordell Stewart is admitting he made the whole thing up -- but says he isn't worried about Kordell's lawsuit.

As we previously reported, Stewart filed a lawsuit against viral video star Andrew Caldwell -- saying Caldwell lied and damaged his reputation when he went on a radio show and said they hooked up.

TMZ Sports spoke with Caldwell who says ... "It's true, I never dated Kordell. But I'm not worried about the lawsuit. If Kordell wants money, he has money. I don't owe him a dime."

What's weird is despite the "I'm not worried" demeanor -- Caldwell seems to admit he hurt Stewart's reputation.

"[What I said] did damage Kordell. He has a son. For his son to think, 'Is my dad gay?' That is damaging."

And it gets even weirder ... Caldwell says Kordell has since trashed HIM while denying the gay rumors -- and he's considering filing a lawsuit of his own.

Seriously.

"I may go after Kordell. Yes, I hurt his name, but he also hurt mine."