Joe Simpson -- I'M NOT GAY!!!

Rumors be damned ... a frustrated Joe Simpson is telling friends and colleagues the stories that he's switched teams are absurd.

There have been several reports out recently that Joe has told his family he's gay and that's why his marriage fell apart. There are also reports about several guys he's rumored to be dating/hooking up with.

The people we spoke with are split -- some believe him, and some think he's just not ready to come out publicly.

So we gotta ask ...