Olympic Champ Tyler Clary -- Declines White House Invite ... I'd Rather Race Cars

Olympic gold medalist Tyler Clary -- the guy who smashed Ryan Lochte's backstroke record -- turned down an invite to chill with President Obama at the White House ... all so he can race stock cars instead.

But sources tell TMZ ... the swimmer has dreams of being a pro race car driver after his swimming career ends ... he's opting to be in Fontana, CA instead to test drive cars for a few days and then hit up the MAVTV 500 IndyCar event on September 15.

We're told Clary is serious about his future racing career ... he's already been courted by several IndyCar and NASCAR teams and sponsors since returning from London.