Taylor Swift left a recent concert in Austin with a 39-year-old man in hot pursuit ... according to police, who just busted the guy.

Frank Andrew Hoover was arrested Thursday by Austin PD, and police docs say he'd been hounding Taylor and her family for months leading up to her Oct. 22 concert at Formula 1 race. The night of the show, he allegedly followed her motorcade from the track to the airport where a private jet was waiting for Taylor and her parents.

Swift's bodyguards told police Hoover got within 25 to 50 feet of Taylor's car, and when they asked what he was doing there, he said he was there to get a pic and "possibly accompany Taylor wherever she goes."

Hoover also allegedly flooded Taylor's dad with emails from Aug. 28 until Oct. 27, and some included creepy messages like ... "Without her, I walk the earth alone forever and she'll continue to experience failed relationships that break her heart."

Swift already had a protective order in place against Hoover, which required him to stay 500 feet away from her.