Jamie Spears withstood a fierce challenge by ex-wife Lynne Spears in the conservatorship battle over Britney ... TMZ has learned.

Sources with knowledge of Wednesday's court hearing tell TMZ, Lynne wanted Jamie out as conservator, but the judge rejected her claims. Lynne felt Jamie had done a poor job ... something she's been saying since the Free Britney movement last spring.

The judge decided Jamie should continue on as conservator of Britney's estate, meaning all of her business affairs. As we reported, Jamie has temporarily bowed out of handling the other part of the conservatorship -- taking care of Britney, her medical needs and so on. The case manager Jamie hired -- Jodi Montgomery -- has been appointed conservator until January, and that's when Jamie will retake the reins. He is still taking care of all of her business affairs.

Jamie has said he's temporarily stepping down because of medical issues, but we know another reason is that there's currently a restraining order that prohibits him from having any contact with Britney's kids -- this after he allegedly put his hands on 14-year-old Sean.

There are a few complications. We're told Britney hasn't spoken to Jamie since last month when the incident went down between him and Sean. Several sources say they actually weren't speaking even before that.

And, there's this. Several sources say Britney is not doing well. As one source put it, "She goes through cycles, and hopefully we're at the bottom now." We posted this video Friday of Britney at an awards ceremony for boyfriend Sam, and she was so uncomfortable she beelined it for the car before even walking into the event.