Simon Cowell -- Life is Complicated with Baby

Simon Cowell was sent packing Friday ... just days after his baby was born, but not because Lauren Silverman threw him out -- because a judge laid down the law in her complicated custody case.

As we previously reported ... Lauren's nasty divorce from Andrew Silverman ended with a custody arrangement for their 7-year-old son Adam, which required Lauren to keep Simon away from the kid for a year. Adam stayed with Lauren over the last few days, which meant Simon needed to skedaddle.

Simon moved into the swanky St. Regis Hotel in NYC ... but we're told he moved back with Lauren when Adam left.

As for the immediate future ... we're told Simon will stay in NYC for 4 to 5 weeks, until their baby is allowed to fly. The new family will then go to London for "X Factor" auditions. They'll be in the UK for around 3 weeks and then return to NYC.

Our Simon sources say he and Lauren will split their time between NYC and London, and since he has a plane it's no big deal.