Andrew Silverman to Simon Cowell: You Were Banging My Wife Before We Split!

Simon Cowell's FORMER good buddy claims Mr. 'X Factor' was sticking it to his wife before they parted ways ... and THAT'S why he's accusing Cowell of adultery ... TMZ has learned.

We know real estate mogul Andrew Silverman isn't buying what Simon is selling -- that he didn't lay a finger or any other part on Lauren Silverman until after the split.

There's some proof to back Andrew -- photos (above) of Lauren and Simon kissing during a yacht vacation a year ago. By the way, Andrew was also on that trip.

Andrew, we're told, just wants out of the marriage and away from Lauren and Simon, and he's willing to settle custody and financial issues quickly. For now, he's asking for full custody of their 7-year-old son, but it appears from what we know that's a negotiable point.

A source close to Lauren tells TMZ ... Lauren and her husband were over before she and Simon entered into any kind of relationship.