Simon Cowell -- Scorned Hubby Says Duh, the Warning Signs Were There!

Simon Cowell's relationship with Lauren Silverman was going on right under Andrew Silverman's nose, and he at the very least "kinda knew" what was going on ... this according to friends of the couple.

Multiple sources tell TMZ ... Lauren would spend hours on the phone at her home talking to Simon, and Andrew heard the whole thing. Several of their friends tell us they believe Andrew knew that Simon was banging his wife, and they think the outrage he's expressing now is a giant crock.

Several sources close to Andrew shade the story slightly differently. They say Andrew is telling them looking back there were plenty of "warning signs" but he just didn't pick up on them. They also acknowledge Andrew knew that Simon and Lauren were "flirting," but he didn't think that necessarily meant they were banging.

As for settling the divorce ... we're told they're still at an impasse, but continue to talk.