Jermaine Jackson and ex-wife Alejandra's custody hearing ended last week with his request for 50/50 custody being denied, and still having to pay spousal and child support -- but the battle isn't over yet!

TMZ has obtained court documents granting Jermaine's request to seek amendment nullifying his marriage to Alejandra -- because she was allegedly married to Nicholas Ray at the time she and Jackson tied the knot! The judge, who called the case "a giant fog," has sent the wayward duo to a mediation appointment in Conciliation Court to sort it all out.

This messy case has a laundry list of he said/she said declarations. Adding to the weirdness ... before she married Jermaine, Alejandra had two children with his brother Randy!