'Family Feud' Supposed to be Hiltons vs. Kardashians but Hiltons Bailed

The Kardashians played against each other on "Family Feud," but TMZ has learned it wasn't supposed to be that way ... it was scheduled to be a showdown between the Kardashians and the Hiltons.

Sources connected to the production tell us ... they were in a mad scramble when they were contacted 4 days before Saturday's taping and informed the Hiltons weren't playing.

Our sources say Paris and Kathy were on-board, but patriarch Rick Hilton was having none of it. We're told Rick -- who runs the monster real estate company Hilton & Hyland -- felt it wasn't right for him to appear because he's not in entertainment ... and it looked "thirsty."

We're told Nicky thought it was nuts for the family to appear so she was out, and Barron and his fiancee ultimately decided not to do it.

Fact is ... the families go back 3 decades and they really like each other, but it wasn't right for the Hiltons.