Patrick Swayze Brother Says Late Actor Called John Leguizamo Difficult

John Leguizamo might not remember Patrick Swayze fondly all these years later -- but he might be pleased to know the feeling was mutual ... so says PS' brother.

We spoke to the late actor's sibling, Sean, who tells TMZ ... he actually recalls having a conversation with Pat on the heels of them wrapping production for their 1995 film 'To Wong Foo' -- in which Patrick and John both starred as drag queens.

According to Sean ... Patrick relayed to him that he found JL to be a difficult person, that he had an attitude and they butted heads a lot on set ... so that part of John's story's confirmed.

Whereas John labeled Patrick neurotic and cantankerous while shooting ... Sean says he's never heard one complaint about his brother, and tells us Pat tended to get along with everyone he worked with. He also says Patrick and 'TWF' costar Wesley Snipes hit it off on the flick.

Sean -- who works in showbiz too -- tells us a story he says occurred between himself and John -- where they met at a wrap party for "What Dreams May Come," and he claims John was less than pleasant. It was at that point he says he knew Pat was right.

Remember ... this feud kicked off earlier this week when John told Andy Cohen that Patrick "couldn't keep up" while they were filming the cult classic comedy, saying he was rigid and wanted to stick to the script ... whereas John was more willing to improvise and cut loose.