Diggs -- the father of a mixed son -- blasted Harrison in a statement to TMZ Sports ... chiding the ex-NFL star's assertion that Kaep didn't understand the hardships of black people.
"I'm still baffled at the idea that one person is justified in telling another person what suffering is, regardless of race or hue or skin or anything else. No one can or should speak on anyone's experience unless they are that person or have walked in their shoes."
Harrison has since apologized ... saying he didn't realize Kaep was mixed .... but Diggs thinks the whole method of thinking is wrong ... and saved his sharpest and most explosive criticism for last.
"When there can be so much negativity coming at African-Americans from so many directions, why do we continue to fight each other? Dare I say it's the same divisive perspective used by slave owners between black workers in the field and lighter workers in the house. It worked back then, must we continue to let it work now?"