What is a phrase that combines two words with opposite meanings?

The term you're looking for is oxymoron, which comes from a Greek word whose literal translation is 'pointedly foolish'. An oxymoron is a figure of speech in which two apparently contradictory terms appear together. Examples include a deafening silence, harmonious discord, an open secret, and the living dead.

See other Vocabulary Questions.
You may also be interested in: What is the term for a word that has two opposite meanings?
Or take a look at: What is the term for a word that's spelled the same as another word but has a different meaning?