Astronomy
I don't think this is something merely assumed. Either possibility presents a conundrum: if all galaxies are matter, then the universe started with more matter than antimatter; if other galaxies can be antimatter, then the universe had an unbalanced distribution of the two.
What's our evidence that the remainder of the universe is all matter?