In Australia's case, it was quite intentionally planned and featured government involvement including picking a date and removing tariffs from colour TVs. This is one reason why theirs came a lot later. They wanted to time the market saturation of colour TVs with the Studio rollout of colour equipment.

Australia was to have one of the fastest change-overs to colour television in the world – by 1978 over 64% of households in Sydney and Melbourne had colour television sets.

Contrast to the much earlier US rollout:

Half of all U.S. households had television sets by 1955, though color was a premium feature for many years (most households able to purchase television sets could only afford black-and-white models, and few programs were broadcast in color until the mid-1960s).

So to answer your question, it depends on where you were. The vast majority of Americans were not watching colour broadcasts when they first aired, and the majority of Aussies were (except 20 years later).