Visitors play golf at Muirfield Golf Club, East Lothian
Visitors to Muirfield golf club, East Lothian, which has lost the right to host the Open championship. Photograph: Andrew Milligan/PA

Muirfield golf club has lost the right to host the prestigious Open championship after taking the stunningly regressive step of voting against reversing its ban on female members. In fairness, Muirfield members seem to have some very real concerns – a letter circulated by those campaigning against the change revealed the terrifying prospect that “lady members” may pose a threat to such noble traditions as “our foursomes play, our match system … our lunch arrangements”. Quelle horreur.

The news has prompted intense debate about male-only spaces and whether or not they should still be allowed, with one major argument cropping up again and again in most of the discussions I have heard. “Why shouldn’t men be allowed their own space, when women have women-only gym and swimming sessions? Isn’t that sexist?”

The answer is that these two things are not the same. In fact, they couldn’t be more different.

You want to know why we have women-only gym and swimming sessions? It’s not some shiny, special privilege bestowed on us lucky women because we just deserve nice things. It is a direct result of the male harassment, sexism and sexual violence that has driven a quarter of women to give up exercising outside altogether and countless others to abandon the gym in frustration.

Perhaps you think this sounds like an exaggeration. It isn’t. When I started visiting my local pool with a group of girlfriends a couple of months ago I was excited about the newfound enjoyment I discovered in exercise. And encouraged by a slow but gradual improvement in my speed and strength, I started to feel the exercise having a beneficial impact on my mental health. But it took all of four weeks before a man came in, stood in the shallow end and ducked underwater with a pair of goggles each time we swam towards him to openly ogle our breasts, doing the same to our bottoms as we swam away. He didn’t even pretend to be doing lengths. The negative impact on our enjoyment of the swim, both that day and afterwards, couldn’t have been much greater if he’d urinated in the pool.

I am far from alone in this experience. In three years the Everyday Sexism Project has received a whopping 984 testimonies from women writing about their experiences of sexism, harassment and assault at the gym; 541 related to swimming pools.

They range from women being patronisingly shown out of the free weights area to having their breasts ogled as they use the running machine; from an unsolicited slap on the bottom to a man masturbating in the shallow end. When victims are driven to use women-only gym or swimming sessions, their choice in when they can use the facilities plummets. This is not a privilege.

Meanwhile, as women change their routines and behaviour to cope with sexual harassment and assault, male-only clubs provide a privileged and exclusive space that facilitates and reinforces their domination of arenas such as politics, business and finance. That a single round of golf at Muirfield costs £220 in green fees gives you some idea of the kind of people who meet there, and the conversations, negotiations and deals that might take place over 18 holes. In a similar way, male-only private members’ culture legitimises the uninterrupted boys’ club atmosphere that has existed among powerful elites for centuries. These spaces crop up again and again in the testimonies of women who report being excluded, sidelined and discriminated against in their careers. How many times have we heard the story of important decisions being made on male-only outings, or promotions being “unofficially” determined over a game of golf? Remind me again how David Cameron, Boris Johnson and George Osborne spent their time at university?

Many women-only spaces exist for one reason: to avoid male harassment. Most male-only spaces also exist for one reason: male privilege. Or, to put it more simply, both male-only and female-only spaces have developed as a direct result of male privilege.

Objecting to a male-only golf club doesn’t have to mean arguing that there should be no male- or female-only spaces left in our society, simply that they shouldn’t remain that way if the only goal is exclusion and hierarchy. It is still, for example, useful for male victims of sexual violence or abuse to have spaces where they are able to only meet other men, and the same goes for women. Within the feminist movement, which advocates for equality in a world that remains deeply unequal, it can be useful for women to have some spaces where they organise without male input. But these are very different cases.

In today’s world, to be debating whether an all-male club should “admit women” is hardly progressive anyway. In 2016, as we are finally beginning to acknowledge the identities and rights of transgender and non-binary people it would be much more productive to have a conversation about a leisure club where gender is not a determining factor at all in admittance. Unless you are planning to swing your golf club using your genitalia, it should be nobody’s business but your own.

It doesn’t get much more ironic than trying to defend a privileged all-male club that exists because women are literally considered inferior beings using the example of an all-female space that has been created because of the violence men direct at women. One of these things does not defend the other. They are part of the same ingrained, structural gender inequality.

• This article was amended on 24 May 2016. An earlier version said Muirfield had a joining fee of more than £100,000. That figure related to a different Scottish golf course.