He is stooped now, standing in the winter sun. A tremor in the stained fingers that light a cigarette. A kind of faded elegance in the white hair and narrow face. An overlay of hesitance, a fragility; the heaviness of numbing medication.

Behind him the hills of the Byron hinterland roll away into smooth green contours. When he withdrew to the New South Wales north coast 15 years ago, Peter Powditch’s work seemed to fold into this landscape. It softened and became paler, elusive, delicate; a suggestion rather than a statement.

The white house that he built stands on a small ridge, filled with light, open to the elemental landscape. “I need to have something to look at, you know.”

Sitting at a table by a garden of vivid subtropical flowers, he smokes and coughs. In 2009 Powditch’s son James was a finalist in the Archibald Prize with a black-and-white portrait of his withered, lined, emphysemic father, bluntly titled Peter Powditch is a Dead Man Smoking. Mitch Cairns’ elegant portrait of him, still smoking, was runner-up in this year’s Archibald.

Powditch, 73, always was a connoisseur of contour. His celebrated Sun Torso series of the early 1970s depicted the voluptuousness of the female form: loose flesh, spray-painted pink whispers of carnality. Erotically charged pots of flowers placed strategically between thighs. “Pink was what I was known for.”

The art critic John McDonald recalls these pictures as “really a big part of the art scene” at that moment. “This was a boom time for Peter. He was one of the cutting-edge artists. They were bold, colourful pictures.”

In the ’70s, Powditch was a star, a maverick at the height of fashion. A bronze Sun Torso won the Sulman Prize when he was 30. “To be honest, in 1972 I was probably the most known artist in Sydney.” Powditch’s view of who would win the rugby league grand final made the front page of The Sydney Morning Herald. “I didn’t know I actually had that power,” he says now. “I didn’t savour it. When it left, I wondered what had happened, what I had done.”

In those heady days an art opening was a major social event. “There would be a band in the street,” Powditch remembers. “All the socialites would come: film directors, actors, people from TV.”

They were days of rebellion, a time to shake off conservatism. It was a revolution. “At a gallery opening we would be rude to all the clients and we would be in the kitchen drinking whisky while they were out in the gallery drinking shitty wine. In hindsight we were just pricks, just arrogant. But I did it out of nervousness. It was easier to do that than stand with someone and explain your paintings. My early works were to get up people’s noses. If somebody liked it, I would be worried.”

Cartoonist Reg Lynch remembers a mural Powditch painted for Sydney airport.

“There was this big elevator that went up to the cocktail lounge, and there was a giant mural that you watched as you went up in the escalator. It was a headland with trees. I was fascinated by that as a kid. It was one of the pictures that inspired me as a visual person.”

Powditch says of the work: “It was quite empty. I felt that was very Australian.”

In the early days, painting for Powditch was a physical act, tactile, painful. “I am haptic; I feel it in my body. I was asthmatic before I started smoking. I would use a sable brush and paint things really precisely and tight, and holding my breath and not breathing. It was just agony. ”

His old friend, the sculptor Michael Buzacott, remembers his acutely tense way of working. “My greatest memory of Peter is him drawing a model and sweating over every decision. Just sitting there looking intensely for five to 10 minutes before marking a mark. This intense concentration.”

McDonald agrees: “He will spend three hours thinking about whether he should move an ice-block stick one inch to the left or one inch to the right and will sit up all night in the studio cogitating.”

But much of Powditch’s work is forgotten, lost in the unremembered. Much more, he destroyed himself. “I thought art was killing me. I felt it was too hard for me. I would paint over everything or burn them. I don’t think it was self-destructive; I was just trying to give up. And yet I would start something the next day.”

Powditch had studied art at East Sydney Tech but finding it too “amateurish” he left after a year to study sculpture under Lyndon Dadswell. He also studied painting with John Olsen, who became a profound influence.

“John explained how to see. He talked about going for a walk on a hill and you stub your toe and there is ants coming out, there is a stick and a rock, and you can paint that as much as the hills.”

Combining these different forms created what Buzacott describes as Powditch’s hybrid style. “He cuts things up. It is very physical. He had to sense his own way.”

In Powditch’s words, he “had to find a way to work that was more with my body and my arms. Cutting and sawing and more sculptural.”

Powditch grew up in Taree, where he was a year younger than the poet Les Murray at school. Murray, he says, grew up in poverty. “His trousers were always too short because he had outgrown them.”

Taree was a place where “people were very unhappy – there was a lot of drinking”. His asthma would miraculously disappear when he went to the nearby beach. His family had a share in a house at Crowdy Head and his coastal connection to that place has imbued and informed much of his work. When later he had four children of his own, he brought them there to holiday. “They regard it as a spiritual home. I suppose I am happy near the sea.”

The aerial view, broken landscapes, the subtle layering of elements, the tender, textural palates of his most recent exhibition in 2012 are filtered through those memories. “With my earlier work – the Sun Torsos – probably I wanted the thing to jump off the wall. But I think what I have done with my later work is in a way just amused myself. The art looks after itself, it is very hard to explain.”

You can hear the pride in his son James. “Nobody paints summer like my father. From the landscapes, the still lifes and those rude nude girls – you can hear the cicadas, smell that salt and feel that old rough verandah. He’s a genius at capturing that bleached-out essence.”

The director of the Australian Galleries, Stuart Purves, sees the evolution of a man who is “the real thing, a real artist”. He sees the paintings thin and become abstracted. “He has broken it down very slowly. He is sort of like a slow-motion surgeon, slicing and carving away. At the end of the operation there is less there but you still see the same thing.”

Says Powditch: “You have got to look for negative space or you have got to go after rhythm, or you have got to have some intent.”

Today, his studio sits idle, the paintings in racks. Intent, for Powditch, has become elusive. “I mean, you slave over a painting and worry about every bit of it. And once you have finished it, it just sits over there gathering dust.”

With a family to support, Powditch spent his working life teaching art, where he felt he could make a difference. “There is always criticism that if you teach, you are not a real artist, but I just loved it,” he says. “I showed off, I performed.”

Lynch, whom he taught in the late ’70s, adored him. “Half the students in my class thought he was a freak. A lot of students didn’t get him, but the ones who did really, really got him. He improvised and visualised and stuff.”

They still email each other. “He will email me strange things or I will email him strange things. He likes found objects and so do I and he likes rubbish and broken things.”

At home in his studio in inner Sydney after days of teaching, he would be making his own work. “I would work late at night until two or three in the morning. But it made me work too hard, I think. It was too hard. I would go to bed all weekend. Or I would go to sleep on the studio floor, curled up on a dusty floor. A bit like penance, or something. I think it is like putting yourself in an uncomfortable situation because you feel that you are not worth anything. All my earlier life my wife was aware that I would have these episodes.”

Until now the only time he has ever not been frantically working was in 1974, when at the height of his fame he took a year off to go to New York on a Whitlam Australia Council grant. He intended it to be a three-year sojourn, until it became obvious there was no more money coming. “I felt more an artist doing nothing there than I did in Sydney working my butt off. There was plenty of things to look at.”

His family well knew that he was obsessive, prone to depression. What was not recognised were his manic phases.

Last year, two of his paintings were shown in the Art Gallery of NSW’s Pop to Popism show. Elated, he flew down for the opening. In his review for The Sydney Morning Herald, McDonald described Powditch’s painting The Big Towel as one of four Australian “stand-outs” and said it looked “as if it was painted yesterday”.

Powditch doesn’t remember much of the episode that followed, which resulted in him being sectioned for 10 days. He went up and did not come down.

“I don’t remember getting on a plane back from Sydney. I don’t have much memory of being driven, and walking outside of emergency, talking to myself and smoking.”

Following the breakdown, he was diagnosed bipolar and medicated with powerful drugs he is still coming to terms with. “I think,” he says, “I was probably bipolar virtually all of my life.”

Made a member of the Order of Australia in 1984 for services to art, he seems now almost forgotten by the wider public. His return to the north coast was a homecoming, a full circle, but it also appears to be a stepping away.

“The intellectual life and curiosity bubbles away mostly in brooding isolation,” McDonald says. “You see him as this person who is kind of pig-headed and introverted and tucked away in Byron Bay. But when you get talking to Peter it is a different person altogether. And you can see in him the person who did those very dynamic pictures in the ’60s.” McDonald believes Powditch withdrew into himself after a nasty court case. He was appointed head of painting at Sydney College of the Arts, and had his appointment overturned when the original incumbent, Richard Dunn, returned from extended leave. He sued the college. “Peter came out of that case very battered and bruised and very badly hurt. Apparently it turned into something like a character assassination.”

After the case, he began making what McDonald calls “little abstracty still life and figures. They were the works of somebody who had disappeared into themselves. He wasn’t going out there saying here are my big museum-scale pieces. Somehow, that was just something he lost interest in doing.”

Says Purves: “He is the sort of person who reluctantly accepts publicity. He is not about compliments at all. That wasn’t what he was pursuing. What he was pursuing was the work.”

But McDonald is not alone in believing Powditch will, in spite of himself, leave a lasting legacy. “I think people will go back to those early works and they will start to look like classics of the period. I have rarely seen a piece from Pete that I didn’t think was a bit of an eye-opener.”

Powditch puts it differently. “I don’t necessarily like my own work. Not really. I do it, but I don’t bathe in its beauty. It is just the work I do and it comes out as it comes out.”