People often ask me what I think should be done about Vladimir Putin, as though he had suddenly popped up and turned a compliant and benevolent Russia into a malicious, growling bear. But Putin has not sprung from nowhere. He is popular in Russia largely because he is an effective leader of a country that views itself as a great power. He has stood up for its status, honour and self-reliance in the modern world. In doing so he is continuing a tradition that goes back not just to the Soviet leaders but to the tsars.

Like all peoples in a crisis – including the British – Russians value a leader who can maintain both internal order and external security. We often forget that for the past 30 years Russia has been going through a prolonged and serious crisis in which both internal order and external security were vulnerable. The breakdown of the Soviet Union in 1991 created an emergency in which parts of the state broke away and serious conflicts erupted within Russia itself. In 1993 it was on the brink of civil war.

In the early 1990s Russia tried to create a western-style democracy and free market, and to entrust its fate to international institutions dominated by the west (the UN, IMF etc). This turned out disastrously: the freewheeling economy soon fell victim to corrupt “oligarchs”, and ordinary Russians lost much of their material security. For many, “democracy” meant continual instability and grinding poverty.

The country emerged as an impoverished and gravely weakened state, whose military alliance, the Warsaw Pact, had dissolved, while the rival alliance, Nato, not only survived but expanded eastwards to claim former Warsaw Pact members. In 1999 Nato bombed Serbia without UN authorisation to prevent genocide against the ethnic Albanians of Kosovo. Russians regarded this as both a violation of international law and an encroachment on their traditional sphere of influence. Then followed the US-British invasion of Iraq, again without a full UN mandate. Russia’s alienation reached a climax with Nato’s expansion into the former Soviet Baltic republics and its announcement that it was preparing to invite Ukraine and Georgia to join.

The cumulative effect of all these western steps was to provoke Putin into a frank diplomatic attack on the US at the Munich security conference of February 2007. “Nato expansion,” he declared, “represents a serious provocation that reduces the level of mutual trust.” The US, he added, had tried to create a “unipolar” world, in which its own interests overruled international law and agreements. “This is extremely dangerous. As a result no one feels safe. I want to emphasise this – no one feels safe.”

Putin took action accordingly. In March 2014, in reaction to what he perceived as a western-backed coup in Ukraine, Russia occupied Crimea, Ukrainian territory that’s home to the Russian Black Sea fleet. This was a gross breach of international law. When I asked a Russian diplomat about it, he agreed that it was, but added: “It is also a breach of the American monopoly on breaking international law.”

By then a basic conflict had become apparent between Russia and the western powers in their outlook on international affairs. Putin, like many Russians, had come to feel that international institutions could not guarantee their security, and that therefore they were back in a 19th-century world of great nations contesting power and wealth with one another.

Thus, Russia returned to a familiar tsarist geopolitical and military posture, believing that all great nations achieve security through the creation and assertion of raw power. In this view, one side’s gain is the other side’s loss. Win-win situations are not envisaged. It follows that the state has the right and duty to mobilise all the resources of society. The economy, culture, the media, science and technology are all regarded as belonging to the state, to be deployed in this rivalry between great powers. This is what Putin means when he talks of “sovereign democracy” and “information security”.

From Peter the Great in the early 18th century to Nicholas II at the beginning of the 20th, the tsars mobilised their military might, their diplomatic skill and their intelligence services to create and maintain their “great power” status among an unstable cluster of rival European powers, notably Britain, France, the Habsburg empire and Prussia. Thus Alexander I defeated Napoleon in 1812 because Russia excelled in all three branches of government. Putin is trying to ensure that today’s Russia performs just as well in all of them.

But unlike the tsars, Putin has added an ideological element – Russia’s version of soft power. In alliance with the Orthodox church, Russian official propaganda lambasts multiculturalism, gay marriage and mass immigration, and claims to stand up for the family and traditional values against a decadent west that has lost its way and has tried to corrupt Russia. This viewpoint has found a strong echo in the west as well, as a reaction against the international financial “oligarchs” who dominate the global economy: Fidesz in Hungary, the Front National in France, Ukip in Britain and President Trump in the US. There is, if you like, a cold war element here; the Communist International has been replaced by the international alt-right movement. However, this ideological element is mainly instrumental, not dominant, as it was in the cold war years.

Putin is not alone in thinking the way he does. Any successor is likely to have similar views. We have to deal with Russia as it is, not spin fantasies about what it would be like without Putin. We need to stand up for what the west is supposed to believe in – free speech, the rule of law, democracy – and defend those interests that are essential to our way of life. Since many Russians also hanker after free speech, the rule of law and democracy, we can afford to be firm on these ideals.

We also need to understand, though, why so many Russians feel as they do. They respect adversaries who stand up to them but who are also prepared to talk and to try to reach agreements with them. Combining a robust belief in ourselves with a willingness to take Russia’s problems seriously is a difficult balancing act; but it is a necessary one if we are to take the heat out of a confrontation with Russia that could turn very nasty quite suddenly. We are dealing not just with a recalcitrant Putin but with a major European power that has its own view of the world.