Former Fairfax correspondent Hamish McDonald joins the ANU College of Asia and the Pacific next year as a visiting fellow.

Just over three years ago the Japanese tried to break out of the cage that looked to be condemning them to a stagnating economy, declining population, and rising friction with their Asian neighbours. In an overwhelming vote in the August 2009 elections, they opted to dump the conservative Liberal Democratic Party that had ruled them for most of the postwar years and elect the Democratic Party of Japan led by Yukio Hatoyama.

It was a reckless leap into the dark for many voters. Several used the phrase Eejanaika (roughly translated: 'Isn't it grand!' or 'What the hell!') when explaining to me why they had switched votes. The conservative commentator Hideaki Kase likened the mood to the millenarian crazes known as okage-mairi that swept Japan under the shoguns, when villagers abandoned their fields to seek supposed magical amulets from far-flung temples.

The Japanese revolt met an immediate and brutal put-down by Washington. Now like the inmates of the mental asylum in One Flew over the Cuckoo's Nest, they have been brought back from their spree and locked up again, their old dependencies waved over them, with the victory of the LDP and even more incredibly, the return of the febrile Shinzo Abe to the prime ministership.

The low turnout in Sunday's election suggests a demoralised country.

There were no burning issues of difference between the parties (the long-mooted rise in the very low consumption tax to tackle the immense public debt was pushed to the background). The winner is likely to turn Japan's nuclear power stations back on, a deep public concern after the Fukushima disaster.

Abe is likely to counter Chinese and South Korean tests of Japan's resolve on disputed islands with more military assertiveness, while worsening the atmospherics with his unrepentant views about Japan's actions in Asia before 1945.

Just what Japan tried in 2009 – and was quickly jumped on by the Obama Administration to overturn – is set out in a new book by two eminent historians of modern East Asia, the Australian National University's Gavan McCormack and Vancouver-based Satoko Oka Norimatsu. Titled Resistant Islands, it draws a wide picture around the efforts by the people of the Okinawa island chain, Japan's southernmost prefecture, to throw off the enormous US military presence lodged on their limited land area since the horrific battles of early 1945, when a quarter of the Okinawan population died as drafted civilian pawns in the defence.

Okinawa is central to the story of the US-Japan security relationship, which has steadily strengthened from the original 'treaty' signed amid great protest in 1960 into Washington's most important 'alliance' in the Asian Century.

While Japan isn't fully incorporated into the highest-level intelligence exchanges with Australia and the other English-speaking powers, the bases it concedes to US forces, most of which are concentrated in Okinawa, have been essential for a close-in posture around East Asia to counter Soviet, North Korean and Chinese power. Close to Okinawa, China is now ramping up its challenge by sea and air incursions to ownership of what it calls the Diaoyu islands, an uninhabited group annexed as terra nullius by Japan in 1895 and named the Senkaku islands.

The potential for conflict is high. Washington has already said that, although it makes no judgment on the ownership question, its defence treaty covers all territory administered by Japan.

Hatoyama's party attempted to break the nexus of bureaucrats and industry that saw tax revenues and savings directed into hard investments (the 'concreting of the Japanese archipelago') and placating farmers rather than urban households. In foreign policy, Hatoyama pushed the idea of an Asian yuai (fraternity) that harked back to the 'continental' orientation and friendship of the 'Yellow' races propounded by the more idealistic Japanese statesmen of a century ago, when others saw Japan joining ranks with Britain as a great sea-power a cut above the backward Asians.

Both these objectives were quixotic and woolly-minded, perhaps, but not so a third policy of Hatoyama. He sought an immediate renegotiation of a deal to relocate the huge US Marine Corps base, Futenma, located in the densely-populated city of Ginowan on Okinawa's main island. Its constant air activity and sleazy off-base nightlife had made it focus of protest. He also talked of a more 'equal' alliance partnership with the US.

Since the end of the Cold War, negotiations had dragged on with LDP governments to find a way to lighten the US footprint in Okinawa while still keeping the forward position for US forces. The key element was closure of Futenma, with some of its Marines relocated to new barracks in Guam and others plus their aviation assets to a minor US Marine base in the northern part of the island at Henoko, to be extended by building a new airfield over adjacent coral reefs. Japan would foot the $US6 billion bill.

With LDP prime ministers (including Abe) spinning out of the revolving-door leadership, Obama's Secretary of State, Hillary Clinton, moved to clinch a deal with the last LDP lame duck PM, Taro Aso, in February 2009. With a high degree of cynicism, the two sides elevated this agreement to treaty status, binding Tokyo to a deal they knew would soon be voted against.

The newly-elected Hatoyama's call for Futenma to be relocated out of Japan altogether or at least out of Okinawa was immediately rebuffed. Obama refused to meet the Japanese prime minister and sent Defence Secretary Robert Gates and Joint Chief of Staff Chairman Admiral Mike Mullen to Tokyo to say the deal was 'non-negotiable'. Washington defence hawks such as Michael Green and Richard Armitage warned of a 'crisis' if Tokyo persisted. Hatoyama and his colleagues were denigrated as 'irrational' and like 'children playing with matches'.

'No major ally had ever been subjected to the sort of abuse and intimidation that characterised the Hatoyama era', write McCormack and Norimatsu. 'Obama, having risen to power in his own country promising change, forbade it in Japan.'

Hatoyama caved in and agreed to pursue the deeply unpopular agreement. But this and other confusion forced him out of office after only nine months, in June 2010. His two successors, Naoto Kan and Yoshihiko Noda, gave nominal support to the agreement. But a year after Hatoyama's resignation, the supposedly essential Futenma relocation was already looking unlikely to ever happen. The US budget crisis put it out of reach financially and the growing range of China's missile forces made concentrations of US forces in Okinawa and Guam more risky.

It would be nice if Canberra's decision to accept a regular six-month 'rotation' of a US Marine task force in Darwin – announced by Obama in the Australian Parliament in November 2011 — was part of an Australian effort to help lighten the load on Japan, but there is no sign of it. Canberra has, however, pursued a closer defence alliance with Japan under Kan and Noda, involving exercises, more intelligence exchanges and possibly technology exchanges (including submarine designs). At the most recent 2+2 talks (of foreign and defence ministers) much was made of shared strategic objectives and values.

Now this might come under a little strain. Abe is a worshiper at Tokyo's Yasukuni shrine, where wartime leaders are venerated among the war dead, and appears to share the view illustrated in the shrine's museum, that Imperial Japan's sweep across Asia was a sincere effort at liberating fellow Asians from Western dominance.

A counter-vision, founded on a historical Okinawa that was a halfway house between China and Japan, rather than a forward military base, is put by McCormack and Norimatsu. As drums beat louder in Northeast Asia, it is tantalising to think what Hatoyama might have achieved with a bit more time and tolerance.