Russia's spy agency is waging a massive undercover campaign of harassment against British and American diplomats, as well as other targets, using deniable "psychological" techniques developed by the KGB, a new book reveals.

The federal security service (FSB) operation involves breaking into the private homes of western diplomats – a method the US state department describes as "home intrusions". Typically the agents move around personal items, open windows and set alarms in an attempt to demoralise and intimidate their targets.

The FSB operation includes the bugging of private apartments, widespread phone tapping, physical surveillance, and email interception. Its victims include local Russian staff working for western embassies, opposition activists, human rights workers and journalists.

The clandestine campaign is revealed in Mafia State, a book by the Guardian's former Moscow correspondent Luke Harding, serialised in Saturday's Weekend magazine.

The British and American governments are acutely aware of the FSB's campaign of intimidation. But neither has publicly complained about these demonstrative "counter-intelligence" measures, for fear of further straining already difficult relations with Vladmir Putin's resurgent regime. Putin, a former KGB lieutenant colonel, was head of the FSB.

British sources admit they have files "five or six inches thick" detailing FSB break-ins and other incidents of harassment against Moscow embassy staff. "Generally we don't make a fuss about it," one said. So pervasive is the FSB's campaign that the British government is unable to staff fully its Moscow embassy. The intrusions are designed to "short-tour" diplomats so they leave their posts early, the source said.

Despite a recent improvement in US-Russian relations, the FSB has also targeted US diplomats and their families. In a 2009 confidential diplomatic cable leaked by WikiLeaks, the US ambassador in Moscow, John Beyrle, complains that the FSB's aggressive measures have reached unprecedented levels.

Mafia State recounts how the KGB first became interested in "operational psychology" in the 1960s. But it was the Stasi, East Germany's sinister secret police, that perfected these psychological techniques and used them extensively against dissidents in the 1970s and 1980s. These operations were given a name, Zersetzung – literally corrosion or undermining.

According to former Stasi officers the aim was to "switch off" regime opponents by disrupting their private or family lives. Tactics included removing pictures from walls, replacing one variety of tea with another, and even sending a vibrator to a target's wife. Usually victims had no idea the Stasi were responsible. Many thought they were going mad; some suffered breakdowns; a few killed themselves.

It was Erich Honecker, East Germany's communist leader, who patented these methods after concluding that "soft" methods of torture were preferable to open forms of persecution. The advantage of psychological operations was their deniability – important for a regime that wanted to maintain its international respectability. Putin spent the late 1980s as an undercover KGB officer based in the east German town of Dresden. Harding was himself the victim of repeated FSB break-ins, and last November was, in effect, expelled from Russia when the foreign ministry said it was not renewing his journalist's accreditation.

Mafia State also reveals:

• FSB officers privately admit the agency was involved in the assassination of dissident spy Alexander Litvinenko. They regret, however, the bungled way it was carried out.

• The British embassy in Moscow has a "polonium" chair sat on by Andrei Lugovoi, the chief suspect in the Litvinenko murder. Uncertain what to do with it, officials have locked it in a room in the embassy.

• Russia's footballing union knew a week before a vote in December that Fifa's executive committee would give Russia, rather than England, the 2018 World Cup.

The FSB never explained why they targeted Harding with such zeal. Other western correspondents have also suffered from occasional "home intrusions", but on a much lesser scale.