Collection Was Never the Problem: “The Defence of the Realm” by Christopher Andrew

Official histories carry a structural problem: the organization commissioning the book owns the archive, and the reader cannot audit what was left out. Christopher Andrew's centenary history of the British Security Service is the genre's most substantial example, and worth reading partly because it makes that problem visible rather than solving it.

It is also very long — over a thousand pages — and reads like what it is: an official account with a house style and little appetite for speculation. Not a book we would recommend reading cover to cover.

The value is not the spy stories. It is the record of one organization deciding, across a century, what to collect, what to believe, and what to act on.

What the book actually covers

The narrative opens with the Secret Service Bureau of 1909 and its home section under Vernon Kell, who ran it for three decades. Early chapters cover counterespionage against German networks and the First World War, when a small registry-driven operation expanded fast, then had to justify itself in peacetime. The pattern recurs: inflation during expansion, insecurity afterward.

The interwar sections deal with communist subversion, British fascism, and the slow recognition of the Nazi threat. This is the book's best material on misallocation: attention followed the threat the organization was already configured to see, not the one that was growing.

The Second World War chapters are the strongest. The double-cross system, running turned German agents against their controllers, was an operational achievement of a high order. Alongside it ran a failure of comparable scale: Soviet penetration of British government, including the Service itself, largely unrecognized at the time. Read them together. The organization managed extraordinary discipline in one direction while blind in another, and the blindness was not a resourcing problem. It was a judgment problem.

The Cold War sections cover the VENONA decrypts, the atomic spies, the growth of vetting, liaison with allied services, and the end of empire. Then the uncomfortable material: counter-subversion in the 1960s and 1970s, the file kept on Harold Wilson under a pseudonym, Northern Ireland, and the move from a body with no statutory basis to one placed on a legal footing by the Security Service Act 1989.

On who reads what: general counsel and risk leads will get most from the wartime chapters, the early Cold War penetration material, and the sections on oversight. Vetting teams should read the screening material closely. The interwar surveillance detail and colonial chapters are for specialists.

Why it matters for your risk posture

Read it for the gap between collection and judgment, the real subject even where the author does not frame it so. The Service was rarely short of material. It was short of the willingness to draw an unwelcome conclusion from what it already held. Corporate intelligence failures have that shape: the signal was in the file, and nobody was obliged to act on it.

Read it because oversight arrived late, arrived from outside, and improved the organization. For most of its first eighty years the Service had no statutory basis. A legal footing and public accountability were resisted in some quarters and, on the evidence here, made the institution more defensible without making it less capable.

Read it for what institutional learning actually looks like, which is slower and more grudging than any framework admits. Lessons were learned after damage, encoded in procedure, then partly forgotten as personnel turned over. Assume your organization behaves the same way, and design that memory into process rather than into people.

Key takeaways

• Collection capacity outruns judgment capacity. An organization can hold the decisive fact for years with nobody obliged to act. Audit escalation paths, not only data sources.
• Configuration determines what you see. Attention followed the threat model already in place, which is why emerging threats were recognized late. Review the model itself on a schedule.
• Oversight strengthened the institution. Statutory footing and external accountability did not degrade effectiveness. They made the work defensible.
• Official access is not full disclosure. The archive was open to the author; the reader cannot see what was excluded. Treat any commissioned history the same way.

About the author

Christopher Andrew is Emeritus Professor of Modern and Contemporary History at the University of Cambridge and a Fellow of Corpus Christi College, Cambridge. He was a co-founding editor of the journal Intelligence and National Security and for many years chaired the Cambridge Intelligence Seminar, which brings academics and practitioners together under confidentiality rules.

Appointed official historian of the Security Service in the early 2000s, he was given wide access to its files, with no stated restriction on his conclusions. He had earlier written on Soviet intelligence with the defectors Oleg Gordievsky and Vasili Mitrokhin — unusual grounding for an academic.

Christopher Andrew, The Defence of the Realm: The Authorized History of MI5, Allen Lane (Penguin), 2009; published in the United States as Defend the Realm by Knopf.

Beyond the Book

Christopher Andrew — the Cambridge historian who made intelligence a serious academic subject

• Access came with a frame. Written from inside the archive, its judgments are more forgiving than an outsider's would be. The fullest available account, not the last word.
• The other side of the same decades. His work with Gordievsky and Mitrokhin covered that period from the opposing service's records, showing how differently one operation looks from either end.
• The long view. The Secret World: A History of Intelligence (2018) sets the same preoccupations — what gets collected versus what gets acted on — inside a global history of intelligence from the ancient world onward. It is a different book, not a shorter one.
• Read alongside: MI6: The History of the Secret Intelligence Service 1909-1949 by Keith Jeffery (Bloomsbury, 2010), the companion official history of Britain's foreign intelligence service.