I have never considered myself as a dangerous radical or enemy of the state, and my background and activities would not suggest it – I have been a Cambridge history fellow, am a member of several smart London clubs and even drive a Volvo. Yet I've been spied on by the state.

The monitoring includes my social media accounts, a flyer for a talk I gave at a private club, details of a lecture at a Cambridge Alumni weekend and a library talk with an internal heading by the Cabinet Office of ‘Not just any cook-along this week’. I know this from various Subject Access Requests made to the Cabinet Office and Foreign Office under data protection laws relating to the personal data which they hold on me and which I have applied for over the last few years.

The Cabinet Office eventually admitted that they held so much material on me – they estimated it would take over 656 hours to collect the information – that my requests needed to be broken down into six-monthly intervals.Footnote ¹ Their releases showed that my activities were brought to the attention of the permanent secretary, Sir Alex Chisholm, and the ‘Cabinet Office COPRA team’ (whatever that is); that my speaking engagements, newspaper articles and crowdfunding activities were monitored; and that information was also collected on other parts of my life. This included employment tribunal and linked defamation cases which I had successfully defended and which had nothing to do with my Freedom of Information (FOI) requests or activities as an historian.

My crime? As an historian, to push back against the censoring of our history by government departments and to highlight their failures to adhere to various Public Records Acts and the Freedom of Information Act (FOIA).Footnote ²

My concerns about historical curation go back to researching a biography of Guy Burgess over a decade ago, where I found huge gaps in the record. There was nothing on his time in the Information Research Department, a secret unit set up at the beginning of 1948 to counter Russian propaganda and which he betrayed months after it was set up. Likewise nothing on his time in the News Department, in the private office of Foreign Secretary Ernest Bevin's deputy Hector McNeil nor the British Embassy in Washington between 1950 and 1951 – though there were papers for the period either side of his time in Washington for diplomats doing the same job. In historical parlance this is known as ‘dry cleaning’ the records.

My suspicions about cover-ups were further confirmed when I began researching a book on Dickie and Edwina Mountbatten, the last viceroy and vicereine of India, in 2016. Their letters and diaries had been extensively quoted in previous books, and a major fundraising campaign had been mounted by Southampton University in 2010 to buy their papers so they could be ‘open to all’.Footnote ³

I was therefore surprised to be told by the Southampton University archive that they knew nothing about some of these diaries and letters, part of a £2.8 million purchase of Mountbatten material under the Acceptance in Lieu scheme and with contributions from the Heritage Lottery Fund, Hampshire County Council and other organisations, and withheld other information as they claimed it was exempt from disclosure.

Eventually, after several years, numerous FOI requests, the intervention of the Information Commissioner and the unprecedentedFootnote ⁴ threat of contempt proceedings against Southampton University, in 2019 a Decision Notice was issued ordering the release of some of the withheld material and that they should provide the correspondence between Lord and Lady Mountbatten and copies of their respective diaries.Footnote ⁵

Southampton and the Cabinet Office appealed the decision but then, just before the November 2021 hearing, dumped 99.8 per cent of the material (over 30,000 pages) on the Internet.Footnote ⁶ The material that they had kept closed for a decade, and fought so hard to prevent being made publicly available before my book The Mountbattens: Their Lives and Loves was published in 2019,Footnote ⁷ proved to be entirely innocuous.

And Southampton University knew this, because in March 2018, some eighteen months before they appealed the Information Commissioner's Decision Notice, I was told in a conversation by an employee that a review of the material organised by Southampton and the Cabinet Office had concluded there was nothing sensitive in the personal diaries and letters.

The tribunal, however, ruled that the Cabinet Office still had the right to apply FOIA exemptions to the diaries and letters, which meant that just over a hundred redactions – some a single word, others several paragraphs – were applied on the grounds that they were communications with the sovereign, or that they would damage international relations or national security.Footnote ⁸

Until just before the four-day hearing in November 2021, Southampton had argued they were bound by the mysterious Ministerial Direction controlling the letters and diaries, but they dropped this argument, saying that simply specific FOIA exemptions would be applied.Footnote ⁹ There was no evidence that the diaries and letters had ever been ‘closed’ – neither the Cabinet Office nor Southampton could cite a specific notice – but an effort was made to argue that by implication they had been caught by the ‘undertakings’ concerning Dickie's official papers (‘strayed records’) in agreements in the 1960s and 1980s, though this could not be. The diaries and letters are expressly defined as AIL Chattels in the 2011 Purchase Agreement – not ‘Excluded Records’, that is, the papers that the Cabinet Office had closed. The 2011 agreement expressly stated that the vendors were free to sell all AIL Chattels and that they are not subject to the Undertakings. Thus, the proviso in the Ministerial Direction couldn't apply in any event to AIL Chattels. On my reading of the Decision, the tribunal did not appear to grapple with this point at all.

The upholding by the tribunal of various requested redactions is also baffling. Some names from the royal household were redacted – even if they were already in the public domain from the London Gazette or other books or, indeed, unredacted on other pages of the diaries, both unpublished and published. Other similar roles were not redacted, so there was no consistency in how the FOIA exemptions were applied.

It was also decided that a reference to the leader of Pakistan, Muhammad Ali Jinnah, in Edwina's private diary should be redacted on the grounds that it would be prejudicial to relations with Pakistan, even though the test is that ‘The public authority must show that there is some causative link between the potential disclosure and the prejudice and that the prejudice is real, actual or of substance. The harm must relate to the interests protected by the exemption.’Footnote ¹⁰ It is hard to believe that such a reference in a private diary seventy-six years ago would still damage relations with another country, especially when there is plenty of evidence already in the public domain from books about Edwina, or by family members drawing on their access to the diaries, that Edwina had a low opinion of Jinnah.

This FOIA exemption – section 27 – has a public interest test and Southampton even then had no obligation to apply it but they did so. An FOIA exemption is available to a public authority in respect of any FOI request but it is not bound to plead it; the authority has a discretion – unless providing the information would be unlawful.Footnote ¹¹ Southampton were perfectly free to publish the material that allegedly would damage relations with India and Pakistan, but Southampton chose not to do so – in what looks like an academic institution censoring history.

It is quite clear, contrary to the Cabinet Office and Southampton's claims, that the diaries and letters were open when purchased – they would not have satisfied the Acceptance in Lieu scheme otherwise – and that the reasonable course of action would have been to review the collection to see what could be released when it was acquired in 2011 and not only after they had been forced to do so a decade later. This could easily have been done by experts at Southampton. Instead through to the hearing in November 2021 Southampton claimed that all the diaries and letters were so sensitive they had to be closed, that digitisation would take years, the material was illegible and fragile, etc.Footnote ¹² The fact the material was self-evidently digitised within a few months shows it was possible and not so fragile that it could not be done quickly and easily. If the papers had been bought by an American university, they would have been available to scholars over a decade ago.

Access to the diaries and letters before my book was completed would have made my book richer and more nuanced and, as I was not awarded my costs, it left me personally with a legal bill of over £400,000. No private individual should be financially penalised for seeking access to material which was purchased with taxpayers’ money on the basis that it would be open to the public, but that is the position I now found myself in.

Millions of pounds of public monies were spent purchasing the total Broadlands Archive (even though we don't know exactly what was apportioned to the diaries and letters) to make this important collection publicly available. And then, given that Southampton and the Cabinet Office deployed two top QCs and a plethora of lawyers, probably well over £1 million has been spent suppressing them. However, the Cabinet Office will not say, even after Parliamentary Questions (PQ)Footnote ¹³ and FOI requests, how much public money has been spent on pursuing this needless appeal against the Regulator.

This was only the start of my problems with the culture of secrecy. After I discovered a wartime Federal Bureau of Investigation (FBI) file which claimed Mountbatten was ‘a homosexual with a perversion for young boys’,Footnote ¹⁴ I requested other listed files held on him, only to be told they had been destroyed. When I asked when that destruction had taken place, the American authorities candidly admitted, ‘After you had asked for them.’Footnote ¹⁵ Presumably this had been at the request of the British Government, previously unaware that such damaging material existed.

The Irish police, the Garda, accepted that they had car logs for the visitors to Mountbatten's holiday home in Ireland for August 1977, the month two sixteen-year-old boys claimed he had abused them, but they would not release them on the grounds that they were part of the investigation into Mountbatten's murder – which took place two years later.

Even though we now have a twenty-year-rule for deposit of historical records, I found that no files on Mountbatten's 1979 murder had been deposited in archives, either in Ireland or Great Britain. The Garda claimed it was still ‘an active investigation’, even though the bomb maker had been convicted, served a sentence and was released under the Good Friday Agreement in 1998.Footnote ¹⁶

Indeed many of the files relating to Mountbatten's funeral, seen by millions around the world on television, are closed because they apparently reveal sensitive information about the procession route, who sat in which carriage, etc.

For my next book, researching the Duke of Windsor's time in the Bahamas during the Second World War, I discovered that, while the Colonial Office Files in the National Archives were thin on him, there were mirror copies of the files in the Bahamas. These were much more extensive and full of revealing detail – such as the duke posting the commissioner of police to Trinidad on the morning of a murder which the duke wanted covered up.Footnote ¹⁷

Last year I requested a 1932 police protection file relating to the Duke of Windsor. Dozens of similar files have been available at the National Archives for twenty years. They contain useful titbits on the then Prince of Wales's movements but nothing remotely secret. The Metropolitan Police refused to release the file on the grounds that it would jeopardise the present safety of the royal family.

That decision was upheld by the Information Commissioner's Office (ICO) so I took the matter to a tribunal. A judge asked if I would supply examples of information from other protection files of the period, but, when I sought to do so, I discovered that the twenty files I had highlighted in my submission, and which had been publicly available for over twenty years, had been withdrawn from the National Archives. They included MEPO 10/35 which reveals Wallis Simpson's affair with a car salesman called Guy Trundle, which has been copied and quoted numerous times by historians and is published in all its juicy detail on the website of the National Archives.Footnote ¹⁸ Yet historians cannot look at the original file.

No terrorist has mounted an attack after spending hours wading through such files, yet on no evidence whatsoever the file was closed. Incidentally, I was told by the Special Branch weeder that there were dozens of other Special Branch reports on Edward and Wallis but only this representative file had been preserved. The others were not deemed worthy of preservation. Says who?

This case highlights the worrying increase in the ‘reclosure’ of files that hitherto had been available at the National Archives. My files disappeared almost overnight and it is clear the process is not transparent and does not appear to be subject to any oversight.

The preservation of royal records is a real problem as the division between family and state records is unclear. King Charles has just announced that his mother's diaries, an invaluable historical source, will be vetted by a long-serving footman, a man with no historical training or full understanding of the significance of the diaries.Footnote ¹⁹ Let us hope that they are at least preserved. It is known that Queen Victoria's daughter burnt many of her mother's diaries and Princess Margaret burnt huge quantities of the Queen Mother's papers. The Royal Archives still give no access whatsoever to files on the reign of Elizabeth II, which include correspondence not just with prime ministers of the UK but premiers and governors general of the Commonwealth realms. They also decide which historians they want to let in or not. Cameras are forbidden and there is no public inventory – rather like a restaurant with no menu.

I am not the only historian who has had problems with royal records. The reputable author Christopher Wilson gave up writing his life of the Duke of Kent's father after being refused access to his papers at Windsor. Barrister and former immigration judge Andrew Rose, the author of The Prince, the Princess and the Perfect Murder: An Untold History, looking to update his book about the Duke of Windsor found that MEPO 38/151 (HRH The Prince of Wales: Protection File: 1924–1935), which he had consulted over a decade ago, was no longer available.

Professor Adrian O'Sullivan researching a book on Charles Bedaux, the millionaire industrialist and close friend of the Duke of Windsor, who committed suicide in odd circumstances while in FBI custody, was originally told by an FBI archivist ‘that they had lots of “stuff”’ on Bedaux and was encouraged to submit a formal FOI request to them.

Dr Alison McClean, a lecturer at the University of the West of England, researching support for both sides in the Spanish Civil War among members of the British aristocracy, has also had trouble, not least at The Royal Archives where she ruefully notes that she was allowed to see only three ‘very thin’ folders of correspondence relating to Queen Victoria Eugenie of Spain and Princess Beatrice Orleans y Borbon (two British-born first cousins of George V who returned to England after the fall of the Spanish monarchy in 1931).Footnote ²¹

Recently the campaigning organisation Index on Censorship published a report on censorship of royal recordsFootnote ²² pointing out that almost 500 files at the National Archives were closed including:

• • Royal Family flying training 1977–1978. Record opening date 1 January 2066

• • Family name of Royal Family members 1952–1960. Record opening date 1 January 2027

• • Remains of the Russian Royal Family 1993 Jan.–Dec. 1993. No release date

• • Family name of the Royal House 1952. Record opening date 1 January 2053

• • Air travel for the Royal Family: Containing information relating to the financial arrangements for and other matters relating to the Royal Family 1936–1952. Record opening date 1 January 2053

• • Visits overseas by members of Royal Family 1954. Record opening date 1 January 2055.

Declassified UK recently reported that over 200 files on overseas trips made by King Charles going back to the 1970s remain closed. They include a 1983 visit to Australia which will only be released when Charles is 121 years old.

Historians cannot look at important historical material from almost a century ago yet Prince Harry can spill intimate secrets from a few months ago. As the former MP Norman Baker, author of And What Do You Do? What the Royal Family Don't Want You to Know (2019), has said, ‘There's no reason for these to be kept secret. The normal excuse given is that it's to uphold the dignity of the crown. But the dignity of the crown is upheld by them not behaving in an undignified manner.’Footnote ²³

There are lots of techniques used by public authorities to avoid disclosure. They can kick the can down the road as long as possible, sometimes amounting to over a year. They can keep changing the exemptions deployed as each is addressed and shown not to apply. They can simply not answer requests and hope the requestor gives up. They can play with semantics in carefully phrased replies which are economical with the truth. They can agree to release documents and then do nothing or redact them so heavily as to make them worthless. They can aggregate separate requests and then refuse on grounds of costs of compliance.

Time and time again, authorities hide behind national security or law enforcement or claim not to have material, only to miraculously find it when evidence of its existence is presented. Intriguingly, only the most sensitive documents are ever affected by damp or asbestos. After I recently requested some files on the Lord Lambton political scandal of fifty years ago, I was told by the Metropolitan Police they had ‘lost’ one of them though it remains in the National Archives catalogue. They have also admitted that they have still not catalogued many of their interwar papers.Footnote ²⁴

A favourite trick by public authorities is to use section 22 (where the information is held by the public authority with a view to its publication, by the authority or any other person, at some future date) but where the material mysteriously never finds its way to the National Archives. A weeder has personally told me that when in doubt reviewing material, they are told to just use an absolute exemption, such as section 23 national security.

That is if the documents have not already been destroyed. A recent FOI request to the Foreign, Commonwealth and Development Office revealed that destruction of their files with no public record is routine:

2013 – 7,066

2014 – 52,352

2015 – 30,634

2016 – 21,886

When I asked for details of the files destroyed, I was told by their information rights team: ‘We have considered the publication of file destruction list and concluded that this would detract from our release programme.’Footnote ²⁵

My experience with Mountbatten is a good example of public authorities often pleading scant resources when responding to FOI requests, yet deploying costly lawyers to battle invariably under-represented requestors and to try and break them financially. It is clear that the Freedom of Information Act 2000 – both in terms of legislative reach and enforcement power – is simply not fit for purpose and that parliamentary unease at the antics of the Cabinet Office and the weakness of the ICO as a regulator is justified.

There are too many loopholes (‘exemptions’ which are very broadly drawn), and the government, particularly the Cabinet Office, has become extremely adept at exploiting them. They include, from my own experience:

1. (i) The licence given to public authorities to entirely change their reasons for refusing to disclose information at almost any stage, however late, in proceedings. This leads to what could be described as the absurd game of ‘whack-a-mole’ – and of course drives up costs. In my case the Ministerial Direction was used to justify closure for a decade until I questioned its existence.

2. (ii) The lack of any mechanism to ensure public authorities adhere to deadlines, or even Decision Notices. Because they can ignore these with impunity, researchers are faced with the prospect of incurring costs to bring a delinquent body into line. My lawyers constantly had to chase both Southampton and the Cabinet Office for responses, even though they had statutory time limits, to the extent of bringing contempt proceedings.Footnote ²⁶ All this cost me money.

There is scope for some simple reforms such as:

• • Statutory deadlines for an authority to respond. There are, for example, no enforceable deadlines for Internal Reviews, which should take no more than forty days. It wouldn't be difficult to tweak FOIA to include unequivocal – and actionable – timetables across the process.

• • ‘Deemed refusals’. Scotland's FOIA includes this provision, by which the absence of a response within the required timetable is taken as a formal refusal, which can then be appealed by the applicant.

• • Sanctions for failures to comply with timetables. Public authorities, routinely flout deadlines – whether statutory or in ICO guidance. The way to discourage this is by an automatic financial penalty, payable to the applicant, for every deadline missed. Train companies, for example, are now required to pay passengers what amounts to a fine for failures to arrive on time: there is no reason why Whitehall should be any different.

• • Severely reduced licence for public authorities to ‘change horses midstream’. If the Cabinet Office pleads section 22 at first FOIA request, it should not be allowed to amend that to a different exemption without the explicit permission of the regulator. And the bar for being allowed to do so should be set extremely high, with accompanying statutory requirements for the disclosure of evidence supporting any such request.

The culture of cronyism needs to go. Either archives are secret or they should be made available to everyone at the same time. Tame journalists are often tipped off about document releases well in advance of the rest of the media and there are a select number of writers who are given privileged and exclusive access to write commercial books.

There needs to be proper, separate oversight. Internal reviews are conducted by the same department and, in my experience, they have all upheld the original decision. There is the Advisory Council on National Records and Archives Committee, but it has little power and its members appointed by the Department for Culture, Media and Sport. It needs to be replaced with a much more robust and independent body and given stronger powers.

The ICO requires more money and staff, it needs to be truly independent of the Cabinet Office (who are the worst abusers of FOIA) and it has to be prepared to use its enforcement powers. Indeed, I believe the ICO should be left with just its data protection role and a new regulatory body for information rights set up.

There also needs to be a sea change in attitudes in Whitehall. The weeders need to have a lighter touch and FOI requests need to be dealt with more quickly, while a rather more enlightened attitude must be taken towards FOI exemptions to really protect what is important.

The balance between accountability and transparency on the one hand and protecting national security on the other is a difficult one to strike. Once records are released the genie is out of the bottle, but it is hard to argue that records which in many cases are over sixty years old, and where the officials involved are dead, should not be released. If our history is to be written accurately, we will have to have all the records made available – not just those a government department believes we should have – and historians should not be penalised for seeking to ensure that happens.

Our duty as historians is to try and tell the truth about the past. We cannot do that without the documents being available. This suppression of documents is profoundly undemocratic and reduces trust in the institutions which are meant to serve us. Southampton University and the Cabinet Office's campaign against me was not about safeguarding national security, international relations or data protection but an attempt to make an example of an historian who refused to believe their lies and break me financially. It's a story about the abuse of state power, the failings of the FOI Act, the tribunals and the ICO, the kowtowing of an academic institution to the state, the interference of the royal household in trying to suppress an archive freely sold by one of their chums for in effect £5 million. Let it not also be about the failure of us historians to stand up and challenge this censoring of our history. It is incumbent on us all to highlight these transgressions and, if necessary, challenge FOI decisions with the ICO and in the tribunals.