Palantir in the NHS - is it actually any good?

AI image generated by Claude from Anthropic

I have written recently about concerns that the American AI giant Palantir has been given a big NHS contract involving sharing patient data, despite a track record in the US of enabling the surveillance of private citizens. But whatever the concerns about its ethics, I had always assumed that Palantir’s software, used by the Department of Health during the pandemic to give the government an up to date snapshot of the state of play, was pretty good.

Then a few weeks ago I got an email from a man who claimed to be an NHS IT worker, though he preferred to remain anonymous and signed as just “John”.

He told me that the software provided by Palantir for the NHS Federated Data Platform was just “not very good”. He stressed that it was not universally bad and was effective at bringing disparate data sources together “but its functionality for data querying, pipelining, analytics, and visualisation is less effective and produces outputs that fall short compared to existing NHS solutions. It is really a very overpriced data warehouse that the NHS is trying to use to meet all its data needs.”

The main complaint seemed to be mission creep. NHS managers were pressing hospitals and GP surgeries, via the Trusts and Integrated Care Boards that run them, not only to sign up to the FDP but to start using Palantir in other areas. This came at the expense of other software with which NHS staff were familiar and liked better.

The result was that large sums were being spent on “upskilling” staff who were being told their skills in Excel, SQL Server, Oracle, Qlik, Tableau, PowerBI, and R were no longer relevant.

And “John” said there was a climate of fear which made it hard for staff to make their views known:

“Efforts to engage constructively about the software's limitations are often met with accusations of resisting change, and in the current climate of the NHS, it is risky to express concerns openly.”

Now, I was in two minds about this story. On the one hand it sounded convincing, given the propensity of the UK public sector over the years to hand large sums to American software companies without much clarity over. what they are getting. On the other hand one anonymous source wasn’t really good enough, even though when I contacted “John” he gave me his real name and established to my satisfaction that he did work in IT for the NHS.

So I looked around and spoke to a more senior figure managing various data projects in the NHS. He confirmed much of what “John” said and gave as an example a project where his team were trying to bring together various sources of data about cancer:”They were told that if you’re going to be doing this analytic work, you have to do it on FDP.” But when they looked at the software, they found that it was inferior to what they currently used, Microsoft Azure.

He explained that the NHS had developed standard components which meant, for instance, that apps and websites worked for people with visual impairments, and functioned as well on a mobile as on desktop.”FDP components are not as good,”he said. “and the user interface in general wasn’t pleasant.”

But this executive did not want to go public, and I did want, if possible to put a name to criticisms that are also all over forums like Reddit. One man who has talked publicly about the FDP controversy is Marc Farr, chair of the Chief Data and Analytical Officer Network. Speaking on behalf of senior NHS data scientists, he has described himself as a “critical friend” to the FDP.

In public letters to a senior figure in NHS data management, Farr has queried the pressure to do everything in the FDP and believes his concerns have resulted in a softening of that approach. When I spoke to him he made it clear that he had no interest in getting involved in the politics of Palantir. “I’m more interested in, as a taxpayer, how do we make the most out of any money that we spend on data, and I think they’re not going about it in a great way.”

He says that while the FDP contract was worth £330 million to Palantir, the total cost of the system to taxpayers will be nearer to £1 billion because of the costs incurred by hospital trusts: “If Trust X wants to implement it, they need 50 days of a developer and two new project managers and some new computer systems and so on.”

His group is all in favour of bringing data together but not of telling organisations that they have to junk perfectly good systems to handle it: “Allow people to link to it and then take it off to their own thing. Allow them to code in the language that they’re used to. Don’t just make everybody stop what they’re doing and learn a new proprietary piece of software.”

Marc Farr also warns that patient safety could be at risk if trusts are told that the FDP is the only game in town and their existing systems must be scrapped: “There are systems already at the moment that manage the cancer referrals for a patient. If you say, well, in the future, we’re going to build one, we’re saying that sounds great, but until you do, you can’t turn off the system that currently works today.”

On Sunday the Observer ran a largely positive double page spread on Palantir’s role in various parts of the UK public sector, crediting its software with cutting crime and waiting lists and boosting operations. But I was particularly struck by these two sentences:

“The civil service has a long history of failed IT projects. Palantir offers instant solutions and a record of demonstrable benefits.”

The civil service does indeed have a long history of failed IT projects, but many of them involved large overseas tech companies offering “instant solutions” being handed lucrative contracts by mandarins with very little knowledge of IT. There was a valiant attempt to change things when the coalition government created GDS, the Government Digital Service, with a brief to bring some tech expertise into Whitehall and run a critical eye over any plans to sign mega deals.

But senior civil servants resented the upstart intruder and while it still exists it has been largely neutered as an IT spending watchdog, unable to say “just wait a minute” when anything like Palantir’s £330 million FDP contract comes along.

Last week, however, there was a hint that the wind might be changing again. The Science Minister Lord (Patrick) Vallance told MPs on the Commons Science, Innovation and Technology Select Committee that the NHS Palantir contract was signed under the previous government and things were now going to change:
”I hope I have been clear in describing a very different way of doing contracts: putting British companies there and procuring innovation here." But he said that, as far as the current contract with Palantir and pressure from some MPs for it to end early was concerned, that was a matter for the Department of Health and Social Care: “It is a contract with DHSC, and if there are issues around that, they are the ones that need to look at that. I cannot."

When I contacted NHS England - a body soon to be abolished - about pressure to use the FDP I was pointed towards an FAQ on its website setting out “the expectation that all providers and integrated care boards onboard* to the NHS Federated Data Platform and start making use of core products, data capabilities and population health management tools by 2028/29.”

And an NHS spokesperson said:“The Federated Data Platform is already delivering huge benefits for patients and the NHS – joining up care, speeding up cancer diagnosis and ensuring thousands of additional patients can be treated each month.”

But with areas such as Greater Manchester insisting they had already set up data sharing systems before the FDP came along and they are not going to close them down, the scene is set for further clashes over Palantir’s expanding role in the NHS.

*Yes, “onboard” is apparently now a verb.