# PHQ-9 “over-diagnosis” paper shows that arithmetic works

by with Comments Off on PHQ-9 “over-diagnosis” paper shows that arithmetic works

A recent paper by Levis et al. (2020) systematically reviews studies looking at depression prevalence in two ways: one using a structured assessment completed by a professional (SCID) and the other using a questionnaire completed by study participants (PHQ-9). The authors conclude that “PHQ-9 ≥10 substantially overestimates depression prevalence.” But this was entirely predictable.

Mean SCID-prevalence was 12.1%.

Mean PHQ-9 prevalence (using a score of 10 or above to decide that someone has depression) was 24.6%.

This is almost exactly what arithmetic predicts; my back-of-envelope estimate of what PHQ-9 would say (see below) gives 23.8%, using estimates of PHQ’s sensitivity and sensitivity from a meta-analysis (88% and 85%, respectively) and the SCID-prevalence found in the review (12.1%).

So the paper’s results are unsurprising.

PHQ-9 (and any other screening questionnaire) gives better predictions in groups with higher rates of depression, such as people who have asked for a GP appointment because they are worried about their mental health.

No clinical decisions – such as whether to accept someone for treatment – should be made on the basis of nine tick-box answers alone. Questionnaires can also miss people who need treatment.

Screening questionnaires are often designed to over-diagnose rather than risk missing people who need treatment, under the assumption that a proper follow-up assessment will be carried out.

When reporting condition prevalence, the psychometric properties of measures should be provided, including what “gold standard” they have been validated against, and the chosen clinical threshold.

Explore PPV and NPV using this app.

### Back of envelope

P(SCID) = .121
P(PHQ | SCID) = .88
P(not-PHQ | not-SCID) = .85
P(PHQ | not-SCID) = 1 – P(not-PHQ | not-SCID) = .15

P(PHQ & SCID) = P(PHQ | SCID) * P(SCID)
= .88 * .121
= .10648

P(PHQ & not-SCID) = P(PHQ | not-SCID) * P(not-SCID)
= (1 – .85) * (1 – .121)
= .13185

P(PHQ) = P(PHQ & SCID) + P(PHQ & not-SCID)
= .10648 + .13185
= 0.23833

Thanks Chris, for pointing out the typo!

# Unsticking social research through lived experience and citizen control

by with Comments Off on Unsticking social research through lived experience and citizen control

Having lived experience and knowing people with lived experience are really effective way of researching social conditions—unavoidably, whether or not you want to—and lead to rich theory.

Compare what activist groups do versus a model of social research in which you have a central institute, running surveys and writing supposedly “independent” reports, making policy proposals. The latter leads to flat, superficial theorising if done without lived experience.

In activist groups with rich communication (e.g., chat groups and regular meetings) the “data collection” is continuous, doesn’t feel like research, and is inseparable from day-to-day individual support and activism. But traditional reports can still be important to get media and government attention: “What’s a Nice Girl Like You Doing in a Job Like This?” is a good example of research drawing on lived experience and traditional research skills.

To unstick social research requires holding onto all methodological advances whilst radically opening up research to citizen control. Sometimes getting a good estimate of the population prevalence and correlates of some form of oppression are important to highlight severity and likely causes. Advances in techniques and software for qualitative analysis can be useful too and ensure best use is made of material.

Academics without lived experience running convenience sample qualitative studies with small numbers of people and pretentious methodology are fundamentally limited in what they can discover. But the same sample from lived experience and lived theory is very different.

There are many professional researchers with lived experience (Max Weber, 1864-1920, was one, with experience of psychiatric inpatient stay). But higher education is a hostile environment now—you couldn’t design a better system to reward junk research and cause burnout if you tried. Such a system is deeply challenging for people who are oppressed.

Your various identities, privileges and oppression (due to race, man/woman/non-binary, cis/trans, wealth, monogamous/poly, how valued your labour skills are, property ownership, disabled, etc.) fundamentally constrain who will answer your calls for research participants, what social phenomena you can understand, who will listen to what you discover. They literally change what you see and hear and what you can research. (Epistemic relativism is a useful concept to make sense of this.)

Some researchers break free of these constraints thanks to contradictory locations; for instance, being articulate and well connected can be used to resist a position of oppression. Though then you can end up being attacked for having helpful privilege, even by “your own side”.

Academics with more secure positions can help, for instance:

1. Support PhD students and colleagues who are oppressed in various ways: grants, decent pay, and mentoring are helpful.
2. Instead of “giving voice” to people through interview excerpts, give a platform.
3. Cite blog posts and reports from activists with lived experience.

# What happens when you write to the Daily Mail about factual inaccuracies

by with Comments Off on What happens when you write to the Daily Mail about factual inaccuracies

On 23 October 2018 16:18 I used the DM factual inaccuracies contact form to complain about an article on using hot baths to cure depression.

Here’s what happened.

I wrote:

Firstly, it notes that “The research was published in the journal bioRxiv.” This is misleading. bioRxiv is not a journal; it is “a free online archive and distribution service for unpublished preprints in the life sciences” – it is not yet a published paper.

Secondly, this is one study with a small sample size and large dropout in one group. There was no statistically significant difference at 8 weeks – and that’s according to a very weak notion of statistical significance which is relatively easy to achieve (p < .05).

Thirdly, even if taking a bath is better than exercise, is exercise an appropriate control group for a study of treatments for depression? Readers might hope you would consult an expert. What do NICE guidelines currently recommend for patients whose depression is as severe as those in the study?

They replied on 24 Oct 2018 at 16:02:

The following article is a description of the bioRxiv report. We are conveying the results of their report, and many of the issues you raise are ones with the bioRxiv report which we do not claim to endorse or state as unequivocal fact.

It is always good to receive feedback from readers, whether positive or negative. The contents of your email have been noted and have been passed to our editor for review.

I can confirm the article has been updated.

We appreciate you taking the time to get in touch.

I’m not reassured very much by the edit, but sharing details of the process in case helpful…

# NHS England mental health clustering implementation “disappointing”

by with Comments Off on NHS England mental health clustering implementation “disappointing”

A document is circulating from NHS England and NHS Improvement (13 Aug 2018) on the current state of payment systems and clustering in mental health services in England.

It cites “local pricing rule 7” from the 2017/18 and 2018/19 National Tariff Payment System (NTPS) and reports on a survey of progress towards implementing the rule.

Here is what rule 7 said (p. 114):

Rule 7: Local prices for mental health services for working age adults and older people
a. Providers and commissioners must link prices for mental health services for working age adults and older people to locally agreed quality and outcome measures and the delivery of access and wait standards.
b. Providers and commissioners must adopt one of the following payment approaches in relation to mental health services for working age adults and older people:

i. episode of care based on care cluster currencies
ii. capitation, having regard to the care cluster currencies and any other relevant information, in accordance with the requirements of Rule 4(b) to (e)
iii. an alternative payment approach agreed in accordance with the
requirements of Rule 4 (b) to (e).

Commissioners and providers (233 in total) were asked, “What payment approach do you have in place with your contracts for working age adults and older people in 2017/18?”

Here are the results:

So only 14 out of 223 responses (6%) reported a move away from block contracts – the whole point of the new payment systems! The report notes, “The results were disappointing.”

Reasons given by respondents for the poor implementation included:

• “limited local capacity to implement a new payment approach”
• “lack of shared confidence in cost and activity data”
• “uncertainty about how the proposed payment approaches would relate to the new operating models that would develop as part of integrated care systems.”

Services are supposed to be “clustering” the patients they see, irrespective of whether the clusters are used for payment. Rule 6 (p. 114):

Rule 6: Using the mental healthcare clusters
All providers of services covered by the care cluster currencies (see Annex B3) must record and submit the cluster data to NHS Digital as part of the Mental Health Services Dataset, whether or not they have used the care clusters as the basis of payment. This should be completed in line with the mental health clustering tool (Annex B3) and mental health clustering booklet to assign a care cluster classification to patients.

The research on clusters is damning. A recent study (Jacobs, et al., 2018) found that clusters were not very good at characterising the costs of different kinds of treatment and support (p. 7):

“Clusters are therefore not performing very well as a classification system to capture similarities and differences between patients. The categories of the current classification system appear to be neither case-mix nor resource homogeneous. We find evidence of large variation in terms of activity and costs within clusters and between providers.”

Surprisingly, the authors argue that clustering should continue (p. 7):

“… any payment approach needs to be underpinned by a solid classification system and to abandon the clustering approach now will thwart all progress. The clustering approach is already relatively well-established among most providers. Scrapping it all and starting from scratch risks putting mental health services back a decade in terms of developing a more transparent and fair funding system.”

Given the survey results above, it’s unclear how much progress would actually be thwarted by ditching clusters.

If you enjoy this sort of thing, you might also be interested in:

# On evidence in psychological therapy

by with Comments Off on On evidence in psychological therapy

It sometimes feels that the demand for evidence for psychological therapies is seen as offensive. How dare researchers, commissioners, and policymakers reduce therapeutic relationships to tickbox questionnaires and symptom reduction! Therapy is something larger, broader, more holistic than that.

There are clearly problems with using short questionnaires consisting of a handful of closed-questions, and then summing these to a distress score before and after care. But the other extreme, trusting the professionals who are paid to provide therapy to say how effective their therapy is, doesn’t sound convincing either – especially for therapists in private practice who charge by the session. If a therapist depends on long-term therapy for their income, it might be challenging for them to think through different perspectives on how helpful their brand of therapy actually is.

Improving Access to Psychological Therapy (IAPT) is criticised for being heavily manualised, too brief, and offering only a narrow range of therapeutic approaches. However, perhaps IAPT is helpful for some people; for instance people experiencing panic attacks triggered by certain situations or specific phobia, in the context of an otherwise typically but not unusually bumpy life. The research evidence and testimony seems to lend support to this.

I am particularly suspicious of wealthy “woke” psychoanalysts arguing passionately for their Freudian or Lacanian approach. People can need long-term – in some cases life-long – and frequent support, especially for conditions and predicaments which have complex causes and in the context of brutal cuts to social security (welfare “benefits”). It is not obvious that this support needs to be steeped in psychoanalytic jargon.

Having someone there who is kind, listens, and is trustworthy might make all the difference to someone’s quality of life and ability to survive a harsh world. There is no need to run a multi-million pound clinical study to demonstrate the efficacy of kindness. However, therapeutic relationships need to foreground honesty concerning their limits. A paid professional who is there for us each week is not the same as a friend, and that needs to be clear before a relationship begins.

# How to find money to save the NHS

by with Comments Off on How to find money to save the NHS

This morning was the launch of the Resolution Foundation report, Healthy finances? Options for funding an NHS spending increase – a response to the rumoured government “birthday present” for the NHS as it turns 70 on 5 July 2018.

The audience included party advisers (I spotted a Whatsapp group chat for a party’s comms team), people from various think-tanks, academics (including an Emeritus Professor from Imperial who had a lot to say), and a representative from at least one (non-militant, at least in the room) campaigning group.

The line up:

• Sarah Wollaston MP, Chair of the Health Select Committee and medic who worked as a GP up until 2010. (Given her generally sensible views, I keep having to remind myself that she’s a Tory).
• Jon Ashworth MP, Shadow Secretary of State for Health, who has a long history as a Labour professional, including as Special Adviser in the Treasury for Gordon Brown.
• Ben Page, Chief Executive of Ipsos MORI and fellow of the Academy of Social Sciences.
• Matt Whittaker, Deputy Director at the Resolution Foundation, who previously worked for the House of Commons Library where he provided stats and economics advice.

The event was chaired by Torsten Bell, Director of the Resolution Foundation, former adviser to Ed Miliband (and, incidentally, architect of the Ed Stone).

#### Interlude: What is the Resolution Foundation?

The launch was set at Resolution HQ in a bright, wide room, with cosy luxurious seats which wouldn’t be out of place in an up-market indie cinema, so I was curious who they are and how it’s all funded.

Resolution Foundation’s website describes it as “a non-partisan and award-winning think-tank that works to improve the living standards of those in Britain on low to middle incomes.” In their most recent annual report, they defined “low to middle” as those in income deciles 2 to 5, whom they say are overlooked in policy debates. Their focus is on working households.

They receive most of their funding via donations from Resolution Trust, founded by Clive Cowdery with a £50m donation, “believed to be one of the largest endowments for public policy research made in the UK”.

Cowdery made his wealth from “sponsoring insurance vehicles” (an FT article says more) and is also founder of financial services investment firm called (again) Resolution.

Resolution Trust backs Prospect Magazine and, intriguingly, WorkerTech, which seems to be about encouraging alternatives to trade unions for the precarious world of Uberified work. (Here are slides from its launch.)

#### What did they say on health funding?

All agreed that the NHS needs more funding, so the question is how much more funding and where the money is coming from. The issue was framed as a tug-of-war between Treasury and Jeremy Hunt (with Hunt wanting more money, in case not clear – it’s not always obvious), constrained by a complex parliament and a wish to keep voters happy.

Matt Whitaker took us through some headlines from the Resolution report, emphasising that it was a prediction of what the government was likely to announce rather than what it should do. (Though it sounded very much like advice.)

Borrowing was seen as likely necessary, so long as the total was below 2% of the projected GDP in 2020-21 (to meet a Tory fiscal target). But borrowing alone would not suffice, so some sort of tax raise is almost certainly on the cards – the problem is how to keep keep Tory voters and donors on side, whilst getting it through parliament.

One possibility is increasing National Insurance contributions (or NICs, pronounced “nicks”), which Gordon Brown did when he was chancellor. This is a progressive tax for workers; however, increased NICs was seen by the report authors as “unfair from a generational perspective” since older people who rely more on healthcare don’t pay national insurance (this generational perspective might need some analysis). A solution proposed was to extend NICs to include those above state pension age who are still working.

Increasing income tax could be another way to get the money. LibDems and SNP might support this, and Scotland recently introduced a change to its tax bands meaning some pay more and others pay less tax. Labour, the authors argue, would likely oppose increases for anyone earning under £80k and some Tory MPs might oppose too.

Another approach suggested was to adjust thresholds for (i) when income tax is payable and (ii) the higher rate of payment. Threshold changes were Tory manifesto promises, but the authors suggest a fiddle (p. 24):

“An alternative approach would be to lift the Income Tax thresholds to those pledged in the manifesto in 2020-21, but to freeze both them and the NICs thresholds in the final two years of the parliament. This would of course cost money in 2020-21, but by 2022-23 it would raise £3.7 billion relative to the default of uprating in line with inflation every year.”

Another promising source of funding would come from reversing George Osborne’s 2016 pledge to cut corporation tax by 2020. This tax uncut could provide £5.2 billion in 2020-21 and £5.7 billion by 2022-23. Other political parties would likely support the move and the authors argue (p. 27):

“The Chancellor might also feel emboldened to act given the way in which the estimated costings of the move from 19 per cent to 17 per cent have shifted since George Osborne first announced it.”

Remarkably little was said about Brexit. Will it torpedo all the projections and render the suggestions (sorry, predictions?) unimplementable? The exception was Sarah Wollaston, who noted that she never believed the infamous £350m bus claim; she expressed reasonable worries about the effects Brexit would have.

#### What might more money mean for mental health?

Although the focus was very much top-level – where’s the money? – speakers did say a little about how it should be spent. For instance, Jon Ashworth quoted numbers on additional doctors and nurses required (it’s thousands), citing a report from IFS – also cited by Sarah Wollaston.

Reassuringly, mental healthcare was mentioned a few times as being important and in need of improvement (though note the history of “warm words”). Ben Page cited public support for increased spending, with mental health being second on the list in an April Ipsos MORI survey of priorities, after Accident and Emergency. Jon Ashworth mentioned improving support for addictions, in particular.

Sarah Wollaston cited the Health and Social Care Committee’s report into integrated care, published yesterday, which discusses detailed contractual changes needed to improve how, e.g., mental healthcare integrates with other services, including discussion of accountable care organisations (ACOs). (Perusing this report just now highlights how difficult it is to have public debates on these issues – it’s technical stuff.)

The coming weeks as we approach July 5th would be a good time to campaign for key specifics on how much money mental healthcare should receive and what it should be spent on. If the NHS received £20 billion more in 2022-23, how much should go to mental health and where?

# Mental health funding FOI responses update

Blame for insufficient mental healthcare funding has been passed around between Department of Health, NHS England, and individual Clinical Commissioning Groups (CCGs), however, the source of funding is the Treasury. Although CCGs and other mediating organisations make decisions about how much funding mental health receives, this is as a proportion of budgets decided at Treasury level. Any budgetary planning at Treasury level must therefore take mental health into consideration, alongside other areas of healthcare.

I am writing to request:

(i) names of individuals at Treasury and above, including advisors by official name or function, who are responsible for decisions made in relation to mental health care budgets;

(ii) documentation on budgetary decisions made, including evidence of how, in calculating the total health budget, mental health needs have been taken into consideration.

To (i) they said they don’t hold the information. To (ii) they said they do, but wouldn’t share it, citing Section 35 of the FOI act.

I asked the Department of Health:

CCGs and other mediating organisations make decisions about how much funding mental health receives, but this is as a proportion of budgets decided at Treasury level. Any budgetary planning at Treasury level must therefore take mental health into consideration, alongside other areas of healthcare.

I am writing to inquire about advice provided by Department of Health to Treasury on mental health budgets.

1. Who in DH provides this advice?

2. What advice has been provided to inform the most recent budget allocation for health?

They also confirmed that they held relevant information but refused to share it, citing s35(1)(a).

I asked NHS England the same question:

[…] I am writing to inquire about advice provided by NHS England to Treasury on mental health budgets.

1. Who in NHSE provides this advice?

2. What advice has been provided to inform the most recent budget allocation for health?

They provided a response.

1. Who in NHSE provides this advice?

Paul Baumann, Chief Financial Officer for NHS England, has responsibility for the organisation’s budgets including providing advice on these budgets. NHS England is an Arm’s Length Body (ALB) of the Department of Health (DH), much of the advice the Treasury would receive on Mental Health would be coordinated by the Department.

2. What advice has been provided to inform the most recent budget allocation for health?

NHS England’s view of the overall funding requirements of the NHS were set out in financial analysis conducted for the Call to Action (July 2013) [see, especially, the technical annex] and the Five Year Forward View (October 2014), which have been shared with DH and Her Majesty’s Treasury.

This analysis projects “do-nothing” expenditure using assumptions about the three main drivers associated with current health care demand and costs: demographic growth, non-demographic growth (e.g. technological development and medical advances) and health cost inflation. Historic trends for these drivers were reviewed and an estimation of future pressures developed for six service level ‘assumption sets’: Acute, Mental Health, Specialised Services, Primary Care, Prescribing and non-activity based costs. This high level analysis thus includes assumptions related to cost and demand growth for mental health services as part of the overall modelling.

Detailed analysis and costing is completed by NHS England on specific initiatives, the output of these models are used to inform budget announcements and the planning guidance information. These costings are developed by the Medical Directorate and Finance Directorate working together.

# Why can psychological therapy be helpful?

Research explaining how therapy might help is saturated with pretentious jargon, e.g., invoking “transference”, “extinction”, heightening access to “cognitive–emotional structures and processes”, “reconfiguring intersubjective relationship networks” (see over here for more).

Could simpler explanations be provided? Here are some quick thoughts, inspired by literature, discussing with people, and engaging myself as a client in therapy:

• You know the therapist is there to listen to you — they’re paid to do so — so there’s less need to worry about their thoughts and feelings. One can and is encouraged to talk at length about oneself. This can feel liberating whereas in other settings it might feel selfish or self-indulgent.
• The therapist keeps track of topics within and across sessions. This can be important for recognising patterns and maintaining focus, whilst allowing time to tell stories, meandering around past experiences, to see where they lead.
• The therapist has knowledge (e.g., through literature, supervisory meetings, and conversations with other clients) of a range of people who may have had similar feelings and experiences. So although we’re all unique, it can also be helpful to know that others have faced and survived similar struggles — especially if we learn what they tried and what helped.
• Drawing on this knowledge, the therapist can conjecture what might be going on. This, perhaps, works best if the conjectures are courageous (so a step or two away from what the clients says) — and tentative, so it’s possible to disagree.
• There can be an opportunity for practice, for instance of activities or conversations which are distressing. Practicing is a good way to learn.
• Related, there’s a regular structure and progress monitoring (verbally, with a diary, or using questionnaires). Self-reflection becomes routine and constrained in time, like (this might be a bit crude but bear with me) a psychological analogue of flossing one’s teeth.
• (Idea from Clare) “… daring to talk about things never spoken of before with someone who demonstrates compassion and acceptance; helpful because allows us to face things in ourselves that scare us and develop less harsh ways of responding to ourselves”
• The therapist has more distance from situations having an impact on someone than friends might have so, e.g., alternative explanations for interpersonal disputes can more easily be provided.
• It’s easier for a therapist to be courageous in interactions and suggestions than for a friend as — if all goes wrong — it’s easier for the client to drop out of the therapeutic relationship without long-term consequences (e.g., there’s no loss of friendship).
• Telling your story to a therapist gives you an audience who is missing all of the context of your life. Most of the context can feel obvious, until you start to tell your story. Storytelling requires explaining the context, making it explicit. For instance who are the people in your life? Why did you and others say and do the things they did? Perhaps this act of storytelling and making the context explicit also makes it easier to become aware of and find solutions.

# A farcical proposal for mental health outcomes measurement

by with Comments Off on A farcical proposal for mental health outcomes measurement

If you’re going to develop a questionnaire for something resulting in a total “score” — quality of life, feelings, distress, whatever — you’ll want all of the questions for one topic to be related to each other (as a bare minimum). This questionnaire probably wouldn’t be very “internally consistent”:

THE GENERAL STUFF QUESTIONNAIRE

1. How often do you sing in the shower?
2. What height are you?
3. How far do you live from the nearest park?

You won’t be able to do much with the result of summing answers to those together to a total score.

This one would:

THE RELIABLE FEELINGS QUESTIONNAIRE

1. How do you feel?
2. How do you feel?
3. How do you feel?
4. How do you feel?
5. How do you feel?
6. How do you feel?
7. How do you feel?
8. How do you feel?
9. How do you feel?
10. How do you feel?

However, you might wonder if questions 2 to 10 add anything… (So internal consistency isn’t everything.)

There are many ways to test the internal consistency of questionnaires, using the answers that people give. One is to use a formula by Lee Cronbach called Cronbach’s alpha. Answers run from 0 to 1. Higher is better (but not too high; see the second example above).

In England, it is now recommended (see p. 12 of Mental Health Payment by Results Guidance) to use scores on a “Mental Health Clustering Tool” to evaluate outcomes. I think there are at least two problems with this:

1. It’s completed by clinicians. It’s unclear if service users even get to know how they have been scored, never mind to what extent they can influence the process.
2. The questionnaire scores aren’t internally consistent.

The people who proposed the approach write (see p.30 of their report): “As a general guideline, alpha values of 0.70 or above are indicative of a reasonable level of consistency”. Their results: 0.44, 0.58, 0.63, 0.57. They also refer to previous studies showing that this would always be the case, due to “its original intended purpose of being a scale with independent items” (p. 30). So, by design, it’s closer to the General Stuff Questionnaire above: a list of “presenting problems” to be read individually.

Not only are clinicians deciding whether someone has a good outcome (are they really in the best position to decide?), but the questionnaire they’re using to do so is rubbish — as shown by the very people proposing the approach!

Undergraduate psychology students wouldn’t use a questionnaire this poor in their projects. Why is it acceptable for a national mental health programme?

# Did You Used to be R.D. Laing? (Full Documentary)

by with Comments Off on Did You Used to be R.D. Laing? (Full Documentary)