There is considerable current enthusiasm for evidence-based health care in general and mental health care, but there is a current debate on how evidence-based practice can be consistently implemented in routine settings both in the UK29 and abroad.30 31
Evidence-based practice is the delivery of interventions for which there is strong scientific evidence that they improve relevant patient outcomes. Although the type of scientific evidence does vary, the gold standard for treatment outcome is the RCT. Where several trials exist they can be considered together through meta-analysis. Knowledge concerning evidence-based practice accrues through the accumulating results of efficacy and effectiveness studies.
Thus the purpose of evidence-based practice is (a) to ensure that the wealth of research evidence informs clinical practice so that those who are in receipt of treatment will receive the treatment that is the best available and represents the current knowledge base, and (b) to ensure that planning and policy is determined by empirical evidence, for those purchasing services to be able to make informed choices and for those receiving services to be empowered by such knowledge. Furthermore, the establishment of an evidence-based practice knowledge base of what works allows the practice of mental health services and individual clinicians to be compared to the evidence base. This increases accountability and establishes guidelines for good practice and improves the quality of mental health services. Limitations in evidence also set the research agenda for the future.
There are, however, critics of the collation of data for evidence-based practice. This mainly focuses around the use of specific meta-analytic techniques that have very limited entry criteria. The Cochrane database, for example, provides valuable searches and evaluations of randomised control trials with strict criteria for entry. Although the evidence may be strong for a particular practice it may be based on a very small number of studies. The main criteria for exclusion are the lack of randomisation of the participants within the trial and the lack of data on all those participants who entered the trial. Although clearly the results of such trials should be less weighted in the final evaluation, such information may be valuable when few other data are available.
Efficacy trials are devised to test whether the therapy has an effect overall on the outcomes of interest. They are carried out in relatively controlled environments, usually by sophisticated university based research teams, and often involve highly expert therapists. For CBT trials the outcomes of interest are a reduction in overall psychotic symptoms, reductions in relapse or reduced rates on admission to hospital, reduced psychopathol-ogy and improvements in functioning. These trials may also include various control groups and process measures to help understand why the treatments work. An effectiveness trial attempts to more closely resemble the real world of routine services, inclusion criteria are wider so the sample treated is more heterogeneous and includes the atypical patients, and the therapists are recruited from the routine services. The measured outcomes are reduced to the minimum and tend to be gross measures that are clinically significant such as relapse or hospital admission; health economic measures to assess cost are also desirable. In special cases an equivalence trial may be designed, in which a new treatment is expected to match the clinical efficacy of an established treatment but may have other benefits, for example in terms of acceptability or cost. These trials have special methodological features that distinguish them from simple comparative trials.32
Figure 18.3 shows how the patient flow in a study should be described. The box of particular interest is the one at the very top that describes those who have been assessed for eligibility. In order to prevent bias in recruitment the best method for ascertaining samples of patients for a trial is to recruit them from a cohort of patients in contact with a service that covers a geographic area (as in Tarrier et al.15 and Lewis et al.17). This ensures that the people who are in the trial do represent those who have the disorder. In the UK it is largely assumed that those patients with schizophrenia in contact with the services will represent those with the disorder requiring clinical intervention. For example, Tarrier et al.15
screened all patients who might have a diagnosis of schizophrenia in a number of NHS trusts, selected those who achieved predetermined criteria and examined their notes further. All putative candidates following this procedure were interviewed to ascertain whether they satisfied the entry criteria. This method has been used as a gold standard and other trials of CBT have used the data from Tarrier et al.15 to compare with their study sample in order to conclude that their sample was representative (see Sensky et al.16). What a comparison of samples allows is just that-if the samples are similar then the results of the trials can be usefully compared, but this information cannot be used as evidence of sample representativeness.
Convenience samples which recruit from clinic attenders or, even more problematically, patients referred to the project by their clinicians are at risk of selection bias. The referrer may only select those possible participants who they view as good candidates for the treatment or conversely patients who are difficult or treatment refractory. Recruitment of referred patients is unfortunately the norm.1416 Even though it may be possible to compare the recruited sample to the whole population of patients who may be eligible in terms of socio-demographic and clinical service contact, this will not be enough data to rule out a systematic bias. In the treatment of panic disorder, Klein33 34 has argued vigorously that in comparisons of psychotherapy verses drug, a pill placebo-drug comparison is necessary to ensure that the sample is not atypical since the efficacy of the drug (in this case imipramine) is well established. This is largely an argument about how representative or typical any sample is, given a reliance on convenience samples.
There are a number of different factors that need to be considered as part of the recruitment process such as service delivery system, academic support, socio-economic status of the area and geography (urban, suburban and rural area). It is unlikely that these will have a specific interaction with the outcome from therapy, but as these factors will affect the generalisation of the trial results it is probably important for the sample to represent a variety.
But ethnicity and cultural mix may potentially affect therapy outcomes. As we know very little about how to target psychological therapy to different cultural groups, it seems reasonable to start investigating a new treatment with a culturally homogeneous group and in later trials modify to accommodate cultural diversity, if such modification would be a requirement of effectiveness in cultural subgroups.
In psychological therapies, especially in the field of psychosis, there has been a dilemma about whether to adopt medical diagnosis as entry criteria to studies. Some clinical psychologists (e.g.
Bentall et al.35) would prefer the adoption of symptomatic entry criteria as schizophrenia is a term covering a group of people with a wide variety of abnormal experiences. So some trials have as their entry criteria a specific symptom experienced as distressing rather than membership of a single diagnostic category.1136 However, even in these studies some patients were excluded on the basis of diagnosis because of not fulfilling other criteria (see below), and it was certainly the view of one of the authors (TW) that in feasibility studies of group CBT some patients with diagnoses other than schizophrenia, e.g. personality disorder or bipolar affective disorder, did not respond in similar ways to the patients with diagnoses of schizophrenia or schizoaffective disorder. Current CBT studies have generally included patients from the schizophrenia spectrum and it is certainly the view of some CBT therapists that the type of therapy offered to people with bipolar affective disorder is different from that designed for schizophrenia.37
Even when diagnosis is used there are too many different systems to choose from (e.g. clinical case note diagnosis, research diagnoses (RDC), DSMIV, ICD10, etc.). The choice of a different system will change the characteristics of the sample. For instance, if people are drawn on RDC criteria they will not necessarily be as chronic as those fulfilling the DSMIV criteria.
As well as criteria for inclusion into trials most studies also exclude people on the basis of specific issues. In trials of psychological therapy for psychosis one usual criterion is that the people who enter the trial are those whose symptoms have remained despite adequate doses of medication. The group chosen on this basis is extremely chronic and refractory and provides an extremely stringent test of the efficacy of psychological treatment.
A further thorny issue is that of co-morbid substance abuse. Most studies will exclude individuals when the abuse is severe, but the criteria for severity are rarely set out clearly so that it is impossible to compare between trials. Patients who are recruited from inner city areas are unlikely to be free of recreational drug use. A small consumption of cannabis may not affect the therapy efficacy, but it is not clear whether any use of class A drugs affects the therapeutic effect of psychological treatment. A more recent trial has been designed to test the efficacy of CBT and family intervention to treat dual diagnosis patients (those diagnosed as suffering from schizophrenia and substance abuse) in which the substance abuse is thought to increase the risk of poor outcomes in the primary disorder.18 In this case severe substance abuse was an entry criterion.
Again some people may have a co-morbid organic condition such as epilepsy that may warrant exclusion, although most trials again would evaluate whether the organic condition is primarily responsible for the symptoms of the disorder which they are trying to alleviate. Deteriorating brain disorders such as Alzheimer's disease may be a reasonable exclusion criterion as CBT relies on the carry-over of changes in one session to subsequent sessions. Similarly, people who have learning disabilities may also have some difficulties with CBT as it is currently devised, although therapists have extended treatment for depression to the learning disabilities field. Current trials also do not support the idea that lower IQ prevents therapeutic changes.38 But all current trials do have a lower cut-off for IQ, usually around 65.
Two main issues affect the inferences about the trial results. The first is the effect of those people who drop out of the therapy and the second is those people who are lost to contact at any stage of the trial. Different systems of dealing with drop-outs can be adopted. Some systems assume that the person would not have changed at all since leaving the trial (LOCF), but this approach has its problems.39 But assuming that the group who drop out would have performed in the same way as those who remained also produces difficulties. Drop-outs may be those people who might never have achieved any change following therapy. Clearly if a treatment produces high levels of drop-outs this might imply something about the acceptability of treatment. A precise description of drop-outs is required but, from the trials submitted so far this is missing in all but a few cases.
More research on drop-outs is clearly required. But in the area of mental health in particular, the research is difficult, if not impossible, to carry out. The new guidelines on research governance40 do not allow for the harassing of people who have dropped out of trials for their reasons for dropping out or for data on their current health status. However, it is not only of interest academically, as it provides some information on the veracity of the theory underlying the disorder, but also essential to inform the health services. For example, Tarrier et al.41 reported that patients who dropped out of treatment tended to be male, unemployed and unskilled, single, with a low level of educational attainment and a low pre-morbid IQ. They had a lengthy duration of illness although at the time of discontinuation they were not severely ill and functioned at a reasonable level. They were likely to be paranoid but not suspicious of the therapist. They were unlikely to be grandiose. They did not understand the rationale for therapy or the potential for benefit but feared it could make them worse.
It is not clear whether it is appropriate or ethical to collect personal information that is kept for routine monitoring purposes for a person who has dropped out of a trial. This information may consist of health service contacts kept on health care databases such as case notes as well as information from third parties. For trials involving people with severe disorders, third-party information from key workers is nearly always included as part of the measurement of outcome. The lack of data on drop-out may affect the relevance and benefit of the trial results to the wider community. It may therefore be unethical not to collect as much of it as possible. The interpretation of new legal rights such as the new UK Human Rights Act should make the position of researchers clearer, but it is also possible that the idiosyncratic interpretations made by local
Research Ethics committees will lead to further confusion in this already complicated area.
Was this article helpful?