Performance
  Systems
    Analysis

What We Can Learn From Evidence-Based Practice1

Eileen Gambrill2

T

he purposes of this paper are twofold: 1) to suggest what we can learn from evidence-based practice (EBP) and related enterprises about reducing gaps among evidentiary, application and ethical concerns and 2) to encourage the cultivation of working relationships with those involved in EBP and applied behavior analysis. Integrating evidentiary, ethical and application issues has been an uphill battle in the helping professions (e. g., Chalmers, 1983; Jarvis, 1990; Silverman, 1998). Consider the experience of Ignaz Semmelweiss who, around 1840, discovered that the death rate of mothers from childhood fever markedly decreased if surgeons washed their hands before delivering babies. Cleanliness was not taken seriously by the medical profession until decades later. Skinner has also addressed this concern (e. g., Skinner, 1981). Baer (1998) recently incisively discussed contingencies that pose an obstacle to careful program evaluation.

The Journal of Applied Behavior Analysis and The Behavior Analyst have included articles regarding this challenging issue (e. g., Bailey, 1991; Foxx, 1996(a)). Some authors suggest that lack of use of methods found to be effective is mainly due to the language we use (Lindsley, 1991). Encouraging professionals to consider practice-related research findings when making important clinical decisions and to honor other ethical obligations such as informed consent requirements described in professional codes of ethics (NASW, 1996) is an on-going concern. Evidence-Based Practice (EBP) and related enterprises such as the Cochrane and Campbell Collaborations provide an evolving source of tools and ideas for doing so, as well as colleagues who share this interest. The key goal of applied behavior analysis as well as EBP is to closely link evidentiary, ethical and application concerns. Both enterprises represent a scientific approach in which claims are rigorously tested. Encouraging collaboration may have the following advantages: 1) provide additional encouragement for rigorous testing of practice-related claims (e.g., regarding effectiveness, validity of assessment measures); 2) provide additional routes for disseminating practice-related research findings; 3) offer opportunities for collaboration; and 4) increase opportunities for doing more good than harm. Harms resulting from premature diffusion of untested methods are amply illustrated in the histories of the helping professions (e. g., Jacobson, Mulick, & Schwartz, 1995; Sharpe & Faden, 1996; Silverman, 1997).

Evidence-Based Practice (EBP)

Evidence-based medicine (EBM) arose as an alternative to authority-based medicine in which decisions are based on criteria such as consensus among experts, anecdotal experience and tradition. It is hoped that professionals who consider research findings regarding decisions and inform clients about them, will provide better, more ethical care than those relying on authority. As the term "evidence-based" has become more popular, it is often misused, that is, methods or reviews that are not evidence-based are described as evidence-based (e. g., Gambrill, 2003). Readers of research will have to be discerning in regard to the rigor of critical appraisal of research reports and accuracy of related claims (e.g., of effectiveness). Given the many misperceptions and misrepresentations of EBP (e.g, Gibbs & Gambrill, 2002; Straus & McAlister, 2000) it is important to describe the process of EBP in order to gain its potential benefits to clients and professionals. The definition of EBP used here originated in the medical and health area (e. g., Sackett, et al., 1996, 1997). It describes a series of steps and related challenges for using clinical expertise to integrate the best external evidence based on research findings, with information about a client's unique characteristics and circumstances and preferences and actions in a context of limited resources (Haynes, Devereaux, & Guyatt, 2002). It is an evolving process designed to attend to interrelated evidentiary, ethical, and implementation concerns. Evidence-based practice (EBP) "is the integration of best research evidence with clinical expertise and [client] values" (Sackett, Straus, Richardson, Rosenberg, & Haynes, 2000, p. 1). Clinical expertise refers to the use of practice skills and past experience to rapidly identify each client's unique circumstances and characteristics, "their individual risks and benefits of potential interventions, and their personal values and expectations" (p. 1). It is drawn on to integrate information from these varied sources (Haynes, Devereaux, & Guyatt, 2002). Client values refer to "the unique preferences, concerns and expectations each [client] brings to an ... encounter and which must be integrated into ... decisions if they are to serve the [client]" (p. 1). EBP requires locating and critically reviewing research findings related to important practice and policy decisions and sharing what is found (including nothing) with clients. Steps include the following:

  1. converting information needs related to practice decisions into four-part answerable questions including description of client, recommended method, alternative and hoped-for outcome
  2. tracking down, with maximum efficiency, the best evidence with which to answer them
  3. critically appraising that evidence for its validity, impact (size of effect) and applicability (usefulness in practice)
  4. applying the results of this appraisal to practice/policy decisions. This involves integrating this critical appraisal with clinical expertise and with a client's unique characteristics and circumstances. It involves deciding whether evidence found (if any) applies to the decision at hand (e.g., is a client similar to those studied, is there access to services described), and considering client values and preferences in making decisions as well as other application concerns
  5. evaluating our effectiveness and efficiency in carrying out steps 1-4 and seeking ways to improve them in the future (adapted from Sackett et al., 2000, p. 4).

Some authors define EBP much more narrowly: "We use evidence-based practice here primarily "to denote that practitioners select interventions based on their empirically demonstrated links to the desired outcomes" (Rosen & Proctor, 2002). This definition leaves out local implementation concerns, other kinds of practice decisions (e.g., selecting assessment measures), the importance of considering client values and expectations and involving clients as informed participants. EBP draws on the results of systematic, rigorous, critical appraisal of research related to important practice questions such as "Is this assessment measure valid?"; "Does this intervention do more good than harm?" Efforts are made to prepare comprehensive, rigorous reviews of all research related to questions addressed. Different questions require different kinds of research methods to critically appraise proposed assumptions (e.g., Gibbs, 2003; Gray, 2001a; Greenhalgh, 2000; Guyatt & Rennie, 2002; Sackett et al., 2000). Kinds of questions include the following:

Reasons Sackett and his colleagues (2000) suggest for the rapid spread of EBM include four realizations made possible by five recent developments. Realizations include 1) practitioner need for valid information about decisions they make, 2) the inadequacy of traditional sources for acquiring this information (e.g., because they are out-of-date, frequently wrong, overwhelming in their volume, variable in their validity), 3) the gap between assessment skills and clinical judgement "which increase with experience and our up-to-date knowledge and performance which decline" (p. 2) and; 4) lack of time to locate, appraise and integrate this evidence (p. 2). For example a variety of biases in published research (submission, publication, methodological, abstracting and framing biases) compromise the accuracy of available material (e.g., Gray, 2001). Economic interests of the pharmaceutical industry and fraud also diminish accuracy of material (e.g., Bekelman, MPhil, & Gross, 2003). Sackett and his colleagues (2000) suggest that five developments have allowed improvement in this state of affairs:

  1. "The development of strategies for efficiently tracking down and appraising evidence (for its validity and relevance)
  2. The creation of systematic reviews and concise summaries of the effects of health care (epitomized by the Cochrane Collaboration).
  3. The creation of evidence-based journals of secondary publication ...
  4. The creation of information systems for bringing the forgoing to us in seconds.
  5. The identification and application of effective strategies for life-long learning and for improving our clinical performance" (p. 3).

EBP is not a method to save money. It consists of using services that "maximize the quality and quantity of life for individual [clients]; this may raise rather than lower the cost of care" (Sackett et al., 1997, p. 4). Sackett and his coauthors argue that EBP is not "old hat" (p. 3). They argue that the belief that everybody already is doing it "falls before evidence of striking variations in both the integration of patient values into our clinical behavior and in the rates with which we provide interventions to our patients" (Sackett et al., 1997, p.3). Indeed, the more one reads about current-day practices in the helping professions, the clearer it is that helping efforts do not have characteristics of EBP as envisioned by its originators. Many (most?) practitioners do not search for external research findings related to important practice decisions. Applied behavior analysts call attention to neglect of effective programs (e. g., Baldwin, 1999). Literature in social work suggests that social workers do not draw on practice-related research findings to inform practice decisions (see e.g., Rosen, 1994; Rosen, Proctor, Morrow-Howell, & Staudt, 1995). Not keeping up with new research findings related to important practice decisions renders our knowledge increasingly out of date. As a result, decisions may be made that harm rather than help clients (e.g., see Jacobson, Mulick, & Schwartz, 1995). In addition, many helpers do not honor requirements for informed consent (e.g., see Braddock, Edwards, Hasenberg, Laidley, & Levinson, 1999).

What We Can Learn From Evidence-Based Practice

First, we can learn that we are not alone in our struggle to encourage professionals to draw on practice-related research and to guide clinical decisions by ongoing, accurate, problem-related data. For example, aims and activities of the Center for Evidence-Based Dentistry can be seen in Figure 1. We can get ideas about how to disseminate knowledge regarding applied behavior analysis more successfully as suggested by the remarkable success of EBP reflecting the ideas discussed later (see Figure 2). Some may think that medicine differs too much from other helping professions such as psychology, social work, or counseling to make valuable comparisons. A key advantage in medicine is the availability of signs (e.g., blood pressure) as well as symptoms (e.g., feeling anxious). However, the similarities outweigh the differences. Consider uncertainty; medical experts suggest that the typical physician works in an atmosphere of uncertainty.

Highlight Variations in Practices

One of the origins of EBM was the study of variations in practices and related outcomes (Gray, 2001). Variations in practices suggest questions such as "Are they all equally effective?" "Are some more effective than others?" "Do some result in more harm than good?" (see also Wennberg, 2002).

Explore Promising Formats for Professional Education

We can add to our skills in using problem-based learning in professional education programs by drawing on related research in medicine (e.g., Evidence-based medicine, 2000 by Sackett and his colleagues). We can explore educational formats developed in EBP, such as problem-based learning (PBL) in which participants learn how to pose answerable questions related to important clinical decisions (e. g., regarding effectiveness, accuracy of assessment measures) and to access and critically appraise related research. Problem-based learning has been explored in medical education for almost 30 years (e. g., Barrows, 1994). A key advantage is provision of multiple opportunities for self-directed learning drawing on external evidence related to a particular problem and related hoped-for outcomes. Helping students to acquire effective self-directed learning skills that contribute to life-long learning is a key aim and students are encouraged to draw on related research findings in making practice decisions. The goal is to produce practitioners who are effective, efficient, and humane in addressing clients' concerns and who learn in the process (Barrows, 1994, p. 5). Students are assigned to groups of 5-7. Each group is given a tutor. Education is problem-focused from the very beginning of medical school. Students are encouraged to come up with different possible causes of a problem and to identify what information they need to choose among different views and what particular underlying processes are related to the problem (e.g, physiological). This links problem solving to basic subjects such as physiology.

Students are asked to identify clear learning objectives (called "educational prescriptions") related to each problem addressed (Sackett, et al., 1997, 2000). These relate to questions raised in the problem analysis phase (e.g., to gain more knowledge of the processes related to a problem). It is in this step that students relate problems to basic curricular domains (e.g., physiology). For example, students could receive scores of learning opportunities during their education in asking: What do we know about behavior and how it is developed and maintained that can be of value with this problem? A key concern in this stage is to identify learning resources that might supply needed information. For example, a problem may require an understanding of certain developmental trajectories. Decisions are then made about objectives to focus on and, if necessary, students agree on task distribution. Students collect additional information outside the group and share what has been found, synthesizing and testing the newly acquired information, and supplementing this knowledge as needed and correcting it as necessary. Reviews of the effectiveness of PBL differ in their estimates of its effectiveness compared to traditional educational programs (e. g., Albanese, 2000; Albanese & Mitchell, 1993; Colliver, 2000).

We can learn how to prepare user friendly books for professionals such as Evidence-based medicine (1997, 2000) that has been so popular. In addition to superb content, appealing features of this book include: 1) size (convenient for slipping in a pocket), 2) clear engaging writing, 3) use of icons, 4) engaging use of terms such as "educational prescription" (referring to what students must find out to problem solve), and 5) helpful teaching tips.

Create Evidence-Based Organizations

We can draw on work in EBP to increase options for evidence-based policies and administrative practices including openness to peer review and arranging feedback so we learn from mistakes. Gray (2001) provides many interesting ideas for developing audit systems that encourage integration of practice and research and for evidence-based purchase of services. We can take the lead from EBP in candidly discussing limited resources and the importance of considering populations as well as individuals (e. g., Eddy, 1996; Gray, 2001). As Eddy (1996) suggests, no matter what economic system of health care we have, resources will be limited. Hard choices will have to be made and should be carefully and explicitly considered.

Increase Access to Practice and Policy-Related Research Findings

Evidence-based practice is closely tied to the Cochrane Collaboration (CC). The CC is a world-wide network of centers that prepare, maintain and disseminate high-quality systematic reviews of the effects of health care (Chalmers, Sackett, & Silagy, 1997). It was founded in 1993 in Oxford, England by about 80 people from a variety of countries. Organizational units of the CC include Cochrane Centers and collaborative review groups. Reviews are entered on the Cochrane Database of Systematic reviews which is available by subscription. Dissemination takes place through a regularly updated electronic journal called The Cochrane Database of Systematic Reviews. In 1972, Archie Cochrane published Effectiveness and efficiency drawing attention to our ignorance about health care and suggested randomized controlled trials (RTCs) as a guide to sound use of resources. The CC evolved in response to his emphasis on systematic, up-to-date, critical reviews of all RTCs in health care. Reviewers describe what they searched for, where they searched, how they searched, and what criteria were used to critically appraise studies found. Reviews are updated regularly and electronically disseminated. The Cochrane Collaboration starts with individuals--health professionals, consumers and researchers--with interests in a particular health problem or group of problems. A question might be: How can people who want to give up smoking be helped?

"The Cochrane Collaboration's potential derives from its commitment to prepare and maintain reviews of research evidence which address questions of relevance to people using the health services; its use of transparent methods in attempts to minimize biases; and its openness to challenge. As a scientific enterprise it has at least two features which are rare if not unique. Firstly, the protocols of Cochrane reviews (that is, information about the Collaboration's "research in progress") are routinely made available for public scrutiny and comment. Secondly, the Collaboration has established a system for incorporating new evidence in systematic reviews prospectively, and improving or correcting them when ways of doing so are identified ..." (Chalmers, Sackett, & Silagy, 1997, pp. 236-237).

There are many Cochrane Centers throughout the world that try to ensure that people who have registered an interest with the Collaboration are placed in contact with other like-minded individuals (see Figure 3). The aim of review groups is to prepare definitive reviews based on randomized controlled trials in relation to a particular treatment. An intensive effort is made to locate all randomized controlled trials, published and unpublished in all languages, worldwide, related to a question using hand searches. Research review protocols are also available on the Cochrane Database. Software developed to expedite efforts include Review Manager.

"Cochrane Review Groups (CRGs) make particular use of two software packages. The first--Review Manager--provides both an organizational and analytic framework for assembling Cochrane Reviews in electronic format; the second--Module Manager--enables administrators and editors to assemble up-to-date reviews prepared by the members of their CRG, as well as information about the CRG itself, for example, the scope of its work, and the strategy used to assemble a specialized register of relevant studies."

The Campbell Collaboration was initiated in 1998. This focuses on questions regarding criminal justice and social welfare. It is patterned on the Cochrane Collaboration. Applied behavioral analysts could nurture ties with the Campbell and Cochrane Collaborations and participate in review groups.

We can also learn from the Cochrane Collaboration how to get research findings into the hands of consumers that will allow them to make informed choices. The two volume book describing research related to obstetrical care prepared as a result of Archie Cochrane's review had little effect on the everyday practice of gynecologists. Enkin and his colleagues then prepared a book for laypeople entitled A guide to effective care in pregnancy & childbirth(1995). This describes the following:

  1. Beneficial forms of care-effectiveness demonstrated by clear evidence from controlled trials.
  2. Forms of care likely to be beneficial. (The evidence in favor of these forms of care is not as firm as for those in category 1.)
  3. Forms of care with a trade-off between beneficial and adverse effects. (These effects should be weighed according to individual circumstances and priorities.)
  4. Forms of care of unknown effectiveness. (There are insufficient or inadequate quality data upon which to base a recommendation for practice.)
  5. Forms of care unlikely to be beneficial. (The evidence against these forms of care is not as firm as for those in category 6.)
  6. Forms of care likely to be ineffective or harmful. (Ineffectiveness or harm demonstrated by clear evidence.)

This book has been popular and is in its second edition. Behavioral associations could take a more active role in producing such books describing the evidentiary status of various services in relation to certain outcomes. These will help both providers and consumers of professional services to be more informed. They should encourage people to ask questions such as, "Are there harmful side effects of this service? Increasing access to practice and policy related research findings increases transparency regarding the evidentiary status of claims. They should also take more initiative in preparing books for consumers describing application of behavioral principles to certain areas such as parent training. Toilet training in a day (1974) by Azrin and Foxx illustrates a successful book integrating research and practice. We should publish more such books.

Increase Critical-Appraisal Skills of Both Clients and Professionals

Increasing transparency about the uncertainty associated with given decisions and of what is done to what effect, is a key hallmark of EBP and this is pursued in many ways including offering critical appraisal workshops for both practitioners and clients in which they learn how to critically appraise different kinds of research (e. g., Altman, et al., 2001; Greenhalgh, 2001; CASP; www.cochraneconsumer.com). Bandolier, available on the Internet, also illustrates honest brokering of knowledge and ignorance. A justification approach (seeking support for a claim) in contrast to a falsification approach (seeking to falsify guesses about what may be true by critical tests, Popper, 1972), encourages inflated claims about "what works" that may result in harm to clients (e. g., Silverman, 1997; 1998).

Increase Options for Involving Clients As Informed Participants

A notable feature of EBP is the concern for client interests and involvement of consumers of health services as active participants in decision-making. Consumers have access to Cochrane Collaboration internet communication networks via which they can raise questions and give comments. Attention to the values and expectations of clients is a key hallmark of EBP (Sackett et al., 1997; 2000). Related literature in EBP offers many ideas for promoting such involvement including evidence-informed client choice forms (Entwistle, et al., 1998), user-friendly brochures (Holmes-Rovner, et al., 2001), and interactive decision making tools (e. g.,Edwards & Elwyn, 2001; O'Conner, et al., 2002). Evidence-informed patient choice (EIPC) (see Figure 4) entails three criteria: 1) the decision involves which health care intervention or care pattern a person will or will not receive; 2) the person is given research-based information about effectiveness (likely outcomes, risks and benefits) of at least two alternatives (which may include the option of doing nothing); and 3) the person provides input into the decision-making process (Entwistle, et al., 1998). A concern for involving clients in making decisions that affect their lives highlights the importance of informed (in contrast to uninformed or misinformed) consent. This concern is also emphasized in applied behavior analysis in its attention to social validity-- the acceptability of goals, methods and outcomes to clients and significant others (see Baer, Wolf, & Risley, 1968; Schwartz & Baer, 1991). Another means of involving clients is including them in the critical appraisal of research including reviews of research findings in a given area. Consumer involvement may reveal that outcomes of concern to clients are not addressed (Hanley, et al., 2001).

Highlight Ethical Obligations of Professionals and Researchers

EBP highlights ethical obligations to clients in many ways as suggested in the sections above, for example involving clients as informed participants and honest brokering of knowledge and ignorance. Such involvement provides opportunities to help clients acquire critical appraisal skills that increase their ability to consider the uncertainty associated with decisions. Avoiding harm is another ethical obligation. The descriptions of variations in services and related outcomes, including harm, emphasizes our obligation to avoid harm when possible. Evidence-based practice originated in part because of inflated claims in the professional literature and out-of-date material (Gray, 2001). That is, there is a keen interest in accurately describing the evidentiary status of claims. We can be more careful in this regard as suggested by Allen and Polaha (2003) in their review of Treatments that work (2001) by Christophersen and Mortweet.

Discover Obstacles to Integrating Practice and Research

Considerable attention is devoted in the EBP literature to discovering application problems. These include obstacles such as client beliefs, authoritarian organizational cultures, and limited local resources. We may obtain promising ideas from literature in EBP about how to minimize or avoid such obstacles.

What Behavior Analysts Can Offer

We can offer empirically based principles of behavior that can guide the analysis of contingencies related to behaviors of interest such as use of evidence-based practices and policies (Figure 5). We can offer guidelines for conducting descriptive and functional analyses regarding problem-related contingencies (including those related to encouraging practitioners to consider practice-related research when making decisions) and for gathering on-going data regarding outcomes (e. g., Bergan & Kratchowill, 1990; Carr et al., 1994; Foxx, 1996(a), 1996(b); Goldiamond, 1984; Schwartz & Baer, 1991; Van Houten & Axelrod, 1993). Behavior analysts use data to guide decisions. This is one way in which behavior analysis is evidence based. We can ask for example: "What is the evidence that a neutralizing activity will decrease the rate of problem behaviors?" and investigate the results (Horner, Day, & Day, 1997). We can offer an echoing voice encouraging accurate rather than inflated claims regarding what has been rigorously tested to what effect.

In Conclusion

Forging working relationships with those involved in EBP and the Cochrane and Campbell Collaboration should contribute to the pursuit of shared goals such as basing practice decisions on the best-available evidence and involving clients as informed participants. We can join our compatriots in other disciplines and professions in striving to involve clients as informed participants in decisions that affect their lives and to decrease the practice-research gap (see Figure 6). Both applied behavior analysis and EBP are radical approaches in recognizing contingencies that others may prefer to remain hidden, for example poor quality parent education programs offered to biological parents involved with the child welfare system. Providing programs that maximize the likelihood of success would be considerably more costly. Although money would be saved by not paying for ineffective services. The suggested collaboration should benefit all involved parties including clients. EBP highlights variations in services and related outcomes including harm and related publications promote rigorous appraisal of practice-related claims. We can join others who value honest brokering of knowledge and ignorance. Highlighting pseudoscience in the guise of science is becoming more common (e. g., Lilienfeld, Lynn, & Lohr, 2003). We can learn from those who have been more successful in promoting the value of their methods. Use of similar terms in lay language and behavioral terminology (e. g., "punishment") has been noted as a unique obstacle regarding applied behavior analysis. The success of EBP in medicine and health suggests that it is not just language that gets in the way of drawing on external research findings related to practice and policy decisions.

References

Albanese, M. (2000). Problem-based learning: why curricula are likely to show little effect on knowledge and clinical skills. Medical Education, 34(9), 729.

Albanese, M. A. & Mitchell, S. (1993). Problem-based learning: A review of literature on its outcomes and implementation issues. Academic Medicine, 68(1), 52-81.

Allen, D. A., & Polaha, J. (2003). Bridging the disconnection between applied research and practice: A review of treatments that work. Journal of Applied Behavior Analysis, 36, 141-146.

Altman, D. G., Schulz, K. F., Moher, D., Eger, M., Davidoff, H, Elbourne, D., et al. (2001). The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Annals of Internal Medicine, 134, 663-694.

Azrin, N. H., & Foxx, R. M. (1974). Toilet training in less than a day. New York: Simon Schuster.

Baer, D. M. (1998). Program evaluation: Arduous, impossible, or political? Paper presented at the 16th Annual Conference of the Northern California Association for Behavior Analysis, Oakland, CA.

Bailey, J. S. (1991). Marketing behavior analysis requires different talk. Journal of Applied Behavior Analysis, 24, 445-448.

Barrows, H. S. (1994). Practice-based learning: Problem-based learning applied to medical education. Springfield, IL: Southern Illinois University School of Medicine.

Bekelman, J. E., MPhil, Y. L., Gross, C. P. (2003). Scope and impact of financial conflicts of interest in biomedical research. Journal of the American Medical Association, 289, 454-465.

Bergan, J. R. & Kratchowill, T. R. (1990). Behavioral consultation and therapy. New York: Plenum.

Braddock, C. H., Edwards, K. A., Hasenberg, N. M., Laidley, T. L., & Levinson, W. (1999). Informed decision making in outpatient practice. Time to get back to basics. Journal of the American Medical Association, 282(24), 2313-2320.

Campbell Collaboration. http://campbell.gseupenn.edu

Carr, E. G., Levin, L., McConnachie, G., Carlson, J. I., Kemp, D. C., & Smith, C. E. (1994). Communication-based intervention for problem behavior: A user's guide for producing positive change. Baltimore: Paul H. Brookes.

Center for Evidence Based Medicine. Armory Bldg., University of Exeter, England. EX4 4RJ. Fax: 01392 263324. E-mail: S.E.Bosley@exeter.ac.uk Director: Brian Sheldon.

Chalmers, I. (1983). Scientific Inquiry and Authoritarianism in perinatal care and education. Birth, 10(3), 151-155.

Chalmers, I., Sackett, D., & Silagy, C. (1997). The Cochrane Collaboration. In A. Maynard & I. Chalmers (Eds.), Non-random reflections on health services research: On the 25th anniversary of Archie Cochrane's effectiveness and efficiency (pp. 231-249). London: BMJ.

Christophersen, E. R., & Mortweet, S. L. (2001). Treatments that work: Empirically supported strategies for managing child behavior problems. Washington, DC: American Psychological Association.

Cochrane, A. (1972). Effectiveness and efficiency: Random reflections on health services. London: Nuffield Provincial Hospitals trust..

Cochrane Centre, UK. NHS R & D Program, Summertown Pavilion, Middle Way, Oxford, OX2 7LG, U.K. E-mail address: general@cochrane.co.uk. FAX: +44 - (0) 1865-516311 Telephone: +44 - (0) 1865 - 516300

Colliver, J. A. (2000). Effectiveness of problem-based learning curricula: research and theory. Academic Medicine, 75, 259-266.

Critical Appraisal Skills Program (CASP). P.O. Box 777, Oxford, England OX3 7LF. E-mail: casp@cix.co.uk, See also their Website: http://www.ihs.ox.ac.uk/casp/

Eddy, D. M. (1996). Clinical decision making: From theory to practice. Sudbury, MA: Jones & Bartlett Publishers.

Edwards, A. & Elwyn, G. (Eds.) (2001). Evidence-based patient choice: Inevitable or impossible? New York: Oxford University Press.

Enkin, M., Keirse, M. J. N., Renfrew, M., & Neilson, J. (1995). A guide to effective care in pregnancy & childbirth (2nd Ed.). New York: Oxford University Press.

Entwistle, V. A., Sheldon, T. A., Sowden, A., & Watt, I. S. (1998). Evidence-informed patient choice: Practical issues of involving patients in decisions about health care technologies. International Journal of Technology Assessment in Health Care, 14, 212-225.

Evidence-Based Medicine Website. (See for example Learning Resources Center. Tool-kit.)

Foxx, R. M. (1996). Translating the covenant: The behavior analyst as ambassador and translator. The Behavior Analyst, 19, 147-161.

Foxx, R. M. (1996). Twenty years of applied behavior analysis in treating the most severe problem behavior: Lessons learned. The Behavior Analyst, 19, 225-235.

Gambrill, E (2003). Evidence-based practice: Sea change or the emperor's new clothes? Journal of Social Work Education, 39, 3-23.

Gibbs, L. E. (2003). Evidence-based practice for the helping professions. Pacific Grove, CA: Thompson/Brooks/Cole.

Gibbs, L. & Gambrill, E. (2002). Evidence-based practice: Counterarguments to objections. Research on Social Work Practice, 12, 452-476.

Goldiamond, I. (1984). Training parent trainers and ethicists in nonlinear analysis of behavior. In R. F. Dangel & R. A. Polster (Eds.), Parent training: Foundations of research and practice (pp. 504-546). New York: Guilford.

Gray, J. A. M. (2001). Evidence-based health care: How to make health policy and management decisions (2nd Ed.). New York: Churchill Livingstone.

Gray, J. A. M. (2001). The origin of evidence-based practice. In A. Edwards & G. Elwyn (Eds.), Evidence-informed client choice (pp. 19-33). New York: Oxford University Press.

Greenhalgh, T. (2001). How to read a paper. London: BMJ.

Guyatt, G. & Rennie, D. (2002). Users' guide to the medical literature: A manual for evidence-based clinical practice. The Evidence-Based Medicine Working Group JAMA & Archives. Chicago, IL: American Medical Association Press.

Hanley, B., Truesdale, A., King, A., Elbourne, D., & Chalmers, I. (2001). Involving consumers in designing, conducting, and interpreting randomised controlled trials: Questionnaire survey. British Medical Journal, 322, 519-523.

Haynes, R. B., Devereaux, P. J., & Guyatt, G. H. (2002). Clinical expertise in the era of evidence-based medicine and patient choice. [Editorial]. ACP Journal Club, March/April, 136:A11, pp. 1-7.

Holmes-Rovner, M., Llewellyn-Thomas, H., Entwistle, V., Coulter, A., O'Connor, A., & Rovner, D. R. (2001). Patient choice modules for summaries of clinical effectiveness: A proposal. British Medical Journal 322, 664-667.

Horner, R. H., Day, M. M., & Day, J. R. (1997). Using neutralizing routines to reduce problem behaviors. Journal of Applied Behavior Analysis, 30, 601-614.

Jacobson, J. W., Mulick, J. A., & Schwartz, A. A. (1995). A history of facilitated communication. American Psychologist, 50(9), 750-765.

Jarvis, W. T. (1990). Dubious dentistry: a dental continuing education course. Loma Linda University, School of Dentistry, Loma Linda, CA 92350.

Lindsley, O. R. (1991). From technical jargon to plan English for application. Journal of Applied Behavior Analysis, 24, 449-458.

McIntyre, N., & Popper, K. (1983). The critical attitude in medicine: The need for a new ethic. British Medical Journal, 287, 1919-1923.

National Association of Social Workers. (1996). Code of ethics. Silver Spring, MD: Author.

O'Conner, A. M., Stacey, D., Rovner, D., Holmers-Rovner, M., Tetroe, J., Llewellyn-Thomas, H., et al. (2002). Decision aids for people facing health treatment or screening decisions. [Cochrane Review]. The Cochrane Library Issue 4, Oxford: Update Software.

Popper, K. R. (1972). Conjectures and refutations: The growth of scientific knowledge (4th Ed.). London: Routledge & Kegan Paul.

Proctor, F. & Rosen, A. (2000). The structure and function of social work practice guidelines. Paper presented at Conference on Developing Practice Guidelines for Social Work Intervention. Washington University. In press in A. Rosen & E. Proctor (Eds.), Developing practice guidelines for social work intervention: Issues, methods, and research agenda. New York: Columbia University Press.

Rose, S., Bisson, J., & Wesseley, S. (2001). Psychological debriefing for preventing post traumatic stress disorder (PTSD) [Cochrane Review]. The Cochrane Library. Issue 4. Oxford: Update Software.

Rosen, A. (1994). Knowledge use in direct practice. Social Service Review, 68, 561-577.

Rosen, A., Proctor, E. E., Morrow-Howell, N., & Staudt, M. (1995). Rationale for practice decisions: Variations in knowledge use by decision task and social work service. Research on Social Work Practice, 15, 501-523.

Sackett, D. L., Rosenberg, W. M., Gray, J. A., Haynes, R. B., & Richardson, W. S. (1996). Evidence based medicine: What it is and what it isn't. British Medical Journal, 312, Jan. 13, 71-72.

Sackett, D. L., Richardson, W. S., Rosenberg, W., & Haynes, R. B. (1997). Evidence-based medicine: How to practice and teach EBM. New York: Churchill Livingstone.

Sackett, D. L., Straus, S. E., Richardson, W. C., Rosenberg, W., & Haynes, R. M. (2000). Evidence-based medicine: How to practice and teach EBM (2nd Ed.). New York: Churchill Livingstone.

Sarnoff, S. K. (2001). Sanctified snake oil: The effect of junk science on public policy. Westport, CT: Praeger.

Schwartz, I. S. & Baer, D. M. (1991). Social validity assessments: Is current practice state of the art? Journal of Applied Behavior Analysis, 24, 189-204.

Sharpe, V. A. & Faden, A. I. (1998). Medical harm: Historical, conceptual, and ethical dimensions of iatrogenic illness. New York: Cambridge University Press.

Silverman, W. A. (1997). Equitable distribution of the risks and benefits associated with medical innovations. In A. Maynard and I. Chalmers (Eds.), Non-random reflections on health services research. London: BMJ.

Silverman, W. A. (1998). Where's the evidence?: Debates in modern medicine. Oxford: Oxford University Press.

Skinner, B. F. (1981). We happy few, but why so few? Invited address to the Association for Behavior Analysis, Milwaukee, WI. May.

Straus, S. E. & McAlister, P. A. (2000). Evidence-based medicine: A commentary on common criticisms. Canadian Medical Journal, 163, 837-841.

Van Houten, R. & Axelrod, S. (Eds.) (1993). Behavior analysis and treatment. New York: Plenum.

Vincent, C. (Ed.) (2001). Clinical risk management: Enhancing patient safety. London: BMJ.

Wennberg, J. E. (2002). Unwarranted variations in healthcare delivery: Implications for academic medical centers. British Medical Journal, 325, 961-964.

          Figure 1The PDF icon means that this file requires the Adobe Acrobat Reader for viewing

          Figure 2The PDF icon means that this file requires the Adobe Acrobat Reader for viewing

          Figure 3The PDF icon means that this file requires the Adobe Acrobat Reader for viewing

          Figure 4The PDF icon means that this file requires the Adobe Acrobat Reader for viewing

          Figure 5The PDF icon means that this file requires the Adobe Acrobat Reader for viewing

          Figure 6The PDF icon means that this file requires the Adobe Acrobat Reader for viewing

* Denotes that the web page is from a site other than behavior.org.
[PDF] or The PDF icon means that this file requires the Adobe Acrobat Reader for viewing Denotes that the page requires the Adobe® Acrobat® Reader for viewing.
For more information on Adobe Acrobat Reader.

1Paper presented at the Annual Conference of the Association for Behavior Analysis, San Francisco, CA, May 25, 2003.

2School of Social Welfare, 120 Haviland Hall — 7400, University of California at Berkeley, Berkeley, CA 94720-7400. Tel: (510) 642-4450; Fax: (510) 643-6126; Email: gambrill@uclink4@berkeley.edu

Return Home

About CCBS

Membership

Links

Store

Behavior Analysis

Computer Modeling
of Behavior

Aging Gracefully Autism Behavioral Safety
 Book Reviews
 Commentaries
 Education
Everyday Life
 Parenting
 Organizational Behavior Management Pets & Animals
 Verbal Behavior Virtual Community

Copyright ©1997-2008 by the Cambridge Center for Behavioral Studies.
All rights reserved.

Feedback or questions about the
Cambridge Center for Behavioral Studies or our website?
Contact our webmaster, Rebekah Pavlik or
our Executive Director, Dr. Dwight Harshbarger.

The Cambridge Center for Behavioral Studies Publication Office
is located at the following address:
336 Baker Avenue
Concord, Massachusetts U.S.A. 01742-2107

Telephone: (978) 369-CCBS (2227)
Facsimile: (978) 369-8584

Visit other sites through the Behavior Analysis Webring:
[ Previous 5 Sites | Previous | Next | Next 5 Sites | Random Site | List Sites ]