NADD Bulletin Volume X Number 4 Article 2

Complete listing

An Evaluation Tool to Assess Treatment Effectiveness and Optimize Outcomes for Individuals with Dual Diagnosis

Amy Nemirow, Ph.D.
Carol Persons, M.D.
Dina McFalls, M.S.


The foundation for good treatment for people with a dual diagnosis is a clear description of target symptoms, which leads to accurate diagnosis and targeted treatment.  Optimal treatment of psychiatric symptoms contributes to the best outcomes for people with mental retardation who also cope with mental illness.  But how do we determine and document whether an individual is receiving optimal treatment for their mental illness?  What is the best way to evaluate the match between symptoms and diagnosis, and the match between diagnosis and treatment?  Is there a way to assess whether this information is accurately described and effectively communicated among residential, behavioral and medical supporters?  And if it is determined that the information is there and is being shared with all team members, does this lead to better outcomes for individuals and their teams?

Philadelphia Coordinated Health Care (PCHC) has developed an evaluation tool to assess the behavioral health supports being provided to individuals with co-occurring mental retardation and mental illness.  It is our belief that this simple checklist can help teams determine whether they are documenting information effectively which can lead to better team communication, better treatment, and ultimately improved outcomes for the people they support. 

Details that Lead to Optimal Outcomes

The underpinning for the tools PCHC has created is good psychiatric treatment based upon accurate description of target symptoms, accurate data collection, clarity in ruling out medical or environmental causes, and good team communication.  The main underlying tenet of good treatment is a match between the target symptoms and the diagnosis, between the diagnosis and the treatment, and between the reduction of symptoms and treatment.  Without accuracy and match between symptoms, diagnosis and treatment, the team is in danger of medicating behavior, overmedicating, and mistreating.  However, with accuracy in target symptom description, improved accuracy in diagnosis can occur, and therefore, treatment outcomes improve as well. 

The key to accurate description of target symptoms is good team communication.  When teams work to identify the root cause of any behavior, ruling out medical issues, pain, trauma, genetic issues, as well as those environmental factors that are so common within the mental retardation service delivery system, then clarification of target symptoms can occur.  Only after the team is clear about target symptoms versus behavior, can accurate data be collected that will inform the treating physician about the effectiveness of any treatment undertaken. 


Background, Part I:  The Team Review Form


In 1985 the Court approved a settlement agreement in  regard to Pennhurst Class Members referred to as the “1985 Decree” (Terri Lee Halderman et al. v. Pennhurst State School and Hopsital et al. Civil Action No. 74-1345).  This settlement was seen throughout the nation as recognition that persons with mental retardation have the right to live and receive care in the community.  In March 1994 the Court held the Commonwealth of Pennsylvania and the County of Philadelphia in contempt for violating their obligations under the 1985 Decree.  Philadelphia County then spent years developing numerous programs and plans  to rectify their noncompliance with the 1985 Decree.  As part of those programs and plans, it was required to certify the medical necessity for the prescription of psychotropic medication(s) for all individuals with a Dual Diagnosis (Mental Retardation and Behavioral Health Disorder).  It was also thought prudent to address this concern in regard to best practice as well as meeting licensing requirements.

A work group was formed to meet this challenge.  The group consisted of agency representatives, behavior specialists, program specialists, nurses, and a physician.  It was decided that by using a form, many of the problem areas in this field could be overcome.  This form would meet the criteria for court order compliance, as well as provide the information required for adequate monitoring of medications.  When the form was complete it was reviewed by local psychiatrists  who were involved in working with many individuals with dual diagnoses.  The form was then adopted and approved, with the requirement that psychotropic medications be reviewed minimally every 90 days.  This 3-part form meets both court and residential licensing needs, and has been approved by treating psychiatrists and behavior specialists.  Recently the form was revised and updated.  It is referred to as the Team Review Form.  This form is now a widely used documentation tool for psychiatric treatment in our region.

What does the Team Review Form Look Like?

The Team Review Form consists of 3 pages, one for each of the main areas of supports for individuals with a dual diagnosis: home and medical supports, behavioral supports, and psychiatric supports. It lists both medical and psychotropic medications, diagnoses, target symptoms, and health concerns on the first page; data regarding frequency of target symptoms and comments from behavior specialists on the second page; and physician documentation of medication changes on the third page; and by having team members fill out all 3 pages and share them with other team members, it is also a way to document that the information is being shared with the entire team. (Copies of the Team Review Form are available from the authors).  The Team Review Form, in bringing together symptom, diagnoses, and treatment information and in enhancing team communication, represents PCHC’s model of treatment (McFalls et al., 2006).  Our model has been developed based on previous models that emphasize the clear description of psychiatric symptoms (Sovner & Hurley, 1992); distinguishing symptoms of mental illness from behaviors that serve other functions (Lowry & Sovner, 1991); and treating clinicians collaborating with residential/family teams (Silka & Hauser, 1997).

How do we know the Team Review Form Works?


PCHC collects data from the Team Review Forms to document and review behavioral health supports.  For example, all individuals in the Pennhurst class who take psychotropic medication must use the form; in Philadelphia county, all of those forms are then submitted to PCHC and reviewed by a psychiatrist, who sends feedback back to the agencies so that they can make corrections or implement suggested changes.  Data from these reviews over the past 10 years indicate that more accurate documentation of psychiatric symptoms and diagnoses can lead to more accurate and effective treatment of mental illness.  Risk management data from the southeast region of Pennsylvania show a reduction in the use of physical restraint over the last few years. We believe, although we do not have empirical support for it, that one of the reasons for this is that when mental illness is managed effectively and behavioral health supports are optimal, then relationships improve, challenging behavior decreases, alternative coping strategies can be developed, and restraints are not needed.

Background, Part II:  Embreeville


In the Southeast Region of Pennsylvania a second protected class of individuals  from a state center, Embreeville, was created by a class action lawsuit undertaken against the Commonwealth in 1993 (United States, et al. vs. Commonwealth of Pennsylvania, Civil Action No. 93-CV-2094), and a later Stipulated Agreement between the parties in 2000 (United States, et al. vs. Commonwealth of Pennsylvania, Stipulated Agreement, Order of March 21, 2000).  As part of the basis for the Stipulated Agreement, a team of independent reviewers hired by the U.S. Department of Justice reviewed several areas of health care supports and found them to be in need of improvement.  A psychologist (Yando, 1999) and a psychiatrist, (Un, 1999), and in a later review another psychologist (Yount, 2002), each reviewed samples of Embreeville class members who demonstrated behavioral issues.  The findings of the reviewers are detailed in Table 1.

Table 1.

§Behavioral Health (Yando, 1999):

§Incomplete reporting/documentation&ldots;

§Medications-behavior relationships rarely defined

§Poor quality data collection, especially related to behaviors

§Medications-side effects, psychopharmacological literature not present

§Behavior support plans lacking

§Treatment options limited

§Teaching programs few in number, deficient in design

§Scant evidence that individual’s habilitation is meaningfully integrated

§Psychiatric (Un, 1999):

§Overall improvement noted in move to community homes

§Improvement needed in data collection

§Plans should include functional goals

§Increase core competency for behavior consultants, staff, including skill training

§Behavior management committees need access to psychiatrist/physician re: treatment decisions

§Psychiatrist to review and monitor medications

§Staff training in psychiatric disorders and functioning

§Health Risk Assessment tool helpful, but limited in psychiatric/behavioral scope—recommend regular review by psychiatrist/behavioral psychologist

§Psychology (Yount, 2002)

§Most records did not contain medication reduction plans

§Few records contained complete documentation regarding medication reviews

§Little evidence data based decision making

§Confusing information concerning medications and/or diagnoses, including different diagnoses on different documents

§Direct communication or consultation between a class member’s behaviorist and psychiatrist found in only a few instances

§Communication with the psychiatrist regarding behavioral data limited to anecdotal info conveyed by direct support staff

§Lack of communication among and between team members, including the psychiatrist and psychologist, suggests an inability on the part of the team to create an acceptable annual plan.


In the Special Report to the parties Regarding Compliance with the Stipulated Agreement, issued in June of 2002, it was recommended that, “ a review of all Embreeville class members who receive psychotropic medication [take place to] evaluate the efficacy of their behavior services” (Records, 2002).

In order to satisfy the court order, and to assist the residential providers who support these individuals, PCHC, at the direction of the Commonwealth of Pennsylvania’s Southeast Regional Office of Mental Retardation, undertook a survey to specifically review the behavioral health supports of the individuals in the Embreeville class who routinely received psychotropic medication.  Based on information documented in the Team Review Form, PCHC developed a tool to review behavioral health supports in two areas: psychiatric care and behavioral supports.  From a 40-item checklist, we identified the 10 items (5 psychiatric and 5 behavioral) which, if flagged as areas of concern, could potentially pose the greatest risk to the individual. These 10 items were then used to generate a Risk Assessment Score (RAS) for each individual reviewed.

Embreeville Review Instrument and Risk Assessment Tool

The independent reviewer in 2002 noted that the Team Review Form was not being used consistently.  We took the information from that form to develop a 40-item supports assessment checklist (23 items for psychiatry; 17 items for behavioral health) to review the documentation of the 90 individuals (out of a total of 166) from the Embreeville class who were taking psychotropic medication. Copies of the supports assessment checklist are available from the authors.  

From the 40-item checklist we selected the 10 items (5 from psychiatry and 5 from behavioral health) that would pose the greatest risk to the individual if not addressed effectively.  These were items that we were familiar with from previous experience with another protected class of people, and that reflect the treatment model and philosophy of team supports that we promote.  For example, medically, we emphasize a good match among target symptoms, diagnoses, and medication; medication reduction plans; and side effects screenings.  We also look for team communication, as evidenced by a timely and accurate flow of information.  And in the behavioral realm, it is crucial to have data collection that can be used to test the effectiveness of treatment.  It is our experience that gaps in these areas can result in less than optimal treatment and increased risk to the individual.  The resulting Risk Assessment items are presented in Table 2.

Table 2.   

Psychiatric items:

Medication is not appropriate to the diagnosis

Diagnosis is not justified by the individual's symptoms or symptoms are not clear or specific

Dosage of medication is not appropriate

Side-effects are not screened or addressed

Allergies Section - no response.

Behavioral Health items:

Symptoms of the person’s psychiatric diagnosis are not listed or not clearly defined in the Behavior Support Plan.

The relationship between psychiatric symptoms and challenging behaviors has not been clarified or addressed.

The record does not reflect ongoing communication among the team members, including the residential staff, the Behavior  Specialist and the prescribing physician

The direct care staff do not understand the connection between symptoms, diagnosis, medications

There have been incidents reported for this individual related to behavioral health issues (e.g., psychiatric hospitalization, restraint, etc.)


For both the psychiatric and the behavioral items, if 3 or more (out of 5) items are flagged, an additional point is added on, so the highest possible score would be 12.  This number became our Risk Assessment Score.   The highest Risk Assessment Score during our first set of reviews was 9.  We determined that scores between 6 and 9 constituted “High Risk,” scores of 4 or 5 were labeled as “Moderate Risk,” and scores between 1 and 3 were “Low Risk.”        

Review Procedure

For the first round of Embreeville Reviews, the meetings took place at the offices of the respective county’s Mental Retardation department.  The individuals were not required to attend, but the teams, including Supports Coordinators, Agency Nurses, Behavior Specialists, and residential Program Specialists, were requested to attend and to bring medical, psychiatric and behavioral documentation regarding the individuals being reviewed.  The review sessions, conducted by PCHC’s Psychiatrist and Behavioral Health Specialist, involved: an introduction and explanation of the concerns regarding compliance with the court order and the importance of optimizing supports for individuals with a dual diagnosis; a review of documentation, specifically the TRF and the Behavior Support Plan; discussion with the team about what we found and how it might relate to how the individual was doing or the challenges the teams might be struggling with at the time; verbal recommendations for how supports might be improved; and the opportunity for team members to ask questions and get clarification from us.  The review sessions were followed up with written reports that included recommendations from both the Psychiatrist and the Behavioral Health Specialist, in four areas: Individual (recommendations regarding clinical issues); Team (recommendations regarding team communication or documentation); Agency (recommendations concerning agency policies around behavioral health supports and their documentation); and Training (recommendations for training that would be helpful to the team in understanding the individual’s dual diagnosis and needs for support).  Risk Assessment Scores were determined after the reviews and sent to the teams along with the reports and recommendations.  Teams could then contact PCHC for more information or to schedule training.

A second round of reviews was conducted using the same procedure approximately one year later.  Only the teams whose Risk Assessment Scores were in the High and Moderate Risk categories were reviewed again.  The second round of reviews was conducted at the office of PCHC.

Prior to conducting a third round of reviews, PCHC was asked by the Commonwealth’s Southeastern Regional Office of Mental Retardation to determine how the behavioral health supports being provided to Embreeville class members compared with those provided to non-class members.  For this reason our third round of reviews included class members whose Risk Assessment Scores continued to be High or Moderate in the second round, and also included non-class individuals selected by the residential providers of the class members already being reviewed. 

A fourth round of reviews is currently in progress.  The fourth round will include the five class members whose Risk Assessment Scores were in the High and Moderate ranges in the third round, non-class members selected by the counties, and non-class members selected by the regional Offfice of Mental Retardation. 


Data from Round I:  As noted above, 90 individuals were reviewed in the first round.  Of these, 55 scored in the High or Moderate risk range.  The average Risk Assessment score for Round I was 4.4.  Of the 55 people who scored in the High and Moderate ranges, the average score was 5.9.   A total of 23 residential provider agencies participated in these reviews.  The 90 individuals reviewed had a total of 32 psychiatrists and 27 Behavior Specialists.  Thirty-one out of the 90 individuals did not have Behavior Specialist supports.  

Data from Round II:  In the second round of reviews, 54 of the 55 people who scored in the High and Moderate range in the first round were reviewed again using the same format.  Forty-two of the 54 (78%) had their risk level reduced to Low or No Risk level in Round II.  The average Risk Assessment score for Round II was 2.4, an almost 50 per cent reduction.  In this round there were 21 provider agencies, 28 psychiatrists, and 20 Behavior Specialists.  The number of individuals who did not have a Behavior Specialist was 18.   

Data from Round III:  Scores continued to decrease in the third round. Nine of the 12 Embreeville class members whose Risk Assessment scores were high or moderate in the second round were re-reviewed in the third round.  For these individuals, their average Risk Assessment Score for Round II was 5; in Round III their average score decreased to 3, an overall improvement bringing them from moderate to low risk.  However, for the nonclass individuals, who had not been reviewed previously and whose supports are not under the same court-ordered level of scrutiny, the Risk Assessment scores were slightly higher.  A total of 14 non-class members were reviewed; the average Risk Assessment Score for these individuals in the third round was 6.  A summary of the results of all three rounds of reviews is presented in Table 3.

Table 3.


Embreeville Dual Diagnosis Reviews: Results   

 Class# ReviewedHighModerateLowNo Risk   


I 903322314   

II 54753012   

III 93240   

Chosen by agency for Round IIINon-Class147250 

Table 1.  Summary of Risk Assessment Scores for all 3 rounds of reviews  



What caused the 50% reduction in risk scores?

Results of our reviews suggest several possible reasons for the reduction in risk scores over time.  Increased attention to behavioral health supports for these individuals may have resulted in better team communication, which leads to more effective treatment and more efficient documentation.  For example, increased attention to accurate identification of psychiatric target symptoms, and increased data collection and reporting, can help to promote clearer communication and better supports.   And increased attention to the match between psychiatric symptoms and diagnosis, and between diagnosis and treatment, would also improve treatment effectiveness. 



The results of our Embreeville Review process thus far seem to indicate that the documentation of psychiatric and behavioral supports that is used in our region and the tools we developed through our reviews, are useful for providers in two ways: they can help to streamline documentation and promote effective team communication; and they can be used as quality management tools to assess the quality of supports and optimize behavioral health outcomes for people with mental retardation and mental illness.

The Embreeville Review Process and the resulting training and attention to clinical issues, saw a significant decrease in the at-risk levels of the individuals involved regarding psychiatric and behavioral health concerns.  The Risk Assessment Tool that was developed is helpful in identifying areas of concern to be addressed by teams either clinically or through staff training.  The tool needs to be validated using other data; for example, determining whether the restraint and/or psychiatric hospitalization frequency for these individuals also decreased during these time periods.  More investigation is underway.


Records, T. (2002).  Special Report to the Parties Regarding Compliance with the Stipulated Agreement, United States et al. v. Commonwealth of Pennsylvania, Order of March 21, 2000.

Terri Lee Halderman et al. v. Pennhurst State School and Hospital et al. Civil Action No. 74-1345.  March 28, 1994. 

Un, H. (1999).  Report of Site Visits, Embreeville Providers, Pennsylvania

United States, et al. v. Commonwealth of Pennsylvania. Civil Action No. 93-CV-2094. April 21, 1993.

United States, et al. v. Commonwealth of Pennsylvania, Stipulated Agreement, Order of March 21, 2000.

Yando, R. (1999).  Report of Site Visits, Embreeville Providers, Pennsylvania.

Yount, R. (2002).  Psychology Findings. In Records, T (2002), Special Report to the parties Regarding Compliance with the Stipulated Agreement.