NADD Bulletin Volume I Number 6 Article 2

Complete listing

Efficacy of a Curriculum on Dual Diagnosis for Direct Care Personnel

Cynthia S. Mester M.A., L.C.P.C.

Research has shown that individuals with mental retardation and other developmental disabilities are vulnerable to the full range of psychiatric disorders to which persons within the general population, or persons who do not have developmental disabilities, are vulnerable (Galligan, 1990; Menolascino, Wilson, Golden, & Ruedrich, 1986; Reid, 1982; Sovner & Hurley, 1983). Although statistics reported in various studies conducted over the past 20 years vary, most studies report a significantly high rate of mental illness among those who are mentally retarded. For example, in the Isles of Wight study conducted by Rutter, Tizard, Yule, Graham, and Whitmore (1976), the data reported from teacher ratings and parental ratings suggested that the prevalence of mental illness among individuals with an IQ of less than 70 was 42% and 30%, respectively. This was five times the rate of what was reported for a sample of peers who were not mentally retarded.

A longitudinal study conducted by Matson and Frame (1986), which included all individuals with mental retardation born during a five year period in one city, found the rate of psychiatric disorders within the sample of individuals who were mentally retarded to be 2.5 times greater than the rate found within the sample of individuals who were not mentally retarded forming the matched control group. A review of the literature conducted by Corbett (1985) revealed that the reported rates of mental illness in samples of children and adolescents who were diagnosed as severely mentally retarded (with IQ scores ranging from 20 to 40) varied from 13% to 50%. More recent research which has sought to “standardize” methods of assessing and diagnosing psychiatric disorders in persons with mental retardation suggests that individuals who are developmentally disabled may be at a 40% or higher risk for developing a psychiatric disorder than individuals who are not developmentally disabled (Loschen, 1997; Menolascino & Fleisher, 1993; Silka & Hauser, 1997).

These stated statistics underscore the tremendous need to address the presence of psychiatric disorders in people who are developmentally disabled. Identifying and accurately treating psychiatric disorders in persons with developmental disabilities are particularly difficult due to the fact that many of these individuals are nonverbal or unable to communicate internal states. Information necessary for understanding problems being experienced by people with mental retardation is often communicated to mental health professionals by secondary sources called direct care personnel. Direct care personnel are those who are employed to work daily and directly with individuals who are developmentally disabled in order to help them achieve the tasks of daily living and maintain personal safety. Training in dual diagnosis, or in the coexistence of mental retardation and mental illness, is particularly necessary for direct care personnel in order to ensure that what they report to mental health professionals on behalf of persons with mental retardation is accurate, objective, and complete.

This author, in collaboration with three other faculty members at Southern Illinois University School of Medicine, has written a curriculum on dual diagnosis to be used by the State of Illinois in its curriculum training package for direct service workers. This curriculum called “Psychiatric Disorders in Individuals with Developmental Disabilities: A Curriculum for Direct Service Personnel” seeks to provide direct care personnel with the knowledge and abilities needed to ensure that the identification, assessment, diagnosis, and treatment of persons with mental retardation who may be suffering from a mental illness are conducted and carried out according to the most accurate, objective, and appropriate means possible. Using a Posttest-Only Control Group Design, the usefulness of this curriculum is being assessed by determining whether sixty randomly assigned direct care personnel who participated in a two-day training on Dual Diagnosis performed on a test created to measure knowledge and problem solving skills related to dual diagnosis significantly better than sixty randomly assigned direct care personnel who did not receive the two-day training on Dual Diagnosis.

Content, Format, and Development of APsychiatric Disorders in Individuals with Developmental Disabilities: A Curriculum for Direct Service Personnel

As stated previously, the curriculum “Psychiatric Disorders in Individuals with Developmental Disabilities: A Curriculum for Direct Service Personnel” seeks to provide direct care workers with the knowledge and abilities needed to ensure that the identification, assessment, diagnosis, and treatment of persons with mental retardation who may be suffering from a mental illness are conducted and carried out according to the most accurate, objective, and appropriate means possible. Based on this, the curriculum was divided into five chapters that contain information relevant to achieving each of the following objectives: (See Chart 1)

Since the objectives serve to illustrate the type and breadth of information contained within the curriculum and since the curriculum is over 350 pages in length, a discussion of the specific content is beyond the scope of this paper; although, development of the curriculum merits further discussion.

Content of the curriculum was based on the following: a comprehensive review of the empirical literature pertaining to developmental disabilities, mental illness, dual diagnosis, and curriculum development; a consensus from focus group meetings attended by experts working in the fields of developmental disabilities and mental health; and a sample of job descriptions for direct service personnel working in the State of Illinois. After the overall goal of the curriculum was determined and the literature and sample job descriptions were reviewed, seven experts, including two psychiatrists, three social workers, and two masters-level psychologists, met approximately one time per week for four consecutive months to further define and refine the content. Beyond the substantive matters, defining and refining the content also focused on the format and the grammar and sentence structure used to convey the content since the curriculum needed to correspond to a level conducive to the audience for whom it was being written.

In the sample of job descriptions reviewed, being hired for a direct care position required a prospective employee to have a high school diploma or GED. In addition, the Illinois Department of Public Health Requirements for Long Term Care (1990) required a person being hired for a direct service position to evidence an 8th grade level of education. Based on this and the need to be neither too basic nor too technical, the curriculum was written to correspond to a tenth grade level of reading. To ensure that this was being accomplished, readability evaluations were conducted on the curriculum as it was being written and after it was completed. Readability levels for the final, completed version illustrated that the content ranged between a 5th grade and a 12th grade level of reading, with the majority of material corresponding to a 10th grade level of reading. To substantiate these levels and to ensure a user-friendly presentation of information, the completed version was reviewed by a Certified Teacher credentialed in the area of Learning Disabilities and experienced in curriculum development.

The completed version of the curriculum consisted of two manuals, a Trainer’s Manual and a Participant’s Manual, since the curriculum was written to incorporate the problem-based method of learning. This method uses sample problems to achieve the following: encourage an active and interactive style of learning; convey information learners need to know; and teach skills (i.e., problem solving, reasoning, critical thinking, and self-directed learning skills) learners need to use and develop. Although this method has proven most effective when a trainer or tutor facilitates the exchange of information within a group of learners, the advantages of this method can still be realized on an individual basis and in a purely written form since the Trainer’s Manual contained some basic design elements: (See Chart 2)

An individual learner would use the Trainer’s Manual since this manual contains both the information to be learned and the prompts to encourage and model (via written language) the skills to be used and/or developed in solving the problems presented. A learner within a group would use the Participant’s Manual since this manual contains the information to be learned, but it does NOT contain the problem-based learning prompts that are to be offered by the trainer who is facilitating the group and using the Trainer’s Manual. In essence, the Participant’s Manual was written to serve as a RESOURCE for learners as they engage in self-directed learning and problem-solving efforts since libraries, the internet, human resources/experts, etc., may not be readily available to direct care workers participating in the problem-based learning group process.

The purpose of the problem-based method is to enable learners to identify the following: Hypotheses, Facts, Learning Issues, and Action Plans (Barrows, 1992). When this takes place within a group, an active and interactive style of learning is encouraged since the roles and responsibilities for the trainer/tutor and learner are clearly and strictly defined. The trainer’s role is to act as a facilitator and not as an expert who lectures about facts. The trainer’s responsibility is to guide learners in (1) identifying the objectives of their learning, (2) developing ideas or hypotheses about a problem, (3) evaluating and analyzing information, and (4) identifying areas of learning they want to further explore. The trainer accomplishes this by consistently and constantly asking “why” and “how” questions of learners. Finally, the trainer helps learners identify a plan for their learning and a plan to resolve the problem(s) presented.

The learner’s role is to not rely on the trainer for expertise and to become a self-directed learner. In a group situation, the learner has the responsibility to share information obtained from his/her self-directed learning with other members in the group who in turn share theirs. This group learning process allows each person an opportunity to contribute to resolving or addressing the problem, and it prevents any one individual from being expected to know all the information needed to accomplish a solution. In addition, it increases the amount of information available to each individual. Within this group process, the learner also has a responsibility to evaluate the information obtained (from both his own self-directed learning efforts and from the self-directed learning efforts of group members) and to evaluate the magnitude and quality of perceived learning (including his own learning and his perceptions of the group’s learning).

This specific method of education is being used in medical schools, high schools, accounting and nursing professions, and other types of educational settings throughout the world (Barrows & Kelson, 1995). The benefit of this method is that an individual can learn how to learn and to not rely on traditional methods of teaching which are dependent on the presence of a given expert at a particular time. The responsibility for learning is shared, and organizations using this teaching method can rely on trainers and various staff who have been trained in this method to facilitate the training and informational exchange for larger numbers of personnel. Organizations may then feel more comfortable in conveying the expectation that personnel are responsible for directing their continuing education since the skills to do so successfully may have been adequately engendered within an individual who has participated in the problem-based method of learning.

In sum, the curriculum “Psychiatric Disorders in Individuals with Developmental Disabilities: A Curriculum for Direct Service Personnel” was designed in a manner that promotes learners to use and develop problem solving, critical thinking, clinical reasoning, and self-directed learning skills as they are exposed to information about dual diagnosis. The format of the curriculum incorporates the problem-based method of learning and an evaluative approach. Throughout the curriculum, learners are prompted to evaluate themselves as learners, as providers of services to individuals with a dual diagnosis, and as employees working within the systems of mental health and developmental disabilities. Information contained within the curriculum centers on principles and practices that would seek to improve the quality of life for individuals who have a developmental disability and a psychiatric disorder.

The extent of information a learner needs to achieve the goals and objectives of the curriculum is contained within both the Trainer’s Manual (individual basis) and the Participant’s Manual (group basis), and the extent of information a trainer needs to conduct the two-day curriculum delivery is contained within the Trainer’s Manual. The Trainer’s Manual was designed to accommodate quality teaching and learning even when the curriculum is delivered by someone who may not have a great deal of experience in the problem-based method, dual diagnosis, and/or curriculum delivery since the words, language, and prompts needed to understand and accomplish each of these are contained within the Manual. This dual-manual system and problem-based method supports a “train the trainer” approach since a learner who has participated in the two-day curriculum delivery can then use the Trainer’s Manual to facilitate training of other personnel within his/her organization. To ensure that the Trainer’s Manual actually included all information needed to conduct the two-day curriculum delivery and to obtain feedback from persons for whom the curriculum was written, the curriculum was pilot tested in May of 1997.

This author and three other faculty members at Southern Illinois University School of Medicine presented the curriculum in a two-day training at the Department of Mental Health and Developmental Disabilities in Springfield, Illinois. Participants in the training were thirty direct care staff working throughout Central Illinois in various agencies responsible for providing vocational and/or residential services to persons with developmental disabilities. At the end of the two day experience, participants were asked to complete an evaluation form that addressed issues pertaining to their perceptions of the curriculum’s content, format, problem-based learning approach, and overall usefulness. The curriculum was rated very highly (receiving only “4” and “5” responses on a five-point Likert scale where “5” corresponded to “excellent”) and judged to be very useful in teaching about dual diagnosis and the problem-based method of learning. Participants identified nothing within the curriculum that needed to be changed, and they endorsed the content and format to be more than appropriate in enabling the goals and objectives of the curriculum to be achieved.

Although feedback was obtained from participants, the feedback had focused on an evaluation of the curriculum that was based on the subjective perceptions of the participants. In order to obtain objective data pertaining to the effectiveness of the curriculum, a test designed to measure knowledge and problem solving skills related to dual diagnosis was created. Using a Posttest-Only Control Group design, this test was used to address the following research questions:

1. What is the estimated reliability of test scores?

2. Do people who participate in a two-day training using the curriculum entitled “Psychiatric Disorders in Individuals with Developmental Disabilities: A Curriculum for Direct Service Personnel” perform on a test designed to measure knowledge and problem-solving skills related to dual diagnosis significantly better than people who do not participate in the two-day training?

3. Do test scores vary based on direct care workers reported (a) years of education, (b) length of time worked in the field of developmental disabilities, and (c) number of trainings, workshops, or in services related to the field of developmental disabilities participated in over the most recent past two years?

Sample

Participants for this research were obtained through the completion of three general steps. First, three large community agencies, one in each of Northern, Central, and Southern Illinois, were selected based on the degree of how representative they were of all agencies employing direct service personnel in Illinois. Secondly, directors of these agencies were contacted via phone and provided with a summary of the training curriculum and an explanation about the research study to be conducted. Lastly, if directors were interested in paying their personnel to receive training and/or participate in the research study, they were asked to generate and mail to this author a list containing names of personnel who had volunteered to participate (i.e., interested personnel volunteered via a sign-up sheet that was passed around at a weekly staff meeting after the director had explained the research and training to them). Since this research incorporates a Posttest-Only Control Group design, one-half of the names comprising each list from each organization were randomly assigned to the experimental group and one-half of the names from each organization’s list were randomly assigned to the control group. Approximately forty subjects, twenty for the experimental group and twenty for the control group, were solicited for participation within each of the three geographical areas for a total of 120 prospective subjects.

Each of the 120 subjects were asked to record demographic information relating to age, job title, gender, work setting, employer, educational level, length of time working within the field of developmental disabilities, previous trainings on dual diagnosis, and previous job-related trainings over the most recent past two years. All subjects were also asked to complete a test designed to measure knowledge and problem solving skills related to dual diagnosis. In addition, subjects comprising the experimental group were asked to complete an evaluation form pertaining to the two-day training experience. Although the trainings and research have been conducted, the Procedures and Results of this full-scale study were presented at the NADD Annual Conference on November 5, 1998, and not presented within the context of this paper since all of the statistical results have not yet been rendered.

Instrument Development

Since this research is based on the use of a test that was specifically created for the purpose of this study, it is necessary to explain how the test was developed. After reviewing the empirical research related to test development, training evaluation, and learning outcomes and based on the nature of the curriculum and consultations with a certified teacher working at the junior high school level, the first draft of the test was written to incorporate two formats. A junior high teacher was chosen as a person with whom to consult since the minimum educational requirements for a direct care worker is an eighth grade education; therefore, the presentation of information needed to be tailored to coincide with the developmental abilities of the prospective test takers. Two, as opposed to one or three, formats were chosen for the design of the test since some test takers may be predisposed to do better, irrespective of the actual content, based on the use of one particular format and since some test takers may be overwhelmed by more than two methods. The two methods chosen were multiple choice and essay.

Twenty-eight questions about dual diagnosis and a corresponding choice of four possible answers are presented in the multiple choice section, and these questions specifically measure factual knowledge addressed in the content of the curriculum. The essay/performance-based section consists of a case example about Ms. Y and substantive questions about her functioning, and these questions specifically measure problem-solving and critical thinking skills related to dual diagnosis addressed in the format of the curriculum. Multiple choice and case example formats were chosen since both knowledge and problem-solving skills needed to be assessed. In addition, the case example method of testing most readily corresponded to the problem-based method approach (since this was limited to a pencil and paper test) and to the approach most encountered by personnel when performing their jobs.

Once the test was completed, the format and content were reviewed and rated by twelve experts in the fields of developmental disabilities, mental health, and/or test development. After incorporating suggested changes, the test was again reviewed and rated by experts in the field, and after a number of revisions using this process, the test was finalized to its current version. Ratings pertaining to the final, current version were recorded by experts on a form that contained questions about the test’s (1) readability, (2) accuracy and adequacy of reflecting the goals and objectives of the curriculum, (3) length, (4) proportional representation of the content contained within each of the five chapters of the curriculum, and (5) relevancy. All seven experts who reviewed the current version of the test had rated each of the five questions as a “5” according to a five- point Likert scale where “5” corresponded to “strongly agree” and coincided with an “excellent” rating (based on how the questions were structured). In addition, none of the experts had recorded any suggestions, comments, problems, or concerns on the subjective section of the form that was reserved and provided for this purpose.

Pilot Study

In order to obtain feedback from the persons for whom the test was written and to obtain data about how the test performs, the test was administered to twenty of the thirty people who had participated in the two-day curriculum delivery in May of 1997. Each of these twenty people completed the test, recorded the total testing time, and evaluated the test using the “Direct Care Curriculum Participant Rating Form.” The responses from the test and rating form were scored and/or entered into an Excel data base. Since the test consists of both a multiple choice and an “essay” section, test responses were scored by two independent raters and according to a scoring rubric, and interrater reliability was assessed using the Pearson Correlation Coefficient for the total test scores rendered by the two raters. Descriptive statistics (i.e., mode, mean, standard deviation, and standard error of measurement) were calculated for responses to the rating form and the recorded test-taking times. The Cronbach alpha reliability coefficient was calculated for the test, and item analysis was conducted for each multiple choice test item. These statistics provided estimates of reliability (internal consistency) for the test scores obtained from the pilot study. Since content validity evidence for the test had already been obtained via ratings from experts and since no “gold standard” or any other kind of standard pertaining to the construct of knowledge about dual diagnosis exists, validity was not statistically addressed within this research study.

Results of the Pilot Study

A summary of the demographic data obtained from the twenty pilot study participants includes the following: Mean age: 42.65 (range 26 to 61); Job title: 8 Training Specialists/Administrators, 9 Qualified Mental Retardation Professionals/ Case Managers, and 3 Client Instructors/Direct Care Staff; Organization: 9 employed at a State Operated Facility and 11 employed at three different private community agencies; Work setting: 13 worked in residential settings, 2 worked in vocational settings, 3 worked in both residential and vocational settings, and 2 worked in neither; Gender: 6 males and 14 females; Highest educational level completed: 1 didn’t graduate from high school, 9 had a high school diploma or GED, and 10 had a Bachelor degree; Time working in the field of developmental disabilities: average of 160 months (or 13.3 years) and a range of 24 months to 382 months (or 2 years to 31.8 years); Number of trainings, workshops, or inservices related to the field of developmental disabilities participated in over the past two years: average of 8.9 and a range of 1 to 50; Number of trainings, workshops, or inservices specifically related to dual diagnosis: average of 2.35 and a range of 1 to 8; and Mean testing time: 66.15 minutes and a range of 42 to 99 minutes (standard deviation: 14.7 minutes). The Cronbach alpha reliability coefficient (i.e., a measure of internal consistency) calculated for the multiple choice section of the test was equal to .815700 or to .82. Using the Pearson Correlation Coefficient, the correlation calculated between one rater’s score and the other rater’s score for each of the nine essay questions ranged from .95014 to 1.00000, and the generalizability coefficient used to calculate the overall interrater reliability (based on a 2 x 9 model for the total test scores rendered by the two raters) was equal to .76945. Since the reliability of the instrument was approximately equal to or greater than .8, which is considered very acceptable throughout the research literature, and since a review of the item discrimination and item difficulty indexes revealed no major problems with any of the items, the test was maintained in its current version and administered to 120 subjects (Allen & Yen, 1979).

REFERENCES

Allen, M. & Yen, W. (1979). Introduction to measurement theory. Belmont, CA: Wadsworth, Inc.

Barrows, H. (1992). The tutorial process (2nd Ed.). Springfield, Illinois: Southern Illinois University School of Medicine.

Barrows, H. & Kelson, A. (1995). Problem-based learning: A total approach to education. Springfield, IL: Southern Illinois University School of Medicine Monograph Series.

Corbett, J. (1985). Mental retardation: Psychiatric aspects. In M. Rutter & L. Hersov (Eds.), Child and adolescent psychiatry modern approaches (pp. 661-678). Oxford: Blackwell Scientific Publications.

Galligan, B. (1990). Serving people who are dually diagnosed: A program evaluation. Mental Retardation, 28 (6), 353-358.

Illinois Department of Public Health Requirements for Long-term Care (1990). Illinois Administrative Code, Title 89. Springfield, IL: Legislative Printing Unit.

Loschen, E. (1997). The field of developmental disabilities: Historical overview and future prospectives. A presentation made at the Department of Psychiatry’s Twenty-fifth Anniversary Continuing Medical Education Symposium, Springfield, Illinois, April.

Matson, J. & Frame, C. (1986). Psychopathology among mentally retarded children and adolescents. Beverly Hills, CA: Sage.

Menolascino, F. & Fleisher, M. (1993). Mental health care in persons with mental retardation: Past, present, and future. In R. J. Fletcher & A. Dosen (Eds.), Mental health aspects of mental retardation: Progress in assessment and treatment. New York: Lexington Books.

Menolascino, F., Wilson, J., Golden, C., & Ruedrich, S. (1986). Medication and treatment of schizophrenia in persons with mental retardation. Mental Retardation, 24, 277-283.

Reid, A. H. (1982). The psychiatry of mental handicap. London: Basil Blackwell.

Rutter, M., Tizard, J., Yule, W., Graham, P., & Whitmore K. (1976). Isles of Wright studies: 1964-1974. Psychological Medicine, 6, 313-332.

Silka, W. & Hauser, M. (1997). Psychiatric assessment of a person with mental retardation. Psychiatric Annals Journal of Continuing Psychiatric Education, 27, 162-169.

Sovner, R. & Hurley, A. D. (1983). Do the mentally retarded suffer from affective illness? Archives of General Psychiatry, 40, 61-67.

For further information contact:

Cynthia S. Mester, MA LCPC
Southern Illinois University School of Medicine
751 N Rutledge, PO Box 19230
Springfield, IL 62794-1412