Progress Monitoring with Psychometric Tools and The Reliable Change Index

In educational terms, progress monitoring concerns ongoing evaluation of student responsiveness as an intervention is applied (Chafouleas, Johnson, Riley-Tillman & Iovino, 2021). In clinical settings, this is akin to the long-recognized requirement to evaluate individuals’ progress during psychotherapy (Howard, Moras, Brill, Martinovich & Lutz, 1996). Emily comprehended that the technical strengths associated with Kayla’s math goals are absent in her social-emotional goals. Now that the school year is ending, Emily wondered what can be determined about Kayla’s anticipated progress. Let’s start with the self-esteem goal for Kayla. Emily’s foundational readings in progress monitoring and her own reflections revealed several problems. To address this goal’s attainment, who will actually judge the presence or absence of “positive” self-esteem? Will it be the special education teacher?  (She might have a risk of bias because she also provides instruction). Alternatively, might it be an unbiased independent observer? What definition of self-esteem might be held by prospective observers? What occasions (as mentioned in the goal) will be used to observe self-esteem? What is the threshold for “positive” self-esteem (as mentioned in the goal)? How reliable is the judgment that “positive” self-esteem was witnessed during each observation? Clearly, this goal suffers many problems that do not plague Kayla’s psychometrically based math goals. Equally problematic, and largely parallel, considerations arise concerning the second social-emotional goal. Who knows what is meant by the goal’s phrase of a “positive choice?” Who knows what is meant by “when feeling anxious?” Would possible observers use the same criteria in the same way when making repeated judgments about Kayla’s actions? A valid and reliable determination of this goal’s mastery, just like with the first goal, seems dubious.

Emily already knew that there are commercially available measures of “self-concept.” One is entitled the Piers-Harris Children’s Self-Concept Scale, 2^nd edition (Piers-Harris-2; Piers & Herzberg, 2002). It is made up of 60 yes-no, self-report items suited to children 7 years and older. To assure that the construct of self-esteem is fully represented the Piers-Harris-2 contains items that concern the following: behavioral adjustment, intellectual and school status, physical appearance and attributes, freedom from anxiety, popularity, as well as happiness and satisfaction. For many purposes, including the present one, a total score (reported as a T-score) can be used. Presumably, part of Kayla’s problem is low self-esteem; improved self-esteem accomplished by participation in special education and related services thus represents a reasonable goal. Part of the IEP’s purpose is to determine if the self-esteem related goal is attained.

Now let’s begin to consider technical issues that confront Emily. It is self-evident that a Piers-Harris-2 Total Score cannot possibly be perfectly reliable. By chance alone, a first and a second Piers-Harris-2 Total Score will differ somewhat (hence the necessity to apply confidence intervals when interpreting scores under any circumstance, such as a one-time measurement of self-esteem during a static, garden variety evaluation). Logically then, when an end-of-the-year observed score seems better than a beginning-of-the-year score, it is necessary to judge whether the observed difference reflects real improvement or is just an artifact of measurement error (i.e., score differences arising from imperfect reliability). Using this line of reasoning, an instrument with good reliability would be especially advantageous when pre-post scores are compared.

To help the team expertly determine if genuine progress has occurred, school psychologists like Emily just need standardized social-emotional instruments whose manuals report reliability statistics. To remind you, a key reliability indicator is standard error of measurement (SEM). SEM is a hypothetical notion. It concerns the standard deviation that would hypothetically arise if the same individual took the same test an infinite number of times. If scores clustered closely around a central value on repeated administrations, the standard deviation of that distribution (the SEM) would be small. Conversely, if scores fan out widely around a central value on repeated administrations, the standard deviation of that distribution (the SEM) would be large. Thus, SEM is just a variant of commonly seen reliability statistics, but one configured for applied use. For example, SEM values can be used to create confidence intervals, as commonly seen for ability and achievement test scores in school psychologists’ reports.

https://ir.canterbury.ac.nz/bitstream/handle/10092/13399/12664317_Reliable%20Change%5ETutorial%5ENZPsS%5E2016.pdf?sequence=1. And, Appendix E instructs you on preparing RCIs.

Emily, in fact, discovered a non-chance level of improvement in Kayla’s scores. This was confirmed when RCI calculations indicated that Kayla’s observed beginning-of-the-year Piers-Harris T-score of 24 (<1st percentile) compared with her end-of-year T-score was > 35 (7th percentile) represented a reliable (non-chance) improvement in global self-esteem. Given our shared human predispositions to misinterpret (e.g., confirmation bias), the objectivity inherent in using RCI seems to represent a huge advantage. As you have heard repeatedly in this text, swapping rationality and objective thinking (the Reflective System) for intuition and subjectivity (the Automatic System) should be welcomed anywhere in our professional lives where it can be found.

A Few Tools Suited to RCI Calculations

Many school psychologists have never heard of RCI (this included Emily before this undertaking). What’s more, the computation needed to use RCI might engender resistance even among those who gain familiarity. Happily, some manuals now include pre-calculated RCI values (or at least most of the statistics needed to quickly generate RCI values). Emily’s former professor directed her toward some such tools. Among broad-band scales, for instance, the Conners CBRS includes a section on “statistically significant change” on pages 156-157 of its manual. The various tables found on these pages indicate how many raw score and T-score points of positive change are needed to reach statistical significance at the 90% confidence level. The following Conners CBRS dimensions have values available–Disruptive Behavior Disorder Indicator, Learning and Language Disorder Indicator, Mood Disorder Indicator, Anxiety Disorder Indicator, and ADHD Indicator. There is no such coverage, however, found in the BASC-3 manual (instead, the BASC-3 has its own tool entitled the BASC-3 Flex Monitor; Reynolds & Kamphaus, 2016).

Further good news is that several narrow-band scales now embrace use of the RCI (see Table 15.2). Regarding ADHD, for example, the ADHD-5 Rating Scale (DuPaul, Power, Anastopoulos & Reid, 2016) is set up for RCI use. The authors address RCI in their manual and the manual reports standard error of difference (the prime value needed to calculate RCIs) for scores related to Inattention, Hyperactivity-Impulsivity as well as a Total score (see the manual’s p. 88-89). In parallel, when organizational skills are a target of intervention, the Children’s Organizational Skills Scales (Abikoff & Gallagher, 2009) might be used. Raw scores and T-scores at the 95% confidence level are reported in the manual (see the manual’s p. 71-73). For monitoring improvement of depressive symptoms, the Children’s Depression Inventory-2^nd ed. (CDI-2 Kovacs, 2011) might be considered. The CDI-2 manual explicitly addresses score changes and guides the user in making judgments about whether a reliable change was achieved. The CDI-2 manual, however, reports necessary T-score value changes without discussing RCI specifically or indicating the confidence threshold (e.g., 90%, 95%) associated with reported values (see the manual’s p. 49-51). School psychologists have other options that might be tapped for use in students’ IEPs. For instance, regarding anger management, the Anger Regulation and Expression Scale (ARES; DiGiuseppe & Tafrate, 2011) is a candidate. The ARES manual reports threshold T-score differences large enough to allow school psychologists to infer that score changes have reached a non-chance level. This is true even though the ARES manual fails to indicate which level of confidence is used (see p. 40-41 in the manual for an explanation of RCI and p. 97 for a table of values). Regarding anxiety, the Multidimensional Anxiety Scale for Children-2nd edition (MASC-2; March, 2013) reports T-scores at the 90% confidence interval (see manual p. 30).

Emily then began to consider the practicalities of RCI use. She realized that many students already have available scores from the very tools cited above; these tools sometimes having been administered during the assessment phase. Thus, accessing assessment-derived scores from popular tests like the Conners CBRS or the MASC-2 might springboard psychometrically measured annual goals in a student’s IEP. Fortunately, end-of-the-year repeated administration of the Conners CBRS or MASC-2 would require relatively little effort. Similarly, the number crunching for pre-post score comparisons is easily accomplished. With these facts in front of her, Emily continued to warm to the idea of expanded participation in progress monitoring.

Emily discovered other realities. The tools listed above might sometimes be used for progress monitoring, but they were not created with that purpose in mind. In other words, most of these tools are marketed to assist in standard assessments (e.g., to assist in a diagnosis), but when deployed for progress monitoring, the scales are being used dimensionally (see Chapter 1 for a discussion of categorical and dimensional considerations). In fact, each manual’s coverage of RCI (and how to document treatment-related improvement) invariably takes a backseat to more conventional uses, such as ruling in or ruling out problems and helping to establish diagnostic conclusions. But Emily found that this was not true for one particular tool–the Behavior Intervention Monitoring Assessment System, 2nd edition (BIMAS-2; McDougal, Bardos & Meier, 2016). In contrast to its assessment-focused counterparts, the BIMAS-2 was originally constructed with sensitivity to change as an explicit mission. Thus, its manual is filled with information about progress monitoring (and screening). Prospective users will need to spend time on the manual and learn all of its capabilities. Simply put, the BIMAS-2 standard form concerns two sets of scales. The first set comprises “Behavioral Concern Scales:”

• Conduct (e.g. problems managing anger, bullying others)
• Negative affect (e.g., anxiety, depression)
• Cognitive/attention (e.g., focus, planning)

The second set comprises “Adaptive Scales:”

• Social (e.g., maintaining friendships, communication)
• Academic Functioning (e.g., academic performance, direction following)

The informant options for the BIMAS-2 include teachers, parents, professionals and self. As one might suspect, the BIMAS-2 is prepared to help create RCI determinations. For school psychologists interested in routine use in progress monitoring, like Emily and the DM3, the BIMAS-2 might ultimately work better than the popular tools discussed above. Emily made note of the BIMAS-2 for future consideration. Readers might wish to do the same.

Table 15.2 Examples of Tools for Quantitative Measurement of Progress (including RCI)
Broadband
Name of Tool	Some dimensions measured
BASC-3 Flex Monitor	Inattention/hyperactivity, internalizing problems, disruptive behaviors, developmental social disorders, school problems
Conners Comprehensive Behavior Rating Scale	Disruptive behaviors, learning and language, mood, anxiety, ADHD
BIMAS-2	Conduct, negative affect, cognitive/inattention
ASEBA Progress & Output App	Internalizing, externalizing, total problems
Narrowband
Name of Tool	Dimension measured
ADHD-5 Rating Scale	Inattention, hyperactivity/impulsivity, total ADHD symptoms
Piers-Harris 2	Self-concept
Children’s Depression Inventory-2	Depression
Anger Regulation and Expression Scale	Management of anger
Children’s Organization Skills Scale	Organization

Screening and Multi-tiered Mental Health Services

Emily’s work on behalf of the DM3 seems to hold promise for role expansion (plus the prospect of truly helping students). But even if adopted, beefed up IEPs and better monitoring of their mental health elements still reflect duties tethered to special education. Kimani’s information-gathering took a somewhat different course and his efforts hold vastly greater possibilities. Because he worked part-time as an adjunct faculty member at a local junior college, Kimani could conduct online literature searches that were impossible for Emily. Consequently, Kimani commenced a search using topic descriptions like “social-emotional,” “screening,” and “school” to discover what the published literature had to say. He also spoke with his sister, a pediatrician, someone who held her own perspective on screening regarding timely detection of social-emotional problems. Favorably, Kimani quickly realized that he was already familiar with the logic behind screening, a central dogma of which is that early problem detection allows maximum time to intervene. As you might suspect, this idea’s intuitive appeal resonates both inside and outside of schools. For example, the American Academy of Pediatrics, as Kimani learned from his sister, proposes routine screening for both developmental and social-emotional concerns beginning with toddlers (Weitzman & Wegner, 2015). Relevant to the upcoming discussion on screening instruments, however, research (Sheldrick, Merchant & Perrin, 2011) suggests that pediatricians’ summary judgments alone aren’t advised. To do so as a means of detecting problems results in unacceptable levels of sensitivity (as low as 14%) and specificity (as low as 69%). This reminded Kimani of his own concern that holistic teacher impressions used to trigger referrals for ED evaluations often seemed inaccurate. Thus, pediatricians are urged in the medical literature to use objective screening tools to enhance these diagnostic utility rates. Doing so would be consistent with the notion that global clinical judgments are often inferior to structured and instrument-associated verdicts (recall the classic work by Meehl & Rosen, 1955 found in Chapter 2). Early problem detection in pediatricians’ offices, however, offers both opportunities and foments challenges. For example, pediatric practitioners recognize they are unable to manage social-emotional problems alone; accordingly, the pediatric literature urges them to link with community resources (Weitzman & Wegner, 2015). Nonetheless longstanding calls for school psychologists and pediatricians to join efforts for the benefit of children (Sheridan, Warnes, Woods, Blevins, Magee & Ellis, 2009; Wodrich & Landau, 1999) seem to have gone largely unanswered. Notwithstanding imperfect pediatrician-school links, early identification efforts of IDEA necessarily depend on health providers (e.g., pediatricians) to detect problems and then direct preschoolers toward school-based (not just health care) services. This is covered in detail by the Birth to Five: Watch me Thrive! module found on the U.S. Department of Education website https://www.acf.hhs.gov/ecd/child-health-development/watch-me-thrive (retrieved November 10, 2020).

Our focus, of course, is the school-based practice of psychology concerning primarily school-age children, not pediatricians’ screening practices. One key educational rationale is that screening can help school districts with their affirmative obligation to identify students with any type of disability (the Child Find proviso of IDEA, see Chapter 10). Screening, thus, can serve as a logical prelude to in-depth, school-based evaluations for one or more of the disability categories described in IDEA. In fact, it is argued that effective screening may redress flaws entrenched in the special education identification process. This is because the special education referral process often hinges on teacher-initiated referrals and, critically, those referrals may reflect teachers’ implicit biases about their students. What’s more, a process dependent on teacher nominations may prioritize overt behavior problems over students’ internalizing problems, such as inconspicuous anxiety or covert depression. Holistic conceptualization of students is vulnerable to bias and a host of heuristics addressed numerous times in this book (e.g., halo effect). In contrast, data driven approaches (e.g., standardized rating scales) tend to dissociate from teacher referrals, which inherently include subjective aspects (Dowdy, Doane, Eklund & Dever, 2013; Eklund et al., 2009). Moreover, use of objective instruments may help to diminish the widespread special education over-representation of ELL and minority students (see Sullivan, 2011; Sullivan & Val, 2013). To this very point, Dever, Raines, Dowdy and Hostutler (2016) found using a self-report screening tool (i.e., the Behavioral Emotional Screening System, BESS, Kamphaus & Reynolds, 2015) might partially ameliorate some current problems with special education identification. The authors of this large study (a nationally representative sample of nearly 5,000 children) concluded, “These findings suggest that a data-driven approach to inform referral for special education may contribute to efforts to reduce the disproportionate placement of students of color and males in special education” (Dever et al, 2016, p. 59).

Several different school-centered screening tools now exist (see review by Jenkins and colleagues, 2014). One commercially available option receives high ratings regarding its psychometric properties and clinical utility; it is the BESS  (Kamphaus & Reynolds, 2015). BESS, available in English and Spanish, screens for those 3-0 to 18-11 years via parent and teacher report and from 8-0 to 18-11 years via self-report. There are 25 to 30 items. The following dimensions are assessed:

• Behavioral and Emotional Risk Index
• Externalizing Risk Index
• Internalizing Risk Index
• Adaptive Skills Risk Index
• Self-regulation Risk Index
• Personal Adjustment Risk Index

The BESS reports T-scores, percentiles, and classification ranges (e.g., “elevated risk”) for its “Behavioral and Emotional Risk Index.” Sometimes school psychologists will also use classification range information available for each subscale. There are also validity scores and critical item indications. Some may question whether this is a tool for a clinic, or perhaps for addressing emotional or behavioral problems during SLD evaluations, but not a universally-applied screener. You will see more on this topic when the notion of tiered levels of services appears later in this chapter.

The BESS also seems to enjoy popularity because it is a member of the wide ranging BASC-3 family. In addition, some researchers have chosen the BESS as a gold standard against which to compare the characteristics and successfulness of other screening tools, further confirming its dominance. For example, a 19-item, teacher-completed instrument entitled the Social, Academic, and Emotional Behavior Risk Screener (SAEBRS; Kilgus & von der Embse, 2015) was compared to the BESS with 704 elementary students. The SAEBRS produced high rates of concordance with the BESS (92% classification agreement), plus strong evidence of sensitivity (.93) and specificity (.93; Kilgus, Eklund, von der Embse, Taylor & Sims, 2016). Interestingly, the same study confirmed the value of a multi-gate procedure (MGP). That is, combining a screening “gate” related to teacher nomination in concert with a screening “gate” from SAEBRS scores was supported. Critically, teacher nominations alone fared less well. Furthermore, the SAEBRS has also been studied empirically to locate optimum cut-scores, including use of a rigorous cross-validation procedure (Kilgus, Taylor & von der Embse, 2018). The link for accessing SAEBRS follows http://www.fastbridge.org/assessments/behavior/behavior/.

Others have also examined the BESS. For example, Kettler, Feeney-Kettler and Dembitzer (2017) investigated the BESS (a single gate procedure) with multi-gate Preschool Behavior Rating Systems (PBSS, Feeney-Kettler, Kratochwill & Kettler, 2009). They found the BESS to be a better predictor of ASEBA scores for parents and teachers.

Kimani already knew about the BASC-3 TRS and PRS from his routine practice, thus the BESS also seemed familiar. Other school psychologists routinely use the Conners CBRS in lieu of the BASC-3. For them Conners CBRS-related screening might be more appealing. To this end, the Conners CBRS Clinical Index, comprised of 24 items, is available as a screener. It provides T-scores in the following areas:

• Disruptive Behavior Disorders
• Learning and Language Disorders
• Mood Disorders
• Anxiety Disorders
• ADHD

The manual, however, does not appear to address screening-related issues per se. For example, there are no indications of preferred screening-related cut-scores. Moreover, if entire student populations were screened using a liberal cut-score (e.g., T-score = 60) there is no way to predict roughly how many students might be flagged on at least one of the areas listed above. In other words, if there were a 16% chance that a randomly-selected student would express an elevated T-score on “Disruptive Behavior Disorders” (this might be expected with a cut-score of T-score = 60) plus a 16% chance of an elevated T-score on “ADHD,” etc., then the screening process might eventuate in too many students needing additional assessment.

Kimani discovered another candidate for universal screening, the BIMAS-2, which you already read about regarding progress monitoring. The BIMAS-2 manual tackles screening head on (see McDougal, Bardos & Meier, 2016). It shows that when T-scores of 60 are used for clinical dimensions (and T-scores of 40 for adaptive dimensions) that satisfactory effect sizes (Cohen’s d) as well as indicators of discrimination (discriminant function analysis) are documented. As mentioned above, however, the rate at which students might actually be screened out via a universally-applied process remains unknown when multiple dimensions (e.g., Conduct, Negative Affect, Cognitive/Attention) are checked.

In light of this information, Kimani can foresee social-emotional screening as an antidote to the possible capriciousness of teachers’ ED referrals. But there are potential limitations (and thorny issues) even when an apparently reliable and valid tool (e.g., BESS) is used. These include: (1.) the question of parental permission, (2.) where to place cut-scores, and (3.) access to appropriate post-screening services. These topics are covered below.

The Question of Parental Permission to Screen

Is it necessary to secure parental permission to screen students for mental health problems? If you, like the DM3, are considering mental health screening, this is a crucial question. The answer seems to be, “it depends.” The NASP ethics seem to say one thing. This is spelled out quite clearly by Eklund and Rossen (2016) in an article on trauma screening that also seems to speak to social-emotional screening generally. First, NASP Principles for Professional Ethics indicate that there is no necessity to secure parental consent to “participate in educational screenings conducted as part of a regular program of instruction” (NASP 2010, p. 4; NASP 2020, p. 42). Second, however, focused screening (e.g., concerning social-emotional problems) seems to be a different matter. Specifically, the federal Protection of Pupil Rights Amendment (PPRA) indicates that LEAs need to “obtain prior written consent from parents before students are required to submit to a survey that concerns” several areas, one of which is “mental or psychological problems.” These facts would seem to mandate active written consent for social-emotional screenings. In fact, Eklund and Rossen opine that counting on passive consent (i.e., permission is assumed unless parents “opt out”) probably violates the PPRA. It’s also important that passive consent may be even less ethically acceptable among parents with low English language proficiency and/or limited literacy. Sometimes, however, state statues indicate what must be done. For example, the state of Arizona makes explicit the necessity for active parental permission. This is spelled out in Illustration 15.1. Thus, professionals working in Arizona schools, including school psychologists, are left with no doubt that it will not suffice to simply count on opt out procedures if they hope to conduct universal mental health screenings. In light of these considerations, it is probably the case that local customs determine how this topic is managed, perhaps with input from each school district’s legal counsel.

Selecting Cut-scores for Screening

Kimani also had to consider the practical question of cut scores. He recognized that when screening it might be better to over-identify than to under-identify. That is because screening will prompt a thorough evaluation (e.g., concerning the possible presence of ED). If a student’s detailed (post-screening) evaluation concludes there is no problem, then there was only wasted time and effort but no real damage. On the other hand, if a true case is missed, then that student is deprived of needed, important and legally mandated services. Similarly, as seen below, screening sometimes prompts delivery of low-intensity services. These services may help but they are unlikely to harm. Cumulatively, these considerations argue for a lenient cut-score, one that could tolerate many false positives but relatively few false negatives (misses). For example, a T-score of 70 (denoting clinical range) may make sense for helping to verify a problem’s presence during detailed evaluation. But a T-score of 60 (denoting at risk) may make more sense for screening purposes. Unfortunately, as you saw above, some screening tools’ manuals are silent on the topic of cut-scores. You will see more below about making decisions based on students’ score levels.

Screening Linked to Multi-tiered Systems of Supports

Is the entire point of screening simply to detect potential cases of missed ED (thus triggering detailed eligibility evaluations)? As you might have surmised, stopping here would prove unacceptable for Kimani and the DM3. After all, the group’s purpose is to do more to help (read “truly help”) students. Thinking big, Kimani and the DM3 contemplated systems-level changes that involve addressing mental health delivery. That is, rather than concerning students one-by-one, schools might gear up with an array of social-emotional interventions. Later, these services could be deployed to students who fail a screening or otherwise indicate a need. Seen this way, any effort at screening is a portal to a much larger process. Consider the following perspective on screening, “Schools planning to engage in such work should also ensure that attempts to participate in screening should include considerations for ….. responding to student needs once identified” (Eklund et al. 2018, p. 41). How might this be done? A logical method would be to offer several levels of social-emotional supports and interventions. This is analogous to the Response to Intervention (RTI) approach regarding academic skills wherein needed services are calibrated to several intensity levels (e.g., see the edited volume by Jimerson, Burns & VanDerHeyden, 2015). When this notion concerns social-emotional needs, it is sometimes referred to as Positive Behavior Interventions and Supports, PBIS (or alternatively School Wide Positive Behavior Interventions and Supports, SWPBIS; Kaurudar, 2018). PBIS often comprises three tiers. For example, Tier 1 involves prevention-oriented procedures. In the PBIS world, supports at this level would be made available to all students regardless of their screening results. Tier 2 PBIS services are envisioned to be more restrictive and used only for students who failed a screening or otherwise expressed a risk (e.g., students with frequent discipline referrals). At Tier 3, PBIS efforts concern ameliorating more obvious social-emotional problems. And Tier 3 interventions are especially tailored to individual students (Sugai & Horner, 2008). But PBIS is not just about more intensity in the face of more needs. A hallmark of PBIS is the provision of evidence-based interventions (Wiest, Stevens et al., 2018).

Let’s get a bit more specific. Wiest, Eber and colleagues (2018) advocate for tiered services for students with internalizing problems (including exposure to trauma), as just one example. Internalizing problems are especially relevant for a couple of reasons. One reason is because teachers may overlook such students instead directing their concern toward classmates whose disruptiveness or noncompliance proves harder to miss (McIntosh, Ty & Miller, 2014). Another reason is that children with internalizing problems have treatment needs that are distinct from counterparts with externalizing problems (Wiest, Eber et al., 2018). Universal, Tier 1, supports may benefit all students, including those with internalizing problems. But when problems persist, grow worse, or come with additional indicators of risk, Tier 2 services might be next needed. Tier 2 services might include evidence-based interventions like the Checkin Checkout Procedure (Maggin, Zurheide, Pickett & Baillie, 2015) or the use of Daily Progress Reports (DRPs, Crone, Hawken & Horner, 2010). These omnibus Tier 2 interventions, however, would necessarily need adaptation for this type of student. This might take the form of “a student struggling with anxiety issues could have expectations on the DPR of ‘stay calm and use deep breathing skills when there are classroom disruptions’” (Weist, Eber et al., 2018, p. 178). Also at Tier 2, a student with a history of trauma might receive a program like Support for Students Exposed to Trauma (Jaycox, Langley & Dean, 2009). At Tier 3, manualized programs like Coping Cat (Kendall & Hedtke, 2006) or Cognitive Behavioral Intervention for Trauma in Schools (Jaycox et al., 2009) might be used. Lower tier services are sometimes provided by teachers, whereas those at higher levels might warrant direct provision by counselors, social workers, therapists or school psychologists.

Realistically, however, in some (many) schools staffing limitations mean that there is neither the time nor the expertise to deliver all needed mental health services. Consequently, over the years more extensive and better organized school mental health services have been advocated. Especially noteworthy are interconnected systems that include employees from both schools and clinics working together on the same campus. Emblematic of an interconnected mental health system is the Southeastern School Behavioral Health Community that exists in South Carolina (Weist, Stevens et al, 2018). A website associated with this program (www.schoolbehavioralhealth.org) indicates the possibility for implementing a tiered system. Such a system involves staff members ranging from teachers to counselors to school psychologists employed by schools working hand and glove with mental health professionals employed by extra-school clinics and agencies. When systems work well, they assure that all students receive some support. But students with more needs receive more services, including evidenced-based interventions delivered by skilled providers. Again, their foundation is PBIS and PBIS is itself shown both to be effective on multiple indicators (e.g., reduced discipline referrals, enhanced academics) and implementable (see Wiest, Stevens et al., 2018). Site of delivery is also a factor, as schools are often the prime venue for the delivery of today’s children’s mental health services (Bruhn, Woods-Grove & Huddle, 2014). What’s more, when mental health services are delivered on a school campus, rather than at an off-campus clinic, transportation problems are circumvented and missed appointments may be minimized or eliminated. Similarly, communication problems can be reduced when therapists and behavioral consultants work on the same campus as classroom teachers.

Concerning Tier 2, Conners CBRS, when used as a screen, or the BIMAS-2 might facilitate students-to-services matches. An elevation on the Disruptive Behavior Disorders Indicator of the Conners CBRS, for example, might direct a student toward one array of Tier 2 services that are ill-suited to a counterpart with an elevation on the Anxiety Disorders Indicator. In fact, it can also be argued that the level of tier selected could be guided by the very same screening instruments that flag students. Recall that T-scores between 60 and 69 on many instruments suggest a student “at risk.” Also recall T-scores above 70 often suggest a student with a “clinically significant” problem. Accordingly, the former would seem to indicate, all other assessment information pointing to the same severity level, Tier 2 services. By the same token, the latter, all other assessment information pointing to the same severity level, to indicate Tier 3 services. Wiest, Eber and colleagues (2018) offer a somewhat similar rationale for matching students to tiers of supports. This would represent the simplest, least expensive, way to locate students for Tier 2 services and marry them to the proper type of service.

But about Tier 3? It seems logical that for Tier 2 vs. Tier 3 decisions that all sources of information (i.e., findings from comprehensive social-emotional assessments) should be used. As was argued in Chapter 1, the more important the assessment-related decision, the more comprehensive the assessment should be (recall Angela and the consideration of social skills group in Chapter 1 as exemplifying a relatively low-stakes assessment). Tier 1 and Tier 2 screening questions, in the aggregate, might be considered to be relatively low stakes. Tier 3, where labels and formal special service designations sometimes come into play, are generally no longer low stakes. In the hope of expanding their roles and truly helping students, Kimani and the DM3 decided to place themselves in the loop so as to conduct in-depth social-emotional assessment at the cusp of any Tier 3 service. They sought to assess all Tier 3 candidates, helping match them to needed services, and potentially themselves deliver some counseling or psychotherapeutic interventions (you will see more about this possibility in the next section as Flora’s efforts are reviewed).

Some of the appeal of screening (besides enhance ED referrals) concerns role expansion. By offering intervention services, starting with those of low-intensity, provided free of potentially stigmatizing labels, and accentuating supports, students’ mental health problems might be headed off before they become severe or chronic. Severe or chronic problems might also be addressed at suitable levels of intensity. This would place school psychologists within the context of mental health providers, rather than as mere supporters of special education (see Kilgus, Reinke & Jimerson, 2015). Procedures like those describe above may also help assure that the estimated one-quarter of students who express a social-emotional problem each year (Perou et al., 2013) do not go untreated. It’s easy to envision why the DM3 might consider screening coupled with PBIS to be a bona fide realization of their hope to do more.

For those aspiring to learn more about mental health screening as a prelude to a multi-tiered system of school-based supports, like members of the DM3, there is a helpful implementation guide https://smhcollaborative.org/universalscreening/(retrieved November 8, 2020). Its title proves to be quite descriptive of its contents, Best Practices in Universal Screening for Social, Emotional, and Behavior Outcomes (Romer et al. 2020). The downloadable document is replete with checklists and forms able to assist in practice.

Counseling and Psychotherapy Informed by Assessment

Flora is every bit as intent as her DM3 colleagues to genuinely help students. But she sensed that her already sizeable counseling practice suffered from two Achilles’ heels: (1.) the singular use of client-centered techniques for her individual counseling cases, and (2.) abundant students enrolled in her social skills training groups. On reflection, she concluded that both her individual and group work were uncoupled from any rational assessment process. She wondered if counseling was indicated for all of her cases and whether her technique matched the needs of students actually requiring services. The same was true regarding the social skills group she inherited. Furthermore, even when she was not a direct treatment provider herself, she wanted to know what treatments she ought to recommend for others to implement. In other words, sometimes parents request suggestions for outside therapists, or mentioned various mental health treatment options under consideration, often at the time of IEP meetings. Flora’s mission for the DM3 concerns assessment-treatment linkage. In essence, it encompasses the following question: “How do my assessment findings intelligently guide recommendations for mental health treatment?”

To help answer this critical question, Flora was directed by a senior colleague toward a helpful website provided by the American Psychological Association’s Division 53 (Society of Clinical Child and Adolescent Psychology). It is entitled “Evidence-based Mental Health Treatment for Children and Adolescents.” Here is the link: https://effectivechildtherapy.org (retrieved November 10, 2020). The site indicates, “…EBTs [evidence-based treatments] are treatments that are based on scientific evidence. Research studies have shown that some treatments work better than others for specific problems…” There are similar books prepared by researchers, (notably Weisz & Kazdin’s 2017 comparably entitled volume Evidence-based Psychotherapies for Children and Adolescents). The meaning is as transparent as it is essential: scientific knowledge, not intuition, determines what is likely to work. And, effectiveness, at least in part, depends on the nature of the circumscribed problem treated. When it comes to psychological interventions, one size does not fit all. Parenthetically, the same is true for psychotropic medications; methylphenidate (e.g., Ritalin®) is routinely used for ADHD, not for bipolar disorder; SSRIs (e.g., Prozac®) are routinely used for depression, not for conduct disorders. Regarding psychological interventions, Table 15.3 provides a hint at various treatments and their associated levels of efficaciousness. For example, straightforward behavioral interventions are well suited to ADHD, cognitive behavior therapy (CBT) to anxiety, family-oriented approaches to anorexia. Client-centered therapy, the workhorse of Flora’s counseling, appears only in the column associated with anxiety and, disappointingly, the technique is devoid of efficacy evidence. The contents of Table 15.4, and by inference similar information, hold obvious implications for school psychologists like Flora who want to help students with mental health concerns. Mechanically enrolling students in generic interventions (such as those that simply happen to be familiar to the provider) may prove futile. School psychologists are too busy to waste time on things that do not work.

Table 15.3 Empirically Supported Psychological Treatments
	ADHD	Anxiety	Anorexia nervosa
Level One “works well”	Behavioral parent training Behavioral classroom training Behavioral peer intervention	Cognitive behavioral therapy (CBT)* Exposure CBT with parents education CBT with medication	Family therapy (behavior)
Level Two “works”	Combined training interventions	Family psychoeducation* Cultural storytelling Stress inoculation	Family therapy (systemic) Insight-oriented psychotherapy
Level Three “might work”	Neurofeedback training	Contingency management Group therapy	none
Level Four “unknown/untested”	Cognitive training	Biofeedback* Play therapy Psychodynamic Social skills	CBT Cognitive training
Level Five “tested and does not work”	Social skills training	Attachment therapy* Client-centered therapy Eye movement desensitization & reprocessing	none
*Partial list only. Source: American Psychological Association, Evidence-based Mental Health Treatment for Children and Adolescents

But what method is used to match treatment and student characteristics? Although the “Evidence-based Mental Health Treatment for Children and Adolescents” website is organized several ways, one way hinges on diagnoses, like those found in DSM-5. This is evident in Table 15.4. The message is that when it comes to treatment planning, diagnoses often. As reflected in DSM-5, diagnoses denote fairly homogenous conditions, children with common symptoms, underlying causes, natural histories, and (to a degree) proven treatments. It is the essence of the nomothetic approach. Accordingly, at least in part, psychological practices should be based on what has been learned about the nature of childhood psychopathology, including how to diagnose and treat it, via application of the scientific method. When knowledge is acquired from the systematic study of many, many children the resulting knowledge can logically be applied as a starting place to help an individual child. For these reasons, one cornerstone of assessment-treatment matching is finding a DSM-5 diagnosis (or denoting a simpler mental health condition like from the list you found in Chapter 1). Of course, idiographic considerations (e.g., students’ strength, unique classroom considerations) will also warrant consideration, especially at Tier 3. You will see more about this in the upcoming case of Lex.

Considering the Medical Model Criticism

A common criticism at this point (i.e., as soon as DSM-5 is mentioned) is that school psychologists should avoid use of “the medical model.” Sadly, accusing behavioral scientists of embracing the medical model often brings critical thinking to a halt. But you are encouraged to engage thinking with the Reflective System and consider the following (rather than relying on immediate emotional reactions, intuition, or simply what others might have said about DSM in schools; i.e., counting on the Automatic System). It is true that we behavioral scientists possess our own valuable procedures. We might study antecedents and consequences to improve overt behavior without resorting to clinical diagnoses (reflecting an idiographic approach). Alternatively, we might understand the unique feelings and perceptions of a student and use that information to capitalize on his strengths to formulate a one-of-a-kind intervention (also reflecting an idiographic approach). These are very often good things to do. But they are hardly the only things to do. An alternative, or complementary, approach is to: (a.) understand (nomothetically) a condition, (b.) determine whether that problem is likely to self-correct, and (c.) if it is not, investigate clearly-specified treatment options and determine whether they should be implemented based on potential payoff, risk of complications, stigma and cost. It’s true that many of us as mental health care providers use logic like this—we do so because this is the rational, scientific, empirical, nomothetic method. But because medicine also happens to use this approach—and this is the key point—the method itself cannot logically be said to be synonymous with practicing like a physician. The approach is merely synonymous with something scientific. After all, psychologists are not accused of practicing via an “astronomical model” just because they use a normal distribution to characterize uncertainty (like Carl Friedrich Gauss, who discovered it to understand his observations of celestial bodies with a telescope). Likewise, psychologists are not accused of practicing with a “meteorological model” just because they deal with distributions of data points and characterize those distributions with measures of central tendency (e.g., mean) and dispersion (e.g., standard deviation) in a manner not dissimilar to that used by meteorologists keeping track of rainfall and temperatures.

Let’s return to the practical. Diagnoses can do more than just indicate fixed intervention programs. They can help reveal why classroom problems appear and what might be devised specifically to help ameliorate them (even if that process is imperfect). Consider Lex, a third grader who was referred with poor work completion, tumbling grades and mounting tension with his teacher. His teacher had the following to say. “Lex can be completely exasperating when he becomes stubborn and non-compliant. I have to tell him the same thing over and over. It looks like he is listening but later he goes back and does the very same thing. His style is one that I would call passive resistance. Just more or less ignore but never really confront me. He hides his schoolwork and refuses to start when instructed to do so. When I ask for an explanation, that’s when I get open defiance. He won’t tell me. It’s not for lack of language. I know he can be very verbal and even articulate. I’ve heard him explain things to other boys on the playground. I’ve tried to improve his behavior. I’ve turned his light from green to yellow to red several times. No effect.”

The school psychologist first hypothesized that this teacher lacked classroom management skills or perhaps that Lex had an enduring problem with compliance and cooperation, something akin to oppositional defiant disorder (ODD). However, once she worked through the various stages of the HR process, she reached a much different conclusion. Background information contraindicated a personal history of non-compliance. Instead, she discovered a personal and family history of anxiety, worry, and perfectionism. Rating scales similarly pointed toward internalizing rather than externalizing characteristics. Classroom observations depicted a quiet, somewhat withdrawn youngster. Most telling, however, was a clinical interview. During it, Lex reveal obsessive worry about making mistakes, a strong desire to repeatedly erase and correct his seatwork, plus hoarding of completed assignments for fear that they may be found to be imperfect if turned in. Critically, repeated corrections and reluctance to turn in papers relate to what he described as a need to have his work products “just right.” Lex described the need to have things executed to his own exact liking so powerful that his teacher’s requests could simply not be honored, “I know she sometimes gets mad at me, but I can’t help myself.” Consulting her DSM-5, Lex’s school psychologist concluded that he warranted no diagnosis of ODD, but instead either satisfied or nearly satisfied criteria for obsessive-compulsive disorder (OCD).

Understanding associated with that diagnosis, in turn, placed Lex’s classroom behavior in context. Deliberate slowness, indecisiveness, and hoarding schoolwork are likely manifestations of OCD, not ODD. Although Lex’s behavior appeared to reflect intentional defiance, the root of this behavior was something else. In light of the insight derived from Lex’s diagnosis, intelligent (assessment-informed) planning for him could then occur. First, straightforward classroom management strategies aimed to encourage compliance and improve rule following would have been misguided. Second, his classroom teacher would seem to benefit from understanding the distinction between willful refusal and internally driven obsessions and compulsions. It would seem crucial to process assessment findings with her, perhaps even using a simplified variant of the bottom-up approach seen in Chapter 13. Third, classroom behavior strategies suitable for students with OCD-spectrum problems would be suggested. These might include, for example, use of a timer (shortening a ritual), picking up work regardless of its apparent completion (preventing a ritual), repeatedly reminding Lex that perfectionism is neither expected nor desired (cognitive restructuring). Fourth, counseling/psychotherapy using CBT, an evidenced-based treatment, would be advised (see Joyce-Beaulieu & Sulkowski, 2015; Weisz & Kazdin, 2017). If CBT were implemented, an on-campus provider might be involved. Alternatively, and depending on the wishes of Lex’s family, an off-campus specialized provider might be used. Fifth, sharing a copy of Lex’s report with his primary care physician would make sense. They may wish to discuss with parents the possibility of psychopharmacology, a treatment option documented to be popular in clinic settings and for which there is evidence of efficacy (Smith, McBride & Storch, 2017). Sixth, providing parents with literature on OCD seems both humane and potentially therapeutic. Although unlikely alone to solve the problem, it can help (Cowan & Swearer, 2004).

Again, it is critical to recognize a role for both idiographic and nomothetic perspectives as assessment and treatment begin to coalesce. Regarding the former perspective, group level knowledge about OCD informs the action of Lex individually (i.e., Lex acts like many other students with OCD). Regarding the latter perspective, an interview with Lex reveals that he harbors personal feelings about being wrong and losing face if he submits imperfect seatwork. Although these considerations are shaded by his diagnosis of OCD, some aspects of his perceptions, feelings, and reactions are those of Lex uniquely. A thorough evaluation (and combining both nomothetic and idiographic approaches) sets the stage for intelligent intervention that is arguably impossible without a proper assessment.

Don’t Forget Cultural Considerations in Planning Treatment

It would take someone skilled in implementing mental health treatments to truly help Lex. It would also require someone with circumspection and knowledge of the systems that operate in schools. But there is even more to consider. Lex is a child of European-American background. This may be fortuitous for Lex’s school psychologist because many EBTs were validated on just such youngsters. But what if he were of Hispanic or African-American background? Extra caution or additional digging for details about interventions would then be needed (Huey & Polo, 2018). Unfortunately, there is relatively little treatment efficacy research that has used predominately ethnic minority youth as participants, verified favorable outcomes for them as subgroups, or has provided disaggregated statistical analysis that can easily inform practitioners (Pina, Polo & Huey, 2019). In addition, even when school professionals are committed to providing mental health services, the diverse skills and extensive staffing profiles sometimes found in clinics can be hard to replicate on a school campus. Thus, the manualized nature of EBTs may need to be revised to fit school usage (Coleman, 2018). In other words, it might be ill advised to simply find and then apply whole cloth an EBT in Weisz and Kazdin (2017) or to do the same thing regarding one from the (Division 53) Evidence-based Mental Health Treatment for Children and Adolescents website. Instead, school psychologists might be better off to select delimited elements of EBTs (i.e., modules). Modules could then be used as circumscribed elements, depending on what local resources and students’ characteristics warrant. This is a complex topic that is often subsumed under Tier 2 or Tier 3 interventions in multitiered systems of school mental health services (Wiest, Eber et al. 2018).

What Flora learned about treatment options and matching them to assessment findings left her feeling empowered. Not surprisingly, she concluded that any single approach to individual counseling, especially one boasting limited treatment effectiveness for most of her caseload (i.e., client-centered therapy), would no longer suffice. Still, disquieting feelings persisted about a professional schedule that involved so much individual counseling and group-level social skills training. Happily, however, a senior colleague directed her toward a comprehensive source whose target audience is school psychologists. Plotts and Lasser’s (2020) School Psychologist as Counselor, argues for role expansion that envisions school psychologists as direct mental health providers. Equally important, the resource, which is published by NASP, covers topic such as informed consent, ethics associated with providing counseling in schools, development of counseling goals and objectives, how to organize counseling sessions, tracking progress as well as information on terminating counseling services. Although she is a recent graduate, Flora was never fully trained on these topics, and lack of formal preparation might similarly prove to be barriers to DM3 members (and colleagues) poised to expand their roles. Flora concluded that counseling is a legitimate school psychology practice but one that should be blended with PBIS. Speaking with Emily, it became apparent that some of her mental health services could be written into students’ IEPs as a related service (see Chapter 10). Included might be oversight and direct treatment. Also, she can see that if many students needed intervention in groups, then group-level interventions might be part of a Tier 1 or Tier 2 services, as indicated by Kimani.

Summary

There are several things that school psychologists can do to expand their roles into the mental health sphere. One is to become active any time a student expresses a discernible social-emotional problem that ought to be addressed in an IEP (or Section 504 accommodation plan). School psychologists could not only help advise about plans but monitor their success. In this regard, several broadband and narrowband scales can be used recurrently and coupled with a statistical technique entitled the Reliable Change Index. Another thing that school psychologists can do is to use psychometric tools to screen all students for social-emotional problems. Universal screening might aid in the proper referral (and ultimate detection) of students with ED. Perhaps even more important, screening results can sometimes be harnessed to a much broader system of social-emotional services. Screening can help inform which student might fit from which MTSS level. Finally, students can be helped when their assessment results are envisioned as more than a tool for special education gatekeeping—when findings help prescribe interventions. This is true because not all types of emotional problems require the same type of intervention. School psychologists can benefit students by accessing off-the-shelf scientific information about disorder-intervention links (i.e., a nomothetic approach). Alternatively, they can sometimes match interventions to specialized emotional, contextual, and behavioral considerations to formulate a unique plan (i.e., an idiographic approach).