From the iSTEEP Senior Scientist, Joseph Witt, PhD

Research Notes

Brief Notes on Science and Practice from Joseph Witt, PhD, Senior Scientist

Why Research is Important

School professionals have an increased awareness of the way the term "research based" is being used by publishers.  It has become a completely meaningless phrase in recent years.  Every type of intervention, assessment, or curriculum product now carries the label research based.  In particular, intervention publishers shamelessly proclaim that their products are based upon research despite a complete absence of scientifically based research. 

Reasonable care can be taken by school-based professionals to determine if a product has a scientifically acceptable research base.  The primary indicator is simple:  does the product have peer reviewed published studies indicating it does what it purports to do?  The scientific peer review process includes a process whereby research is reviewed by people who are disinterested third parties that scrutinize studies for scientific merit.  Studies which are not sound do not get published in quality journals.

Very few products have a true research base because conducting the research and going through the peer review process can take 2-4 years.  What publishers present instead is "data" from a school here or there that used the product and got good results and everyone "liked" the program.  The problem with this claim is the "research" complied with no standards, the schools presented are specially selected, and it is not known if other products were being used at the same time. 

With RTI systems, ask for published peer reviewed data showing improved achievement as the result of implementation of that specific RTI model.  RTI outcomes should include improved performance on state tests and/or decreased need for special education.

Knowledgeable professionals sometimes question even the best conducted research because they perceive it to have been conducted in "Lab" somewhere that bears little resemblance to their school and their students.  In some cases, this is true.  Some of the more mature tools available, however, have been extensively field tested in "real" schools. 

Why does the research matter to schools?  Research is important in considering services for students for two reasons.  First, you can start with tools that have been shown to be effective.  Federal laws such as NCLB and IDEA require this.  Second, for something to actually be effective in a school, it has to be used and used with fidelity.  For a tool to be used it helps if everyone believes it can work because this helps professionals to the day to day activities, without which, even the most effective tools do not succeed.  By communicating to professionals that the tool is effective it helps to create a sense of efficacy which can help each person with their personal decision to use the tool with fidelity. 

STEEP has two type of peer reviewed published research.  There is research on the model as a whole indicating improved achievement (e.g., state test scores) and reduced need for special education. In addition to research on the model as a whole, there is research on the major components or parts of STEEP.  For example, STEEP uses one  probe rather than three for universal screening and there is award winning research to support this tactic (See Ardoin, et al 2004).  Similar research exists for other components of STEEP including the procedures for determining and appropriate intervention. See the research page for more information pertaining to the research support for STEEP.  Click here.  STEEP is, of course, research based but STEEP is research proven with multiple peer reviewed papers. 

 

Interpreting CBM Data:  GOM and Skill Assessment

iSTEEP was built on a foundation which is both philosophical and research supported that each and every student can learn and improve.  Important academic outcomes, such as literacy and numeracy, are viewed within a hierarchy of skills from rudimentary (phonemic awareness) to basic (oral reading fluency) to advanced (interpreting fine literature).  These skills or behaviors are arranged in an instructional hierarchy which, when properly understood and used, takes students to very high levels of achievement. 

This view has influenced the development and nature of iSTEEP assessments.  The assessments used in RTI are most frequently those which derive from curriculum-based measurement (CBM).  CBM type assessments originated in Precision Teaching (PT) with individuals such as Clay Starlin using oral reading fluency probes and phonemic awareness probes in the late 1960’s and early 1970’s prior to the “invention” of CBM by Stan Deno in the late 1970’s.  The use of assessment in iSTEEP is more aligned with the original goals of PT than with CBM as it has evolved. 

A major difference between PT and CBM is the manner in which performance criteria are established used.   Binder (1990) conducted an analysis of CBM and PT and observed differences in how standards for performance (now called benchmarks) are developed and used.  PT uses performances standards.  A performance standard is the level of performance for a skill that will support retention, maintenance and application of the skill.  Data are used to set performance standards for important prerequisite skills such as phonemic awareness and oral reading fluency so that acquisition of these tool skills leads to smooth and quick progression through a curriculum or skill hierarchy. 

In contrast CBM uses norms (e.g., class, school, district, or national averages) to set standards.  Binder (1990) suggested this may be a “dangerous practice”: 

If an entire class performs below the mastery level (i.e., that level of performance required to support effective function) then the class norm is not a fair mastery criterion… For example, most competent adults can write answers to between 70 and 100 simple addition problems in a minute. Few classrooms provide either the materials or sufficient practice to enable students to achieve this level, although children in Precision Teaching classrooms routinely do so. We know that students will often come up to high expectations, or settle for low ones. If our objective is merely to keep students from falling below the average, to keep them out of the "special needs" category, then the CBM strategy may suffice. But if we seek to support true mastery at each step in the curriculum, to help all children become masterful students, then we must use performance criteria that are objective definitions of competence.  (Binder, 1990).

These differing notions of competence are most obvious in the definition of fluency used by the two systems.  Tindal (1989), and advocates of CBM, have indicated that in CBM "There is no objective standard of fluency. We have to know the normative information." PT, as notedpreviously, has shown in several studies that “fluency represents an objective standard of performance that can be determined objectively: the level of speed plus accuracy sufficient to ensure retention, endurance and application of skills and knowledge” (Binder, 1990).          

In advocating the use of norms to set standards, CBM has also adoptedthe “normal curve” mentality common to education and psychology.  With a construct such as “intelligence,” for example, individuals significantly below the norm and significantly below the norm can be labeled (gifted vs person with a disability).  CBM has applied this philosophy to skills such as oral reading fluency and students below the norm are “at risk”.  This logic is sometimes blindly applied regardless of whether the district is a high performing or a severely challenged district even though the score required to be markedly below the norm is quite different in the two districts.  With PT, a student needs intervention based upon an independently derived standard which would be the same in both districts.

The ultimate expression of the values and use of CBM was the development of the concept of General Outcome Measure (GOM).  As the CBM group was conducting statistical analyses of CBM data they found the many skills, such as Oral Reading Fluency, correlate well with other measures such as reading achievement tests.  Because of this correlation, CBM assessments were thought to be good measures of “overall reading ability” and labeled General Outcome Measures.  This was a significant inferential leap because with a wave of the statistical wand a one minute assessment of oral reading fluency skill became a “valid” way to get a good overall assessment of student reading ability.   

The foregoing has provided a context into which iSTEEP assessments and intervention may be situated.  iSTEEP has evolved out of the PT tradition and has adopted a methodology for interpretation of assessment data derived from the research on PT and the good work done by Ed Lentz, Ed Daly, Brian Martens, Chris Skinner and many others.  As such, assessment data are translated into instructional plans using a combination of objective performance standards and the instructional hierarchy.  Hence, within iSTEEP oral reading fluency, for example, is not viewed as a GOM or a general measure of overall reading performance.  Rather, oral reading is viewed as a skill or behavior which is important by itself but gains added strength when situated within a skill hierarchy leading to reading competence.  Intervention planning utilizes knowledge about oral reading fluency and other skills to determine where intervention should begin, what should be taught first, and when intervention can be ended.  

This is not meant to be overly critical of those who choose to use CBM assessments as GOM’s.  However, it should be emphasized that the evidence for CBM assessments as a GOMs is mostly correlational.  Statistically significant correlations are not difficult to obtain because students who score high on one measure tend to score high on a second measure and the same goes for students who score low.  If an overall measure of reading is needed, then use a standardized achievement test designed for that purpose and not a one minute probe.