Research Notes
Brief Notes on Science and Practice from Joseph Witt, PhD, Senior
Scientist
School professionals have an increased awareness
of the way the the term "research based" is being used by publishers.
It has become a completely meaningless phrase in recent years.
Every type of intervention, assessment, or curriculum product now
carries the label research based. In particular, intervention
publishers shamelessly proclaim that their products are based upon
research despite a complete absence of scientifically based research.
Reasonable care can be taken by school-based
professionals to determine if a product has a scientifically acceptable
research base. The primary indicator is simple: does the
product have peer reviewed published studies indicating it does what it
purports to do? The scientific peer review process includes a
process whereby research is reviewed by people who are disinterested
third parties that scrutinize studies for scientific merit.
Studies which are not sound do not get published in quality journals.
Very few products have a true research base
because conducting the research and going through the peer review
process can take 2-4 years. What publishers present instead is
"data" from a school here or there that used the product and got good
results and everyone "liked" the program. The problem with this
claim is the "research" complied with no standards, the schools
presented are specially selected, and it is not known if other products
were being used at the same time.
With RTI systems, ask for published peer
reviewed data showing improved achievement as the result of
implementation of that specific RTI model. RTI outcomes should
include improved performance on state tests and/or decreased need for
special education.
Knowledgeable professionals sometimes question
even the best conducted research because they perceive it to have been
conducted in "Lab" somewhere that bears little resemblance to their
school and their students. In some cases, this is true. Some
of the more mature tools available, however, have been extensively field
tested in "real" schools.
Why does the research matter to schools?
Research is important in considering services for students for two
reasons. First, you can start with tools that have been shown to
be effective. Federal laws such as NCLB and IDEA require this.
Second, for something to actually be effective in a school, it has to be
used and used with fidelity. For a tool to be used it helps
if everyone believes it can work because this helps professionals to the
day to day activities, without which, even the most effective tools do
not succeed. By communicating to professionals that the tool is
effective it helps to create a sense of efficacy which can help each
person with their personal decision to use the tool with fidelity.
STEEP has two type of peer reviewed published
research. There is research on the model as a whole indicating
improved achievement (e.g., state test scores) and reduced need for
special education. In addition to research on the model as a whole,
there is research on the major components or parts of STEEP. For
example, STEEP uses one probe rather than three for universal
screening and there is award winning research to support this tactic
(See Ardoin, et al 2004). Similar research exists for other
components of STEEP including the procedures for determining and
appropriate intervention. See the research page for more information
pertaining to the research support for STEEP.
Click here.
iSTEEP was built on a foundation which is both philosophical and
research supported
that each and every student can learn and improve.
Important academic outcomes, such as literacy and numeracy, are
viewed within a hierarchy of
skills from rudimentary (phonemic awareness) to basic (oral reading
fluency) to advanced
(interpreting fine literature).
These skills or behaviors are arranged
in an instructional hierarchy which, when properly understood and used, take students to very high levels of achievement.
This view has influenced the development and nature of iSTEEP assessments.
The assessments used in RTI are most frequently those which derive from
curriculum-based measurement
(CBM). CBM type assessments
originated in Precision Teaching (PT) with individuals such
as Clay Starlin using oral reading fluency probes and phonemic awareness
probes in the late 1960’s and early 1970’s prior to the “invention” of
CBM by Stan Deno in the late 1970’s.
The use of assessment in iSTEEP is more aligned
with the original goals of PT than with CBM as it has evolved.
A major difference between PT and CBM is the manner
in which performance criteria are established used.
Binder (1990) conducted
an analysis of CBM and PT and observed
differences in how standards for performance (now called
benchmarks) are developed and
used. PT
uses performances standards.
A performance standard is the level of performance for a skill that will
support retention, maintenance and application of the skill.
Data are used to set
performance standards for important prerequisite skills such as phonemic
awareness and oral reading fluency so that acquisition of these tool
skills leads to smooth and quick progression through a curriculum or
skill hierarchy.
In contrast CBM uses norms (e.g., class, school,
district, or national averages) to set standards.
Binder (1990) suggested
this may be a “dangerous practice”:
If an entire class performs below the mastery level
(i.e., that level of performance required
to support effective function) then the class norm is not a fair mastery
criterion… For example, most competent adults can write answers to
between 70 and 100 simple addition problems in a minute. Few classrooms
provide either the materials or sufficient practice to enable students
to achieve this level, although children in Precision Teaching
classrooms routinely do so. We know that students will often come up to
high expectations, or settle for low ones. If our objective is merely to
keep students from falling below the average, to keep them out of the
"special needs" category,
then the CBM strategy may suffice. But if we seek to support true
mastery at each step in the curriculum, to help all children become
masterful students, then we must use performance criteria that are
objective definitions of competence.
(Binder, 1990).
These differing notions of competence are most
obvious in the definition of fluency used
by the two systems. Tindal
(1989), and advocates of CBM, have indicated
that in CBM "There is no objective standard of fluency. We have to know
the normative information." PT, as noted
previously, has shown in several studies that “fluency represents an
objective standard of performance that can be determined
objectively: the level of speed
plus accuracy sufficient to ensure retention, endurance and application
of skills and knowledge”
(Binder, 1990).
In advocating the use of norms to set standards,
CBM has also adopted the “normal curve” mentality common to education and psychology.
With a construct such as “intelligence,” for example, individuals
significantly below the norm and significantly below the norm can be
labeled (gifted vs person with a disability).
CBM has applied
this philosophy to skills such as oral reading fluency and students
below the norm are “at risk”.
This logic is sometimes blindly applied
regardless of whether the district is a high performing or a severely
challenged district even though the score required to be markedly
below the norm is quite different in the two districts.
With PT, a student needs
intervention based upon an
independently derived
standard which would be the same in both districts.
The ultimate expression of the values and use of
CBM was the development of the concept of General Outcome Measure (GOM).
As the CBM group was conducting statistical analyses of CBM data
they found the many skills, such as Oral Reading Fluency, correlate well
with other measures such as reading achievement tests.
Because of this correlation, CBM assessments were thought to be
good measures of “overall reading ability” and labeled
General Outcome Measures.
This was a significant inferential leap because with a wave of the
statistical wand a one minute assessment of oral reading fluency skill
became a “valid” way to get a good overall assessment of student reading
ability.
The foregoing has provided a context into which iSTEEP assessments and
intervention may be situated.
iSTEEP has evolved
out of the PT tradition and has adopted
a methodology for interpretation of assessment data derived
from the research on PT and the good work done by Ed Lentz, Ed Daly,
Brian Martens, Chris Skinner and many others.
As such, assessment data are translated
into instructional plans using a combination of objective performance
standards and the instructional hierarchy.
Hence, within iSTEEP oral reading fluency, for example, is not
viewed as a GOM or a general measure of overall reading
performance. Rather, oral
reading is viewed as a skill
or behavior which is important by itself but gains added strength when
situated within a skill
hierarchy leading to reading competence.
Intervention planning utilizes knowledge
about oral reading fluency and other skills to determine where
intervention should begin, what should be taught first, and when
intervention can be ended.
This is not meant to be overly critical of those
who choose to use CBM assessments as GOM’s.
However, it should be emphasized that the evidence for CBM assessments as a GOMs is
mostly correlational.
Statistically significant correlations are not difficult to obtain
because students who score high on one measure tend to score high on a
second measure and the same goes for students who score low.
If an overall measure of reading is needed, then use a standardized
achievement test designed for
that purpose and not a one minute probe.