Learn with Us
At the Evidence Resource Center, we aim to help Pennsylvania’s school communities thoughtfully apply high-quality, relevant research to their own settings. Explore these topics to brush up on research basics, ESSA tiers, and the fundamentals of evidence-based decision making.
Elements of Evidence
See also:
To lead educational change in an evidence-forward culture, it's important to be able to recognize the fundamentals of high-quality evidence. Below, we've curated resources that can help education leaders re-learn research concepts that are important to school improvement, and master the art (and science) of evidence.
Study Design
Federal evidence standards place a high degree of emphasis on study design – the way that a research study's experimental groups are formed, and how the treatments (in our case, educational strategies) are administered to those groups.
Study Design Types
This diagram shows types of study designs:
- Experimental: Participants are randomly assigned to treatment and control groups, before intervention begins
- Quasi-Experimental: Non-random but purposeful assignment to treatment and control groups
- Correlational: A statistical association is measured between an intervention and an outcome
- Other: Such as survey data, case studies, or other qualitative measures where a test has occurred
Recognizing Research Methods
Quantitative Research
Quantitative research involves collecting and analyzing data in the form of numbers. These values are collected through measurements and reported with statistical analyses. The highest-quality quantitative data comes from experiments using treatment and control groups, but quantitative data can also come from surveys.
You'll hear words like:
- experimental
- quasi-experimental
- randomized
- randomized controlled trial
- correlational
Qualitative Research
Qualitative research attempts to understand its sample population through observation and/or dialogue. Several methods can be used to collect qualitative data, from one-on-one interviews or focus groups, to direct observation, to the use of anecdotes or personal experiences. (Denzin and Lincoln 1994, p. 14)
You'll hear words like:
- case study
- focus group
- interviews
- participant observation
Understanding Group Formation & Equivalence
Random assignment is the highest-quality group formation, because it ensures that the groups are as similar as possible before they receive an intervention. But in education, random assignment can be hard to achieve, for several reasons:
Teachers/school leaders can face an ethical dilemma, desiring to move a certain student into the group receiving the treatment because (s)he is struggling.
Self-selection (such as an after-school program students choose to attend) can complicate results, as students who self-select into an intervention may have other inherent characteristics that impact their achievement.
Differences in classrooms — from the teacher delivering the intervention, to the time of day the class session takes place — may impact results.
These (and others) are known as confounding factors. When looking for high-quality evidence, you're looking for groups that have had as few confounding factors impacting the groups as possible, and that the researcher has controlled for those confounding factors when calculating their results.
What does the Tier 1 Group Formation look like?
- Uncompromised random assignment (Equal chances of being in the treatment or control group)
- Low attrition (no adding, switching, or dropping out of the group)
- At least 350 participants in the sample
- Multiple sites (more than one school*)
What does the Tier 2 Group Formation look like?
- Groups are formed by Matching (statistical methods to create groups), Before and After (using time to create comparison groups), or Nonequivalent (nonrandom assignment to groups)
- Baseline equivalence is established (ensuring groups are similar on key factors before beginning the intervention)
- Low attrition (no adding, switching, or dropping out of the group)
- At least 350 participants in the sample
- Multiple sites (more than one school*)
What does the Tier 3 Group Formation look like?
- Groups are formed by nonrandom methods
- Statistical controls are used to account for selection bias and other confounding factors
Note: ESSA is not specific about Tier 4 group formation requirements.
*The What Works Clearinghouse and American Institutes for Research (and therefore, the ERC) define a site as one school building; the USDOE's ESSA non-regulatory guidance defines a site as an LEA, locality, or state. Your state's definition may vary.