IES lowcost shortduration FY2017

Connec-ng Research, Policy and Prac-ce

84.305L: Low-‐Cost, Short-‐Dura7on Evalua7on of Educa7on Interven7ons 84.324L: Low-‐Cost, Short-‐Dura7on Evalua7on of Special Educa7on Interven7ons

Phill Gagne Allen Ruby

Na-onal Center for Educa-on Research

Kimberley Sprague

Na-onal Center for Special Educa-on Research ies.ed.gov

Overview •  Overview of IES and its mission •  Requirements •  Speciﬁcs –  Purpose –  The project narra-ve •  •  •  •  • 

Signiﬁcance Partnership Research Plan Personnel Resources

•  Other important sec-ons of the applica-on •  Preparing and submiFng an applica-on ies.ed.gov

2

Legisla-ve Mission of IES •  Describe the condi-on and progress of educa-on in the United States •  Iden-fy educa-on prac-ces that improve academic achievement and access to educa-on opportuni-es •  Evaluate the eﬀec-veness of Federal and other educa-on programs

ies.ed.gov

3

Organiza-onal Structure of IES Oﬃce of the Director Standards & Review Oﬃce

Na7onal Center for Educa7on Evalua7on ies.ed.gov

Na7onal Center for Educa7on Research

Na7onal Center for Educa7on Sta7s7cs 4

Na7onal Board for Educa7on Sciences

Na7onal Center for Special Educa7on Research

IES Grant Programs: Research Objec-ves •  Develop or iden-fy educa-on interven-ons (i.e., prac-ces, programs, policies, and approaches) –  that enhance academic achievement –  that can be widely deployed

•  Iden-fy what does not work and thereby encourage innova-on and further research •  Understand the processes that underlie the eﬀec-veness of educa-on interven-ons and the varia-on in their eﬀec-veness ies.ed.gov

5

Partnerships & IES Priori-es IES seeks to... •  Encourage educa-on researchers to develop partnerships with stakeholder groups to advance relevance of research and usability of its ﬁndings for day-‐to-‐day work of educa-on prac--oners and policymakers •  Increase capacity of educa-on policymakers and prac--oners to use knowledge generated from high quality data analysis, research, and evalua-on through wide variety of communica-on and outreach strategies

(See hYp://ies.ed.gov/director/board/priori-es.asp) ies.ed.gov

6

Short Descrip-on

•  Carried out by Partnerships

–  New or established –  Minimum: research ins-tu-on and a state or local educa-on agency

•  Purpose

–  Carry out rigorous evalua-ons of educa-on interven-ons implemented by state or local educa-on agencies •  High importance to the educa-on agency •  Use secondary data (e.g., administra-ve data) •  Low-‐cost: maximum grant of $250,000 •  Short-‐dura-on: 2 years

ies.ed.gov

Impetus for Low-‐Cost Grant Program •  Take advantage of opportuni-es to use administra-ve data to do evalua-ons •  Provide useful informa-on to educa-on agencies in a more -mely manner than tradi-onal evalua-ons •  Create addi-onal opportuni-es for research ins-tu-ons and educa-on agencies to work together •  Iden-fy the strengths, weaknesses, and applicability of this type of evalua-on ies.ed.gov

General Requirements •  Focus on student educa-on outcomes

–  84:305L: For students from prekindergarten through postsecondary and adult educa-on –  84.324L: For infants/toddlers through students in grade 12 with or at-‐risk for disability

•  Research occurs in an authen-c educa-on seFng •  Evaluate educa-on interven-ons using secondary data •  Partnership between research ins-tu-ons and state and local educa-on agencies •  Disseminate ﬁndings in ways useful to agency decision-‐making ies.ed.gov

Student Popula-on •  84.305L: Students from prekindergarten through postsecondary and adult educa-on •  84.324L: Students from infants/toddlers through grade 12 with or at-‐risk for disability –  A student with a disability is deﬁned in Public Law 108-‐446, the Individuals with Disabili-es Educa-on Improvement Act of 2004 (IDEA) –  Addi-onal requirements for iden-fying students at risk for developing a disability –  see h%p://ies.ed.gov/ncser/deﬁni3on.asp ies.ed.gov

10

Focus on Student Educa-on Outcomes •  Research must address educa-on outcomes of students. For both 305L and 324L these include –  Academic outcomes –  Social and behavioral competencies that support student success in school

•  For 324L, these also include –  Developmental, func-onal, and transi-onal outcomes for students with or at-‐risk for disability ies.ed.gov

11

Infants/Toddlers 324L

Developmental outcomes pertaining to cogni-ve, Student Outcomes communica-ve, linguis-c, social, emo-onal, adap-ve, func-onal or physical development.

Prekindergarten 305L & 324L

School readiness (e.g., pre-‐reading, language, vocabulary, early math and science knowledge, social and behavioral competencies) Developmental outcomes

324L

ies.ed.gov

12

Grade K -‐ 12 305L & 324L

324L

Outcome Student Outcomes Learning and achievement in reading, wri-ng, mathema-cs, and science; Progress through the educa-on system (e.g., course and grade comple-on or reten-on, high school gradua-on, and dropout); Social and behavioral competencies important to academic and post-‐academic success. Func-onal outcomes that improve educa-onal results; Transi-ons to employment, independent living, and postsecondary educa-on.

ies.ed.gov

13

Addi-onal 305L Student Outcomes Grade Postsecondary

(Grades 13 – 16) (baccalaureate and sub-‐baccalaureate)

Outcome Access to, persistence in, progress through, and comple-on of postsecondary educa-on; for students in developmental programs, addi-onal outcomes include achievement in reading, wri-ng, English language proﬁciency, and mathema-cs; success in gateway math and science courses, introductory English composi-on

Adult Educa7on

Student achievement in reading, wri-ng, English (Adult Basic Educa-on, language proﬁciency, and mathema-cs; access to, Adult Secondary persistence in, progress through, and comple-on Educa-on, Adult ESL, of adult educa-on programs and HS equivalency ies.ed.gov prepara-on)

14

Educa-on Interven-ons •  The wide range of educa-on curricula, instruc-onal approaches, professional development, technology, and prac-ces, programs, and policies that are implemented at the child/student, classroom, school, district, state, or federal level to improve student educa-on outcomes •  The interven-on is of high importance to the SEA or LEA •  The implementa-on of the interven-on is managed or overseen by the SEA or LEA (not just allowing a researcher or organiza-on to implement the interven-on) •  Implementa-on of the interven-on must occur in Year 1 of the project •  The Interven-on is expected to produce meaningful improvement in student educa-on outcomes within a short period (e.g., within a quarter, semester, or year) •  At a minimum, the administra-ve data (or other secondary source) contains student educa-on outcomes (primary data collec-on not supported by grant) ies.ed.gov

Applica-ons must be from a Partnership •  Applica-ons must include at least one Principal Inves-gator (PI) from a research ins-tu-on and at least one PI from a U.S. state or local educa-on agency –  PI from research ins7tu7on: Must have the ability and capacity to conduct scien-ﬁcally valid research and exper-se in the educa-on issue to be addressed –  PI from state or local educa7on agency: Must have decision-‐making authority for the issue within his or her agency ies.ed.gov

16

Partnership •  Partnership may be new or exis-ng –  Expected to con-nue through the project and, perhaps, agerward

•  Research ins-tu-on has a broad deﬁni-on –  Ability and capacity to conduct scien-ﬁcally valid research

ies.ed.gov

17

Partnership: SEAs •  State educa-on agencies –  Examples: educa-on agencies, departments, boards, commissions –  Oversee early learning, elementary, secondary, postsecondary and/or adult educa-on –  For 305L, oversee infant and child care, and/or early interven-on services –  Also includes educa-on agencies in District of Columbia, the Commonwealth of Puerto Rico, and each of the outlying areas ies.ed.gov

Partnership: LEAs •  Local educa-on agencies which are primarily public school districts •  County or city agencies that have primary responsibility for infant and child care (324L), early interven-on services (324L), or prekindergarten •  Community college districts •  Tribal educa-on agencies •  State and city postsecondary systems –  Individual postsecondary ins-tu-ons cannot serve as a partner –  A postsecondary system that applies as an educa-on agency partner cannot also serve as the research ins-tu-on partner in the same project

•  Adult educa-on providers (deﬁned under WIOA) can serve as the partner when there is no state or local educa-on agency for adult educa-on

ies.ed.gov

19

Addi-onal Partners •  Partnerships may include more than one state or local educa-on agency if they share similari-es and interests •  Intermediary/service districts that provide services to mul-ple districts but do not have decision-‐making authority over implemen-ng programs and policies, •  Non-‐educa-on state and local agencies may be partners as long as an educa-on agency is a partner •  Partnerships may include more than one research ins-tu-on if they have shared interests and make unique contribu-ons •  Partnerships may include other non-‐research organiza-ons (e.g., issue-‐oriented or stakeholder groups) that will contribute to the partnership and its work ies.ed.gov

20

Dissemina-on/Products •  Projects are to aid educa-on agencies in decision-‐making •  Required dissemina-on –  Oral brieﬁng on results to educa-on agency –  WriYen brief, wriYen for non-‐technical audience, made available free to public

•  Recommended dissemina-on –  Partner presenta-ons to academic and prac--oner audiences –  Partner publica-ons in academic and prac--oner journals –  Toolkit or guide for other educa-on agencies on how to conduct a similar study

ies.ed.gov

Check the Fit of Your Research and Low-‐Cost, Short-‐Dura-on Evalua-on Grant Program •  If you are not looking at student outcomes, then IES is not the appropriate funding agency •  If you need -me and eﬀort to build a partnership and prepare for an evalua-on, consider: –  Researcher-‐Prac--oner Partnerships under 84.305H

•  If the interven-on you want to evaluate

Is not implemented by a state or local educa-on agency, Cannot be evaluated using secondary data, Will not be implemented in Year 1 of the project, or Is not expected to improve student outcomes within a short period (e.g., a quarter – a year). –  Then consider the: –  –  –  – 

•  Educa-on Research Grants Program (84.305A) or •  Special Educa-on Research Grants Program (84.324A)

ies.ed.gov

Low-‐Cost Evalua-on: Purpose •  Promote joint evalua-on research by research ins-tu-ons and state and local educa-on agencies –  On an educa-on interven-on iden-ﬁed as having great importance by the educa-on agency –  That includes prac--oner input into the research –  That will provide -mely rigorous evidence for the agency’s decision-‐making regarding the interven-on –  And the results of which will be broadly disseminated in ways easily accessible to researchers, prac--oners, and the public

ies.ed.gov

23

What should the partnerships do during the grant? •  Iden-fy a speciﬁc educa-on interven-on •  Implemented by an SEA or LEA in Year 1 of project •  Of high priority to that agency •  Intended to improve student educa-on outcomes within a year

•  Carry out an evalua-on of that interven-on –  –  –  – 

Using a RCT or an RDD design (or a SCD under 324L) Using secondary data Es-mate overall impacts If data available: •  Es-mate subgroup impacts for important subgroups •  Examine other moderators and mediators of interest, ﬁdelity of implementa-on, and comparison group prac-ce

ies.ed.gov

24

Expected Products of the Grant •  Causal evidence of the impact of a clearly speciﬁed interven-on implemented by an SEA or LEA –  Overall impacts –  Impacts for available subgroups of interest

•  Advice for the SEA or LEA –  Con-nuing and/or expanding the use of the interven-on –  Further research needs, e.g., •  Evalua-on, e.g., varia-on in impacts, modera-on and media-on, generalizability, replica-on •  Development, e.g., modiﬁca-ons to the interven-on or its implementa-on ies.ed.gov

25

The Project Narra-ve

(Maximum of 15 Pages) •  •  •  •  • 

Signiﬁcance Partnership Research Plan Personnel Resources

ies.ed.gov

26

Signiﬁcance •  The educa-on interven-on to be evaluated –  The educa-on problem/issue the interven-on is to address within the SEA/LEA •  Relevance to other SEAs or LEAs (secondary importance)

–  Components of the interven-on –  Ra-onale for why the interven-on can improve student outcomes within a short period (e.g., 1 quarter -‐ 1 year) •  May include theory of change •  Diﬀerence from status quo •  Related ﬁndings from previous studies and how this study will improve upon past work ies.ed.gov

Signiﬁcance •  The implementa-on of the interven-on

–  Who will implement it and how will it be implemented •  The educa-on agency will implement or will oversee implementa-on –  Adequate funding available for implementa-on –  Implementa-on during Year 1 of the project at a level expected to impact student outcomes

•  Sources of secondary data to be used in the evalua-on

–  How these data are collected –  How these data will be obtained by researchers by the 1st quarter of Year 2 of the project

ies.ed.gov

The Project Narra-ve

(Maximum of 15 Pages) •  •  •  •  • 

Signiﬁcance Partnership Research Plan Personnel Resources

ies.ed.gov

29

Partnership •  Describe the partners –  The research ins-tu-on and the educa-on agency •  Oﬃces or divisions within the agency whose coopera-on is necessary

–  Any other members of the partnership –  Common interest in and beneﬁt from this evalua-on –  The process through which the partners determined the speciﬁc interven-on to evaluate –  Data sharing agreement – the strategy to obtain the secondary data and provide it for analysis by the 1st quarter of the second year ies.ed.gov

The Project Narra-ve (Maximum of 15 Pages) •  •  •  •  • 

Signiﬁcance Partnership Research Plan Personnel Resources

ies.ed.gov

31

Research Plan •  State research ques-ons and hypotheses •  Describe sample and seFng –  Deﬁne popula-on and how your sample and sampling procedures will allow inferences to the popula-on –  Exclusion and inclusion rules and their jus-ﬁca-on –  Strategies used to increase par-cipa-on and reduce aYri-on –  Describe the seFng and its implica-ons for the generalizability of your study

ies.ed.gov

32

Research Plan: Design •  Discuss how design will support causal inferences and iden-fy poten-al threats to internal validity •  Discuss how degree of equivalence at baseline will be determined •  Discuss possibility of bias from overall and diﬀeren-al aYri-on •  305L: Required use of RCT or RDD –  Poten-al to meet WWC evidence standards without reserva-ons

•  324L: Required use of RCT, RDD, or Single-‐Case Experimental Design

ies.ed.gov

33

Research Plan: Design Randomized Controlled Trial (RCT) •  Note unit of randomiza-on and jus-fy choice •  Describe process for random assignment and maintaining its integrity •  Diﬀerent Approaches to RCTs -‐ Poten-al Issues –  En-re popula-on-‐mandatory: Treatment ﬁdelity –  Volunteers: Comparison group status –  LoYeries: AYri-on of non-‐accepted par-es –  Staggered roll out: LiYle -me for true comparison –  Varia-ons of program/policy: Issue of overall signiﬁcance ies.ed.gov

34

Research Plan: Design Regression Discon;nuity Design (RDD) •  Appropriateness of assignment variable •  Show true discon-nuity •  Discuss possibility of manipula-on of design variable and analyses to determine such manipula-on •  Sensi-vity analyses to assess inﬂuence of key procedural or analy-c decisions on results

ies.ed.gov

35

Research Plan: Design Single-‐Case Experimental Design (324L Only) •  Jus-fy the use of a single-‐case experimental design as opposed to an RCT or RDD (e.g., a focus on students with a low-‐incidence disability) •  Describe the repeated, systema-c measurement of a dependent variable before, during, and ager the ac-ve manipula-on of an independent variable (i.e., interven-on) •  Include outcome measures that are not strictly aligned with the interven-on •  Describe any quan-ta-ve analy-c techniques, in addi-on to visual analysis, for analyzing the resul-ng data (e.g., between-‐case eﬀect size calcula-ons) ies.ed.gov

36

Research Plan: Sta-s-cal Power •  Detailed descrip-on of power analysis –  Jus-fy method used to calculate power –  Jus-fy parameters used and assump-ons made

•  Provide power for main analyses and important subgroup analyses •  Along with iden-fying minimum detectable eﬀect for your analysis, jus-fy its –  Reasonableness –  Prac-cal meaning

•  Reviewers should be able to check power calcula-ons ies.ed.gov

37

Research Plan: Outcome Measures •  Student educa-on outcome measures relevant to states, districts, and schools –  Found in administra-ve data or other secondary data

•  Discuss reliability, validity, and appropriateness •  Must be collected during Year 1 of project –  Addi-onal data from previous years of interven-on’s implementa-on may also be used if appropriate to the evalua-on design

•  Clearly link measures to ra-onale for the interven-on ies.ed.gov

38

Research Plan: Op-onal Measures •  If available in secondary data, describe measures of –  Intermediate outcomes –  Moderators (subgroups expected) –  Mediators (intermediate outcomes) –  Fidelity of implementa-on –  Comparison group prac-ce

ies.ed.gov

Research Plan: Analysis •  Detail impact analyses –  Make clear how analyses directly answer your research ques-ons –  Show that analyses are based on the design –  Address clustering of students in classrooms in schools –  Address missing data –  If mul-ple datasets are to be linked, detail how this will be done

•  Describe any other analyses to be done (e.g., subgroups, other moderators, mediators, and ﬁdelity of i mplementa-on) ies.ed.gov 40

The Project Narra-ve (15 Page Maximum) •  •  •  •  • 

Signiﬁcance Partnership Research Plan Personnel Resources

ies.ed.gov

41

Personnel •  Iden-fy all key personnel on the project team

–  The PI from the research ins-tu-on who has previous experience carrying out the proposed evalua-on design (RCT, RDD, or SCD) –  The PI from the educa-on agency who makes program decisions –  Other key personnel

•  Roles and responsibili-es on the project –  –  –  – 

Each individual’s roles and responsibili-es on the project Their qualiﬁca-ons (i.e., exper-se and experience) for their role Their % FTE on the project Past success at working in similar partnerships

•  PI qualiﬁca-ons for managing a grant of this type •  Ensure objec-vity of evalua-on ies.ed.gov

42

Resources: To Conduct the Project •  Describe the ins-tu-onal resources of all the ins-tu-ons involved in the partnership and how these resources will contribute to building the partnership and to the research –  Ins-tu-onal capacity to manage the grant –  Resources available at the partner ins-tu-ons to support the project –  Plans to acquire any major resources not yet in hand (e.g., secondary data) –  Joint LeYer of Agreement by partners (Appendix D) –  LeYer of Agreement to provide administra-ve data (App. D) ies.ed.gov

43

Resources: Dissemina-on of Results •  Results expected to be useful to the SEA/LEA partner and, perhaps, other SEA/LEAs –  Findings of both beneﬁcial impacts or no impacts

•  Describe your capacity and resources to disseminate ﬁndings –  Required dissemina-on through an oral brieﬁng for the agency and a wriYen brief freely available to the public –  Dissemina-on to other audiences (e.g., researchers, policymakers, prac--oners, students and their families, public) ies.ed.gov

44

Other Important Sec-ons of the Applica-on •  •  •  •  • 

Appendix A Appendix B Appendix C Appendix D Budget & Budget Narra-ve

ies.ed.gov

45

Appendix A Page Limit: 3 •  If you are resubmi;ng an applica3on, use up to 3 pages to discuss how you responded to reviewer comments

ies.ed.gov

46

Appendix B (Op-onal) Page Limit: 5 •  Figures, charts, or tables that supplement the project narra-ve •  Timelines for the project (very useful) •  Examples of instruments used in the collec-on of the administra-ve or other secondary sources of data •  Do NOT include narra3ve text

ies.ed.gov

Appendix C (Op-onal) Page Limit: 5 •  Examples of materials used in the interven-on: –  curriculum materials –  computer screen shots –  training documents –  assessment items –  other materials

•  Do NOT include narra3ve text ies.ed.gov

48

Appendix D (Required) No Page Limit •  Required LeYers of Agreement –  Joint LeYer from the research ins-tu-on and the SEA/LEA •  Document par-cipa-on and coopera-on in the partnership •  Set out each’s roles and responsibili-es under the project

–  LeYer from the oﬃce in charge of the agency’s data •  Project will have access to data required in -me to do analysis

•  Op-onal LeYers of Agreement –  Separate LeYers from other organiza-ons taking part –  LeYers from any consultants and schools taking part ies.ed.gov

49

Budget & Budget Narra-ve •  Maximum project length is 2 years •  Maximum award is $250,000 –  Funds must be used for evalua-on only (e.g., cannot be used for implementa-on of the interven-on or primary data collec-on) –  Award size depends on project scope

•  Include a detailed budget form (SF 424) AND a budget narra-ve that links the ac-vi-es, personnel, etc. from the Project Narra-ve to the funds requested ies.ed.gov

50

Preparing Your Applica-on •  Important dates •  Informa-on sources –  Read the RFA –  Talk with a program oﬃcer

•  Review process

ies.ed.gov

Important Dates & Deadlines Applica7on Deadline August 4, 2016 4:30:00 PM DC Time

ies.ed.gov

LeWer of Applica7on Start Dates Intent Due Package Date Posted May 19, May 19, 2016 July 1 to 2016 September 1, 2017

52

Informa-on Sources •  Request for Applica-ons –  hYp://ies.ed.gov/funding/

•  Abstracts of Projects –  hYp://ies.ed.gov/funding/grantsearch/index.asp

•  Applica-on Package –  www.grants.gov

•  Program Oﬃcers –  [email protected] 84.324L –  [email protected] 84.305L –  [email protected] 84.305L

ies.ed.gov

53

Peer Review (Standards & Review Oﬃce) •  Compliance screening for format requirements •  Responsiveness screening for program requirements •  Assignment to review panel –  2 to 3 reviewers (substan-ve and methodological) –  The most compe--ve proposals are reviewed by full panel •  Many panelists will be generalists to your topic •  Panels contain experts in relevant methodologies

–  Panel provides an overall score plus speciﬁc scores on Signiﬁcance, Partnership, Research Plan, Personnel, and Resources ies.ed.gov

54

No-ﬁca-on •  All applicants will receive e-‐mail no-ﬁca-on that the following informa-on is available via the Applicant No-ﬁca-on System (ANS): •  Status of award •  Reviewer summary statement

•  If you are not granted an award the ﬁrst -me, consider resubmiFng and talking with your Program Oﬃcer

ies.ed.gov

55

For More Informa-on

hYp://ies.ed.gov/funding

Kimberley Sprague Na-onal Center for Special Educa-on Research [email protected]

Phill Gagne Allen Ruby Na-onal Center for Educa-on Research [email protected] [email protected]

ies.ed.gov

56