Connec-ng Research, Policy and Prac-ce
Evalua&on of State and Local Educa&on Programs & Policies (84.305H)
Allen Ruby, Ph.D.
Na-onal Center for Educa-on Research
Jacquelyn Buckley, Ph.D.
Na-onal Center for Special Educa-on Research
ies.ed.gov
Overview • Overview of IES and its mission • Requirements • Specifics – Purpose – The project narra-ve • • • • •
Significance Partnership Research Plan Personnel Resources
• Other important sec-ons of the applica-on • Preparing and submiNng an applica-on ies.ed.gov
2
Legisla-ve Mission of IES • Describe the condi-on and progress of educa-on in the United States • Iden-fy educa-on prac-ces that improve academic achievement and access to educa-on opportuni-es • Evaluate the effec-veness of Federal and other educa-on programs
ies.ed.gov
3
Organiza-onal Structure of IES Office of the Director Standards & Review Office
Na&onal Center for Educa&on Evalua&on ies.ed.gov
Na&onal Center for Educa&on Research
Na&onal Center for Educa&on Sta&s&cs 4
Na&onal Board for Educa&on Sciences
Na&onal Center for Special Educa&on Research
IES Grant Programs: Research Objec-ves • Develop or iden-fy educa-on interven-ons (i.e., prac-ces, programs, policies, and approaches) – that enhance academic achievement – that can be widely deployed
• Iden-fy what does not work and thereby encourage innova-on and further research • Understand the processes that underlie the effec-veness of educa-on interven-ons and the varia-on in their effec-veness ies.ed.gov
5
Partnerships & IES Priori-es IES seeks to... • Encourage educa-on researchers to develop partnerships with stakeholder groups to advance relevance of research and usability of its findings for day-‐to-‐day work of educa-on prac--oners and policymakers • Increase capacity of educa-on policymakers and prac--oners to use knowledge generated from high quality data analysis, research, and evalua-on through wide variety of communica-on and outreach strategies
(See h\p://ies.ed.gov/director/board/priori-es.asp) ies.ed.gov
6
Short Descrip-on
• Evalua-on of State and Local Programs & Policies (State/Local Evalua-on) – New or established partnerships – Carry out rigorous evalua-ons of educa-on programs or policies (from pre-‐k to adult educa-on) that are implemented by state or local educa-on agencies ies.ed.gov
General Requirements • Focus on student educa-on outcomes • Partnership between research ins-tu-ons and state or local educa-on agencies (SEAs/LEAs) • Evaluate SEA/LEA educa-on programs and policies
ies.ed.gov
Focus on Student Outcomes • IES funds research to improve the quality of educa-on for all students through advancing the understanding of and prac-ces for teaching, learning, and organizing educa-on systems • Research must address educa-on outcomes of students – Academic outcomes – Social and behavioral competencies that support student success in school ies.ed.gov
9
Student Popula-on • Students from prekindergarten through postsecondary and adult educa-on – Typically developing students – Students with disabili-es or at risk for disabili-es • see h%p://ies.ed.gov/ncser/defini3on.asp for specific requirements for iden-fying students at risk for disabili-es status
ies.ed.gov
10
Ul-mate Outcomes of Interest: Student Outcomes Grade Prekindergarten (center based)
Outcome School readiness (e.g., pre-‐reading, language, vocabulary, early math and science knowledge, social and behavioral competencies)
Kindergarten – Grade 12
Learning, achievement, and higher-‐order thinking in reading, wri-ng, mathema-cs, and science; progress through the educa-on system (e.g., course and grade comple-on or reten-on, high school gradua-on, and dropout); social and behavioral competencies important to academic and post-‐academic success.
ies.ed.gov
11
Ul-mate Outcomes of Interest: Student Outcomes Grade Postsecondary
(Grades 13 – 16) (baccalaureate and sub-‐baccalaureate)
Outcome Access to, persistence in, progress through, and comple-on of postsecondary educa-on; for students in developmental programs, addi-onal outcomes include achievement in reading, wri-ng, English language proficiency, and mathema-cs; success in gateway math and science courses, introductory English composi-on
Adult Educa&on
Student achievement in reading, wri-ng, English (Adult Basic Educa-on, language proficiency, and mathema-cs; access to, Adult Secondary persistence in, progress through, and comple-on Educa-on, Adult ESL, of adult educa-on programs and HS equivalency prepara-on) ies.ed.gov
12
Addi-onal Outcomes for Students With or At-‐Risk for Disabili-es • Applicants are encouraged to also include outcomes accepted under the NCSER grant programs
• Development outcomes for younger students – cogni-ve, communica-ve, linguis-c, social, emo-onal, adap-ve, func-onal or physical development
• Func-onal outcomes for older students – that improve educa-onal results and transi-ons to employment, independent living, and postsecondary educa-on ies.ed.gov
Applica-ons must be from a Partnership • Applica-ons must include at least one Principal Inves-gator (PI) from a research ins-tu-on and at least one PI from a U.S. state or local educa-on agency – PI from research ins&tu&on: Must have the ability and capacity to conduct scien-fically valid research and exper-se in the educa-on issue to be addressed – PI from state or local educa&on agency: Must have decision-‐making authority for the issue within his or her agency ies.ed.gov
14
Partnership • Partnership may be new or exis-ng • Research ins-tu-on has a broad defini-on – Ability and capacity to conduct scien-fically valid research
ies.ed.gov
15
Partnership: SEAs • State educa-on agencies – Examples: educa-on agencies, departments, boards, commissions – Oversee early learning, elementary, secondary, postsecondary and/or adult educa-on – Also includes educa-on agencies in District of Columbia, the Commonwealth of Puerto Rico, and each of the outlying areas
ies.ed.gov
Partnership: LEAs • Local educa-on agencies which are primarily public school districts • Community college districts • Tribal educa-on agencies • State and city postsecondary systems – If there is a state or city higher educa-on agency that oversees the postsecondary system, include them as an agency partner – If there is no state or city educa-on agency that oversees the postsecondary system, the system can apply as the sole agency partner – A postsecondary system that applies as an educa-on agency partner cannot also serve as the research ins-tu-on partner in the same project
• Adult educa-on providers (defined under WIOA) can serve as the partner when there is no state or local educa-on agency for adult educa-on
ies.ed.gov
17
Addi-onal Partners • Partnerships may include more than one state or local educa-on agency if they share similari-es and interests • Intermediary or service districts that provide services to LEAs • Non-‐educa-on state and local agencies may be partners as long as an educa-on agency is a partner • Partnerships may include more than one research ins-tu-on if they have shared interests and will make unique contribu-ons • Partnerships may include other non-‐research organiza-ons (e.g., issue-‐oriented or stakeholder groups) that will contribute to the partnership and its work ies.ed.gov
18
Evaluate SEA/LEA Educa-on Programs and Policies • Program or policy of high importance to the SEA or LEA • Substan-al modifica-on of exis-ng prac-ce • Implemented by the SEA or LEA • Implemented under rou-ne condi-ons
ies.ed.gov
Check the Fit of Your Research and the State Evalua-on Grant Program • If you are not looking at student outcomes, then IES is not the appropriate funding agency for your research • If you need -me and effort to build a partnership and carry out ini-al research in prepara-on for an evalua-on, consider: – Researcher-‐Prac--oner Partnerships under 84.305H
• If the program/policy you want to evaluate is not implemented by or of high importance to the SEA/ LEA, consider: – Educa-on Research Grants Program (84.305A) or – Special Educa-on Research Grants Program (84.324A) ies.ed.gov
State/Local Evalua-on: Purpose • Promote joint evalua-on research by research ins-tu-ons and SEAs/LEAs – On an educa-on program/policy of key importance to SEA/LEAs – That will directly contribute to SEA/LEA program and policy decisions – Provide opportuni-es to develop the partnership through the evalua-on
• Foster longer-‐term research partnerships – Provide and support the use of rigorous research-‐based evidence in decision making – Con-nue prac--oner input into research agenda ies.ed.gov
21
What should the partnerships do during the grant? • Iden-fy an educa-on program or policy • Implemented by an SEA or LEA • Of high priority to that agency • Intended to improve student educa-on outcomes
• Carry out an evalua-on of that program/policy – Overall impacts – Subgroup impacts – Fidelity of implementa-on – Cost analysis ies.ed.gov
22
Expected Products of the Grant • Causal evidence of the impact of a clearly specified program/policy implemented by an SEA – Overall impacts – Impacts under a variety of condi-ons and/or by subgroup
• Conclusions on and revisions to the theory of change that guides the program/policy – Contribu-ons to our theore-cal understanding of educa-on processes and procedures
ies.ed.gov
23
Expected Products of the Grant • If a beneficial impact is found… – The organiza-onal supports, tools, and procedures needed for sufficient implementa-on of the core components of the program/policy under rou-ne prac-ce should be iden-fied
• If a beneficial impact is not found… – A determina-on should be made whether and what type of further research would be useful to revise the program/ policy and/or its implementa-on
• The financial costs of the program/policy ies.ed.gov
24
The Project Narra-ve • • • • •
Significance Partnership Research Plan Personnel Resources
ies.ed.gov
25
Significance In the Significance sec2on, clearly describe… • The educa-on program or policy to be evaluated – Components – Processes and materials to support implementa-on – Evidence it is ready to be or already implemented – How it differs from exis-ng prac-ce in the same loca-on or different loca-ons
ies.ed.gov
Significance • Its implementa-on – By an SEA or LEA – The target popula-on and sites – End users of the program or policy and how they are to carry it out – Under rou-ne condi-ons
ies.ed.gov
Significance • The theory of change – How the program or policy is to effect changes that ul-mately lead to beneficial impacts on student outcomes – Intermediate outcomes in this process
• Ra-onale for tes-ng its impact on student educa-on outcomes – In widespread use but not well-‐evaluated – A component of the program/policy – Explain why the program/policy is likely to produce improved student outcomes rela-ve to other prac-ce • Theore-cal jus-fica-on • Empirical work ies.ed.gov
The Project Narra-ve • • • • •
Significance Partnership Research Plan Personnel Resources
ies.ed.gov
29
Describe the Partnership • Describe the partners – The research ins-tu-on and the educa-on agency – Any other members of the partnership – Common interest in and benefits from this evalua-on – The process through which they decided to propose a State/Local Evalua-on project – Past or ongoing collabora-ons and results from them – Management structure and procedures to keep the project on track and ensure quality of the research – Data sharing and housing agreement ies.ed.gov
Partnership Development Plan • Partnership’s decision-‐making process • Improving the educa-on agency’s capacity to par-cipate in and use educa-on research – Iden-fy the agency’s interests in capacity building – The agency’s specific understanding of the proposed research design and the validity and generaliza-on of the evidence provided from it – The agency’s general capacity to understand and use research
ies.ed.gov
The Project Narra-ve • • • • •
Significance Partnership Research Plan Personnel Resources
ies.ed.gov
32
Research Plan • State research ques-ons and hypotheses • Describe sample and seNng – Define popula-on and how your sample and sampling procedures will allow inferences to the popula-on – Exclusion and inclusion rules and their jus-fica-on – Strategies used to increase par-cipa-on and reduce a\ri-on – Describe the seNng and its implica-ons for the generalizability of your study
ies.ed.gov
33
Research Plan: Design Ra2onale for the Selected Research Design • Causal inference • Threats to internal validity • Describing and jus-fying the counterfactual • Degree of equivalence at baseline • Bias from overall and differen-al a\ri-on • Meet WWC evidence standards (with or without reserva-ons) ies.ed.gov
34
Research Plan: Design Preferred Design: Randomized Controlled Trial (RCT) • Note unit of randomiza-on and jus-fy choice • Describe process for random assignment and maintaining its integrity • Different Approaches to RCTs -‐ Poten-al Issues – En-re popula-on-‐mandatory: Treatment fidelity – Volunteers: Comparison group status – Lo\eries: A\ri-on of non-‐accepted par-es – Staggered roll out: Li\le -me for true comparison – Varia-ons of program/policy: Issue of overall significance ies.ed.gov
35
Research Plan: Design Alterna2ves to the RCT Design • If RCT is not possible, jus-fy why • Alterna-ves to minimize or model selec-on bias – Regression discon-nuity designs – Well-‐designed quasi-‐experimental designs • Compara-ve interrupted -me series
ies.ed.gov
36
Research Plan: Sta-s-cal Power • Detailed descrip-on of power analysis and jus-fica-on for method used to calculate power – Including assump-ons
• Power for main analyses and important subgroup analyses • Prac-cal meaning of minimum detectable effect sizes • Reviewers should be able to check power calcula-ons
ies.ed.gov
37
Research Plan: Outcome Measures • Student educa-on outcome measures relevant to states, districts, and schools – Onen found in administra-ve data – Can include researcher-‐developed measures but not as the primary outcome measures
• Provide reliability, validity, and appropriateness • Include measures of intermediate outcomes – For example, if program/policy is to change instruc-on, describe measures of instruc-on
• Link measures to theory of change ies.ed.gov
38
Research Plan: Moderators & Mediators • May explain differen-al impacts of interven-on • Iden-fied in theory of change • Describe how they will be measured in both treatment and control • Discuss if doing exploratory or confirmatory analysis of each one examined • Describe analysis plan
ies.ed.gov
39
Value of Mixed Methods (Quan-ta-ve & Qualita-ve) • To be\er understand any causal rela-onship – To avoid relying on non-‐empirical based explana-ons
• To understand the fidelity of implementa-on and comparison group prac-ce • To examine intermediate outcomes and mediators • Have a team that can do both
ies.ed.gov
Research Plan: Fidelity of Implementa-on • Describe your plan for determining the fidelity of implementa-on of the program or policy • Describe how your fidelity measures capture core components of the program or policy – Note their psychometric proper-es • Discuss how data will be analyzed and will contribute to overall evalua-on • Ini-al study of fidelity can be used to provide input to SEA/LEA to improve implementa-on ies.ed.gov
41
Research Plan: Comparison Group Prac-ce • Describe who makes up comparison group • Detail how you will measure whether they are similar/different from treatment group • Detail how you will measure what they receive in place of the treatment • Determine if control group receives components similar to interven-on and how much
ies.ed.gov
42
Research Plan: Analysis Detailed descrip2on of data analysis procedures • Make clear how analyses directly answer your research ques-ons and can be done based on the design • Quan-ta-ve: Sta-s-cal procedures, model, and sonware • Qualita-ve: Methods to index, summarize, and interpret data • Will quan-ta-ve and qualita-ve data be used for separate or combined analyses? • Address clustering of students in classrooms in schools • Address missing data • Include plans for analyses of subgroups, mediators, moderators, and fidelity of implementa-on • Examine a\ri-on (overall and differen-al) ies.ed.gov
43
Research Plan: Cost Analysis • Document financial costs of program implementa-on – Detailed enough for another SEA/LEA to use
• Can include a cost-‐effec-veness or cost-‐benefit analysis but not required
ies.ed.gov
44
The Project Narra-ve • • • • •
Significance Partnership Research Plan Personnel Resources
ies.ed.gov
45
Personnel • Iden-fy all key personnel on the project team – Roles and responsibili-es on the project – Qualifica-ons (i.e., exper-se and experience) to carry out the roles and responsibili-es – % FTE on the project (one key person should have enough -me to maintain progress of project) – Past success at working in similar partnerships
• PI qualifica-ons for managing a grant of this size and type • Ensure objec-vity of evalua-on ies.ed.gov
46
Resources • Describe the ins-tu-onal resources of all the ins-tu-ons involved in the partnership and how these resources will contribute to building the partnership and to the research – Ins-tu-onal capacity to manage the grant – Resources available at the partner ins-tu-ons that will be used – Plans to acquire any major resources not yet in hand – Joint Le\er of Agreement by partners (Appendix D)
ies.ed.gov
47
Resources • If districts or schools are taking part… – Districts and schools should document their involvement – E.g., Le\ers of Agreement in Appendix D
• If secondary data is being analyzed… – The organiza-on holding those data should document their willingness to provide the data – E.g., Le\ers of Agreement in Appendix D
• If district/school staff are taking part… – E.g., through surveys, observa-ons, logs – Discuss how their coopera-on will be obtained (e.g., use of incen-ves) and their current knowledge of the project ies.ed.gov
48
Resources: Dissemina-on of Results • Results are expected to be useful to the SEA/LEA partner and other SEAs/LEAs – Both findings of beneficial impacts or no impacts
• Describe your capacity to disseminate findings • Iden-fy all your audiences and how you will disseminate the results to them – The SEA/LEA (through an ongoing process) – Other educa-on agencies, policymakers, and prac--oners – The research community – The public ies.ed.gov
49
Other Important Sec-ons of the Applica-on • • • • •
Appendix A Appendix B Appendix C Appendix D Budget & Budget Narra-ve
ies.ed.gov
50
Appendix A Page Limit: 3 • If you are resubmi;ng an applica3on, use up to 3 pages to discuss how you responded to reviewer comments
ies.ed.gov
51
Appendix B Page Limit: 15 • Figures, charts, or tables that supplement the project narra-ve • Timelines for the project (very useful) • Examples of measures to be used – E.g., tests, surveys, observa-on, and interview protocols • Do not include narra-ve text
ies.ed.gov
Appendix C Page Limit: 10 • Examples of materials used in the program or policy: – curriculum material – computer screen shots – training documents – assessment items – other materials
ies.ed.gov
53
Appendix D No Page Limit • Le\ers of Agreement from all the research partners – Joint Le\er from key partners – Separate Le\ers from other organiza-ons involved – Le\ers should clearly state the organiza-on’s expected role in the partnership and their commitments to the project – Similar le\ers from any consultants, districts, and schools taking part – Le\ers from holders of data should make clear that the data described in the applica-on will be provided for the proposed use by the project ies.ed.gov
54
Budget & Budget Narra-ve • Maximum project length is 5 years • Maximum award is $5 million – Funds must be used for evalua-on only (e.g., cannot be used for implementa-on of the program/policy) – Award size depends on project scope
• Include a detailed budget form (SF 424) and a budget narra-ve that links the ac-vi-es, personnel, etc. from the Project Narra-ve to the funds requested ies.ed.gov
55
Important Dates & Deadlines Applica&on Deadline August 4, 2016 4:30:00 PM DC Time
ies.ed.gov
LeWer of Applica&on Start Dates Intent Due Package Date Posted May 19, May 19, 2016 July 1 to 2016 September 1, 2017
56
Informa-on Sources • Request for Applica-ons – h\p://ies.ed.gov/funding/
• Abstracts of Projects – h\p://ies.ed.gov/funding/grantsearch/index.asp
• Applica-on Package – www.grants.gov
• Program Officers –
[email protected] –
[email protected] ies.ed.gov
57
Peer Review (Standards & Review Office) • Compliance screening for format requirements • Responsiveness screening for program requirements • Assignment to review panel – 2 to 3 reviewers (substan-ve and methodological) – The most compe--ve proposals are reviewed by full panel • Many panelists will be generalists to your topic • Panels contain experts in relevant methodologies
– Panel provides an overall score plus specific scores on Significance, Partnership, Research Plan, Personnel, and Resources ies.ed.gov
58
No-fica-on • All applicants will receive e-‐mail no-fica-on that the following informa-on is available via the Applicant No-fica-on System (ANS): • Status of award • Reviewer summary statement
• If you are not granted an award the first -me, consider resubmiNng and talking with your Program Officer
ies.ed.gov
59
For More Informa-on
h\p://ies.ed.gov/funding
Jacquelyn Buckley Na-onal Center for Special Educa-on Research
[email protected]
Allen Ruby Na-onal Center for Educa-on Research
[email protected]
ies.ed.gov
60
