Connecting Research, Policy and Practice
IES Grant Writing Workshop June 3, 2015 Christina Chhin, Ph.D. National Center for Education Research Amy Sussman, Ph.D. National Center for Special Education Research
@IESResearch ies.ed.gov
Purpose of the Workshop This workshop will provide instruction and advice on writing a successful application to the Institute of Education Sciences’research grant programs, specifically to the: •Education Research Grants Program (84.305A) •Special Education Research Grants Program (84.324A)
ies.ed.gov
Grant Writing Is A Process
ies.ed.gov
3
Agenda • Introduction to IES • Grant Writing Tips • General Requirements • Grant Research Topics • Grant Research Goals • Four Sections of the Project Narrative
ies.ed.gov
4
What is IES? • Research arm of the U.S. Department of Education, non-partisan by law. • Charged with providing rigorous and relevant evidence on which to ground education practice and policy and share this information broadly. • By identifying what works, what doesn't, and why, we aim to improve educational outcomes for all students, particularly those at risk of failure. ies.ed.gov
IES Organizational Structure Office of the Director Standards & Review Office National Center for Education Statistics
ies.ed.gov
National Board for Education Sciences
National Center for Education Research
National Center for Education Evaluation
6
National Center for Special Education Research
FY 2016 Funding Opportunities
ies.ed.gov
7
Primary Research Grant Programs • Education Research Grants (84.305A) • Special Education Research Grants (84.324A) These grant programs are organized by research topic and research goal.
ies.ed.gov
8
RFA Changes of Note • For 84.305A only: – No Goal 2 applications – Maximum budget award amounts reduced for each Goal – For Goal 3, maximum duration increased to 5 years • For BOTH 84.305A and 84.324A: – Dissemination plans are required under the Resources section – Goal 3 (Efficacy and Replication): Exact replications of previous studies are welcome as well as replications that modify the conditions under which the intervention is implemented – Goal 4 (Effectiveness): Evidence from only one prior efficacy study is required (instead of two) ies.ed.gov
9
Agenda • Introduction to IES • Grant Writing Tips • General Requirements • Grant Research Topics • Grant Research Goals • Four Sections of the Project Narrative
ies.ed.gov
10
What You Need to Do Sell your research idea, promote yourself as the best person to do the research, and build goodwill and trust. How? By demonstrating that you know what the problem is and have a way to address it. ies.ed.gov
Opening Paragraph • Sets the scene for readers: – Identifies the significance of the work to be done and what actually will be done – Readers use it to organize information in rest of the application – You can lose your readers right off with an unclear opening
ies.ed.gov
Statement of Purpose Should: – Be short and attention-getting – Contain the problem statement and your contribution to solving it. Your fellow researchers, friends, and family members should be able to understand it and see its relevance. NOTE: It’s not as easy to do as you may think. You have to know a lot about what it is you want to do before you can describe it succinctly. ies.ed.gov
Theory of Change (ToC) • • • •
The model underlying your research. A roadmap to your project narrative. A source for generating research questions. Constantly evolving.
NOTE: Some fields and scholars use terms like Logic Model or Logical Framework to mean things similar to what we are calling a Theory of Change here.
ies.ed.gov
ToC & Your Research Plan In your research plan, you need to specify exactly what it is you’re exploring, creating, validating, or testing. You also need to specify how you will do these things. Strategies/Activities What are the pieces that you’ll be exploring, creating, testing, etc.? Outcomes Indicators: What will you measure, and how you will measure it? Populations: Who and where (both in treatment and control/comparison)? Thresholds: What effect (size) should you expect? Timeline: When should you be collecting what data? ies.ed.gov
Helpfulness of the Program Officer • Share your framework and statement of purpose with the Program Officer – Ensure you are submitting to the correct competition/topic – Springboard for further discussion
ies.ed.gov
Clarity of Writing • Readers (e.g., application reviewers) often complain about lack of clarity. – Significance too general – Lack of detail regarding intervention, development cycle, or data analysis – Use of jargon and assumptions of knowledge – Poor writing (e.g., grammar), awkward constructions, etc.
ies.ed.gov
Resources for Researchers • Visit links for faculty and researchers on IES website http://ies.ed.gov/resourcesforresearchers.asp
• Review past webinars and participate in future webinars for the FY 2016 competitions http://ies.ed.gov/funding/webinars/index.asp
ies.ed.gov
18
Agenda • Introduction to IES • Grant Writing Tips • General Requirements • Grant Research Topics • Grant Research Goals • Four Sections of the Project Narrative
ies.ed.gov
19
All Proposed Studies Must • • • • •
Measure Student Education Outcomes Be Relevant to Education in the U.S. Address Authentic Education Settings Specify 1 Research Topic Specify 1 Research Goal
ies.ed.gov
20
NCER Ultimate Outcomes of Interest: Student Outcomes Grade
Outcome
Prekindergarten
School readiness (e.g., pre-reading, language, vocabulary, early math and science knowledge, social and behavioral competencies)
Kindergarten – Grade 12
Learning, achievement, and higher-order thinking in reading, writing, mathematics, and science; progress through the education system (e.g., course and grade completion or retention, high school graduation, and dropout); social skills, attitudes, and behaviors that support learning in school
ies.ed.gov
21
NCER Ultimate Outcomes of Interest: Student Outcomes Grade Postsecondary (Grades 13 – 16)
Outcome Access to, persistence in, progress through, and completion of postsecondary education; for students in developmental programs, additional outcomes include achievement in reading, writing, English language proficiency, and mathematics
Adult Education
Student achievement in reading, writing, and (Adult Basic Education, mathematics; access to, persistence in, progress Adult Secondary through, and completion of adult education Education, Adult ESL, programs and GED preparation) ies.ed.gov
22
NCSER Ultimate Outcomes of Interest: Student Outcomes Grade
Outcome
Birth - 5
Developmental outcomes and school readiness
Kindergarten – High School
Achievement in core academic content (reading, writing, mathematics, science); behaviors that support learning in academic contexts; functional outcomes that improve educational results; and transitions to employment, independent living, and postsecondary education
ies.ed.gov
23
Who Should Read the RFA? • You, the proposed Principal Investigator • Your team members, including your co-PIs, statistician, methodologist, developer • Your assigned sponsored projects officer • Anyone else participating in the preparation of the application
ies.ed.gov
24
Agenda • Introduction to IES • Grant Writing Tips • General Requirements • Grant Research Topics • Grant Research Goals • Four Sections of the Project Narrative
ies.ed.gov
25
Grant Topics • All applications to the primary research grant programs must be directed to a specific topic – Note on SF 424 Form, Item 4b (Agency Identifier Number) – Note at top of Abstract and Project Narrative
ies.ed.gov
Research Topics Education Research (305A)
Special Education Research (324A)
Cognition & Student Learning
Autism Spectrum Disorders
Early Learning Programs and Policies
Cognition & Student Learning in SpEd
Education Technology
Early Intervention & Early Learning in SpEd
Effective Teachers & Effective Teaching
Families of Children with Disabilities
English Learners
Mathematics & Science Education
Improving Education Systems
PD for Teachers & Related Services Providers
Mathematics & Science Education
Reading, Writing & Language Development
Postsecondary & Adult Education
Social & Behavioral Outcomes to Support Learning
Reading & Writing
SpEd Policy, Finance, and Systems
Social & Behavioral Context for Academic Learning
Technology for Special Education
ies.ed.gov
Transition Outcomes for Secondary Students with Disabilities
Topics: Nota Bene • Must address student education outcomes • Grade range varies by topic • Your project might fit in more than one topic
ies.ed.gov
305A: Topics and Their Grade Range
ies.ed.gov
29
Choosing among Overlapping Topics • What literature are you citing? • To which topic is your area of expertise best aligned? • If your focus is on a specific population of students/teachers, go to that program/topic: – Is your focus on a specific type of student/teacher (e.g., English Learners), or are you studying them as a subgroup of your sample? ies.ed.gov
Agenda • Introduction to IES • Grant Writing Tips • General Requirements • Grant Research Topics • Grant Research Goals • Four Sections of the Project Narrative
ies.ed.gov
31
Grant Research Goals • All applications to 84.305A/84.324A must be directed to a specific goal – Note on SF 424 Form, Item 4b – Note at top of Abstract and Research Narrative
• The goal describes the type of research to be done • Every application is directed to a specific topic/goal combination ies.ed.gov
What Topic X Goal Fits Your Project?
ies.ed.gov
33
Program Officer Role • As you are thinking through which is the appropriate topic and goal for your project, we encourage applicants to reach out to your program officers. • They are a valuable resource.
ies.ed.gov
34
FY 2016 Research Goals • • • • •
Exploration Development & Innovation* Efficacy & Replication Effectiveness Measurement
* Only for 84.324A ies.ed.gov
35
Maximum Award Amounts Goal
Maximum: 324A (direct + indirect)
Maximum: 305A (direct + indirect)
Exploration With secondary data With primary data
2 years, $700,000 4 years, $1,600,000
2 years, $600,000 4 years, $1,400,000
Development & Innovation
4 years, $1,500,000
Not competed in FY 2016
Efficacy & Replication Follow-up study Retrospective
4 years, $3,500,000 3 years, $1,300,000 3 years, $800,000
5 years, $3,300,000 3 years, $1,100,000 3 years, $700,000
Effectiveness Follow-up study
5 years, $4,000,000 3 years, $1,600,000
5 years, $3,800,000 3 years, $1,400,000
Measurement
4 years, $1,600,000
4 years, $1,400,000
ies.ed.gov
36
NCER Grants by Goal (2004-2014)
ies.ed.gov
NCSER Grants by Goal (2006-2014)
ies.ed.gov
Goal Requirements • Your application must meet all Project Narrative and Award Requirements listed for the goal you select in order for your application to be considered responsive and sent forward to review. • We strongly encourage you to incorporate the recommendations into your Project Narrative. • All applications must include a Dissemination Plan. ies.ed.gov
39
Exploration Projects: Malleable Factors • Malleable factors must be under the control of the education system – Something that can be changed by the system
• Examples – Student characteristics: behavior, skills – Teacher characteristics: practices, credentials – School characteristics: size, climate, organization – Education interventions: practices, curricula, instructional approaches, programs, and policies ies.ed.gov
Development & Innovation Projects: Key Features • Iterative development process • Well specified theory of change • Data collected on feasibility and usability in authentic education settings • Fidelity must be measured • Pilot data on student outcomes ies.ed.gov
41
Efficacy & Replication Projects: Key Features • Testing a causal question • Ask what might be needed to implement intervention under routine practice, even if you intend to test under ideal conditions • Consider role of developer to avoid conflict of interest for developer-evaluators • Do not require confirmatory mediator analyses (primary research question) but recommend exploratory ones ies.ed.gov
42
Effectiveness Projects: Key Features • IES expects researchers to – Implement intervention under routine practice – Include evaluators independent of development/distribution – Describe strong efficacy evidence for intervention (from at least one previous efficacy study)
• Does not expect wide generalizability from a single study – Expects multiple Effectiveness projects to this end – Sample size is not a key distinction from Efficacy
• Does not require confirmatory mediator analyses but encourages exploratory ones • Cost of implementation is limited to 25% of budget ies.ed.gov
43
Measurement Projects: Key Features • Assessments may also be developed in other goals, but not as the primary focus • Primary product of measurement grant is the design, refinement, and/or validation of an assessment • Include an assessment framework • Must link the assessment to student education outcomes
ies.ed.gov
44
Expected Products • Expected Products for each goal can help you identify the right goal for your project • At the end of a funded project, IES expects you to provide…
ies.ed.gov
Agenda • Introduction to IES • Grant Writing Tips • General Requirements • Grant Research Topics • Grant Research Goals
• Four Sections of the Project Narrative
ies.ed.gov
46
Project Narrative • Four Required Sections – Significance – Research Plan
– Personnel – Resources
• Each of these sections will be scored individually by the peer reviewers • In addition, reviewers provide an overall score of Scientific Merit ies.ed.gov
47
The Application’s Project Narrative • • • •
Requirements vary by Topic & Goal READ THE REQUIREMENTS CAREFULLY 25 pages, single spaced Project Narrative is supported by Appendices, but all critical content for reviewers should be included within the 25 pages of the Project Narrative.
ies.ed.gov
Significance Section • Describes the overall project – Your research question to be answered, intervention to be developed or evaluated, or measure to be developed and/or validated • Provides a compelling rationale for the project – Theoretical justification • Theory of Change – Empirical justification – Practical justification ies.ed.gov
Significance • Do not assume reviewers know significance of your work • Do not quote back RFA on general importance of a topic, – e.g., RFA paragraph on lack of reading proficiency of 8th and 12th graders based on NAEP data
• Do quote back RFA if your project is addressing a research gap or consideration identified in the RFA – e.g., disproportionality in discipline (Social/Behavioral); impact of early childhood policy initiatives (Early Learning) ies.ed.gov
Significance – 2 Key Problem Areas 1. Description of Malleable Factor/Intervention – Unclear what intervention is: confuses reviewers • Many components and may be applied at different times – how fit together – Graphic may help
– Unclear how to be implemented to ensure fidelity – Intervention not shown to be strong enough to expect an impact • Especially true for information interventions – e.g., provide data on students, short teacher workshops
– Overly focused on actions not content • Ex.: 20 hours of PD held over 10 weeks but no detail on what is to be covered in the sessions ies.ed.gov
Significance – 2 Key Problem Areas 2. Theory of change – Why a malleable factor is expected to be related to a student outcome – Why the proposed intervention should improve outcomes versus current practice – Why an assessment/instrument will measure a specific construct – When well laid out, a theory of change makes clear what is expected to happen and in what order – Easy for reviewers to understand research plan – why measure certain outcomes – Graphic can be helpful ies.ed.gov
Theory of Change Should Describe: •
How the intervention addresses the need and why it should work
– – –
Content: what the student should know or be able to do; why this meets the need Pedagogy: instructional techniques and methods to be used; why appropriate Delivery System: how the intervention will arrange to deliver the instruction
•
Which aspects of the intervention are different from the counterfactual condition
•
Key factors or core ingredients most essential and distinctive to the intervention
ies.ed.gov
Simple Theory of Change
ies.ed.gov
54
Logic Model Graphics Don’t Do This! • Do not overwhelm the reader • Do not use color as a key because applications are reviewed in black and white
ies.ed.gov
[Processes 3-17 repeat twice in Year 2]
DEVELOPMENT MODEL FOR “WL” INTERVENTION DEVELOPED BY “ABC” WITH THE ASSISTANCE OF “DEF”
16. DEF interviews coaches, SLTs, and PLTs about PD and WL processes
13-15b. DEF observes/consul ts SLTs, documents implementation
15a.Coaches assist SLT in implementing PD with faculty
WL Debrief
WL Debrief
WL Debrief
Implement Prof Devt
PLT
PLT
WL Debrief
Begins weekly meetings
PLT
8a1. Coaches assist PLTs in using all data to ID area for prof devt
6-8b. DEF observes PLTs, documents implementation 7a. Coaches teach PLT to conduct Learning Walks (WL) 6a. Coaches facilitate PLT identification of annual instr. goal
PLT WL Debrief
approx. 3 wks after school begins
WL Debrief
PLT
13a.ABC researchers train SLT to research best-practices in PD area
PLT
PLT
WL Debrief
Profile Analysis Set instructional goals & WL focus
PLT
SLT 1
[4 weeks at end of prior year or beg of current year]
WL Debrief
Research [PLT appoints SLT1 to address PD Topic 1]
KEY = Begin Process
= Feedback for next process = Delegation of PD PLT
= Primary Leadership team
SLT
= Secondary Leadership team
ies.ed.gov
8a2. Coaches assist PLTs in weekly WLs and Debriefing through mid-year
PLT
PLT PLT
SLT 1
14a. SLT and Coaches create PD unit
18a. ABC and DEF submit Annual Report to DOE and schools
17. ABC reviews/revise s model based on DEF findings
PLT PLT
[Processes 4-11 WL repeat to mid-year] Debrief 12. Coaches & PLTs choose 2-4 teachers (based on WLs) to become Dynamic Leadership Team 1]
PLT WL Debrief
PLT WL Debrief
PLT
WL Debrief
WL Debrief
11. Coaches share/ implement revisions with PLTs
5. DEF guides Coaches and PLTs in constructing and analyzing profiles based on data collected 4. Coaches and DEF work with PLT to collect climate data
WL Debrief
10. ABC reviews/revise s model based on DEF findings
approx. 9 wks after school begins
9. DEF interviews coaches and PLTs about WL process
1a. PI Recruits and Trains Coaches 1b. DEF develops data collection tools
2a. ABC PI matches Coaches to schools (n=5)
3b. DEF develops electronic profile development tool 3a. Coaches collect 3 yrs. stud. ach. & demo. data per school
2b. DEF trains coaches to use all data tools
Research Plan Section • Describe the work you intend to do – How you will answer your research question; develop your intervention; evaluate the intervention, or develop and/or validate your assessment
• Make certain Research Plan is aligned to Significance section – All research questions should have justification in Significance
• Step-by-step process – A timeline is strongly recommended! ies.ed.gov
Identify Setting, Population, & Sample • Identify the places you will be doing research • Identify the population you are addressing • Identify the sample – Inclusion and exclusion criteria – Sample size (issues of power for analysis) – The importance of attrition and how to address it – External validity: can you generalize to your population or only to a subset of it • If using secondary data, discuss these for the datasets you will be using ies.ed.gov
Specify Your Outcome Measures • For both proximal and distal outcomes • Sensitive (often narrow, aligned with intervention) measures • Measures of broad interest to educators • Measures not expected to be linked can be used as additional evidence • Describe reliability, validity, and relevance • Do not include measures not linked to research questions • Consider issue of multiple comparisons ies.ed.gov
Specify Features of All Other Measures • Measures that feed back into iterative development process • Fidelity of Implementation – Operating as intended – Able to address comparison groups
• Feasibility
ies.ed.gov
Qualitative Measures • Describe – Actual items to be used – How items link to constructs – the validity of these measures – Procedures for collection and coding (address inter-rater reliability) – How qualitatively collected measures are used in analysis of quantitative outcomes (e.g., test scores)
ies.ed.gov
Measurement Projects • • • •
Alternate forms – horizontal equating Vertical equating, if measuring growth Test fairness Non-student instruments must be validated against student outcomes
ies.ed.gov
62
Analysis Depends on Design • Describe how your analysis answers your research questions • Describe analyses of qualitative data
ies.ed.gov
Analysis (cont.) • Show your model – Identify coefficients of interest and their meaning – Show different models for different analyses – Include Equations Yijk i j ij ijk
• • • •
Address clustering Describe plan for missing data Check for equivalency at start and attrition bias Use sensitivity tests of assumptions
ies.ed.gov
64
Personnel Section • Describe key personnel – Show that every aspect of project has an individual with expertise to do it • • • •
Appropriate methodological expertise Substantive person for all issues addressed Do not propose to hire a key person with X expertise Project management skills
– Show that every aspect of project has enough time from an expert
• Orient CVs so specific to project – 4 pages plus 1 page for other sources of support ies.ed.gov
65
Personnel Strategies for PI • Senior Researcher as PI – Show adequate time to be PI – Make credentials clear (not all reviewers may know)
• Junior Researcher as PI – Show you have adequate expertise not only to do work but to manage project • Continuation of graduate research • Management skills as graduate student
– Reviewers more comfortable if you have senior person(s) on project to turn to for advice • Co-PI, Co-I, contractors, advisory board • Have them on for enough time to be taken seriously 66 ies.ed.gov
Resources • Show institutions involved have capacity to support work – Do not use university boilerplate
• Show that all organizations involved understand and agree to their roles – What will each institution, including schools, contribute to the project – Show strong commitment of schools and districts and alternatives in case of attrition
• If you have received a prior grant award for similar work, describe the success of that work ies.ed.gov
67
Resources (cont.) • Dissemination Plan – Describe your capacity to disseminate information about the findings from your research – Identify the audiences that you expect will benefit from your research – Discuss the ways in which you intend to reach these audiences
ies.ed.gov
68
Resources (cont.) • Appendix D should back up the Resources section • Detailed Letters of Agreement from research institutions, States, districts, schools – Do letters show that partners understand their role in the project (e.g., random assignment to condition; time commitments)? – Do letters show that you have access to all necessary data to do the proposed work? ies.ed.gov
69
Budget and Budget Narrative • Provide a clear budget and budget narrative for overall project and each sub-award • Provide detail on the assumptions used in the budget (e.g., assumptions for travel) • Budget categories are described beginning on pg. 100 of 84.305A RFA, and pg. 112 of 84.324A RFA • Check RFA for specific budget requirements for Research Goals • Ensure alignment among Project Narrative, Budget, and Budget Narrative ies.ed.gov
70
Appendices Appendix
Content to be included
A
Response to reviewers for resubmitted applications – Limit 3 pages
B
Figures, charts, tables that supplement project narrative; examples of measures to be used in the project (optional) – Limit 15 pages
C
Examples of materials to be used in the intervention or assessment that is the focus of your project (optional) – Limit 10 pages
D
Letters of agreement from partners, data sources, and consultants (optional)
E
Data Management Plan (Required for Efficacy/Replication and Effectiveness Applications) – Limit 5 pages 71
ies.ed.gov
Important Dates and Deadlines
Application Deadline
August 6, 2015 4:30:00 PM DC Time
ies.ed.gov
Letter of Intent Due Date May 21, 2015
Application Package Posted May 21, 2015
72
Start Dates
July 1, 2016 to Sept 1, 2016
Finding Application Packages • FY 2016 Application Packages are available at www.grants.gov
ies.ed.gov
73
Review Application Requirements Request for Applications Available at http://ies.ed.gov/funding
Application Package Available at Grants.gov
ies.ed.gov
74
Peer Review Process • Applications are reviewed for compliance and responsiveness to the RFA • Applications that are compliant and responsive are assigned to a review panel • Two or three panel members conduct a primary review of each application • The most competitive applications are reviewed and discussed by the full panel Note: Scientific Review Office is separate from Program Officers in NCER and NCSER ies.ed.gov
75
Peer Review Process • Peer Review of Grants Process – http://ies.ed.gov/director/sro/peer_review/applic ation_review.asp
• Lists of Peer Reviewers – http://ies.ed.gov/director/sro/peer_review/revie wers.asp
ies.ed.gov
76
Help Us Help You • Read the Request for Applications carefully • Call or e-mail IES Program Officers early in the process • As time permits, IES program staff can review draft proposals and provide feedback
Don’t be afraid to contact us!
ies.ed.gov
77
Wrap-up and Final Q&A Christina Chhin NCER
[email protected] (202) 219-2280 Amy Sussman NCSER
[email protected] (202) 219-2126 ies.ed.gov
