Connec-ng Research, Policy and Prac-ce
First Impressions: How to Win Grants and Influence Your Research Career Meredith Larson, Ph.D.
Na-onal Center for Educa-on Research
and
Amy Sussman, Ph.D.
Na-onal Center for Special Educa-on Research
ies.ed.gov
Goals 1. Iden-fy two things you should have with you at all -mes. 2. Describe a theory of change and discuss why it’s essen-al grant wri-ng. 3. Have the tools necessary for refining your research statement of purpose. 4. Have a sense of what program officers and reviewers want from you and your applica-ons. All of this is to help prepare you to write winning proposals and ar-culate your research agenda to the broader (research) community. ies.ed.gov
Our Assump-ons of You and What You Want • We assume that you want – to help improve educa-on – to carve out your own research agenda – to get funding to conduct your research
• We assume that you have – exper-se in a wide range of topic areas and research methods/analyses – varying levels of familiarity with grant wri-ng ies.ed.gov
What You’re Facing Whenever you ask someone to support your research, remember that it all comes down to convincing them that they should care about your topic and that you have the necessary competencies to reach your goals. …so how do you do it?
ies.ed.gov
What You’re Facing Audience awareness 1. Seek common ground. • Highlight core values or concerns that your research addresses. • Look for ways to build off of things that are familiar.
2. Be helpful and thorough. • Make things easy to understand, remember, and find. • Be engaging and helpful.
ies.ed.gov
What You Need to Do Sell your research idea, promote yourself as the best person to do the research, and build goodwill and trust. How? By demonstra-ng that you know what the problem is and have a way to address it. ies.ed.gov
Explaining Your Research You should have two things ready at all -mes: 1. Statement of Purpose (preferably an “elevator speech” or “research pitch”) 2. Theory of Change
ies.ed.gov
Statement of Purpose We will come back to this later, but in general a ready-‐made Statement of Purpose should
– Be short and a_en-on-‐ge`ng and – Contain the problem statement and your contribu-on to solving it.
Your fellow researchers, friends, and family members should be able to understand it and see its relevance.
NOTE: It’s not as easy to do as you may think. You have to know a lot about what it is you want to do before you can describe it succinctly.
ies.ed.gov
Theory of Change (ToC) • This is the model underlying your research. • It is a roadmap to your grant narra-ve. • It is a source for genera-ng research ques-ons. • It is constantly evolving.
NOTE: Some fields and scholars use terms like Logic Model or Logical Framework to mean things similar to what we are calling a Theory of Change here.
ies.ed.gov
ToC: What is It? Theories of Change help define how and why an interven-on (e.g., curriculum, policy, prac-ce) should lead to par-cular outcomes. – Makes assump-ons explicit – Suggests the causal rela-ons
ies.ed.gov
ToC: What is it? • Some associate them with program evalua-on studies, but they are also useful for explora-on work, work on cogni-ve or behavioral processes, etc. • They are frequently represented visually, but they can be simply text.
ies.ed.gov
ToC: Why You Should Care Reviewers will evaluate your research on how clear your theory and model are. A strong ToC will help make your case for you. Four Criteria Used to Judge ToCs (Connell & Klem, 2000): 1. 2. 3. 4. ies.ed.gov
Plausible Doable Testable Meaningful
ToC: How to Make One • Start with the long-‐term outcome and work backwards. – What precondi-ons need to exist to lead to the outcome? – What is occurring in the context that could hinder or support the outcome? – What assump-ons are you making?
• Draw it out. • Write it up. • Get feedback. Revise. Repeat. ies.ed.gov
Create the General Framework Come up with the broad framework. Intermediate Long-‐term Strategies/ State Ini-al Outcomes Outcomes Ac-vi-es Context Generally, you should start with the outcomes. You can then jump all the way back to thinking about the ini-al state or hop around. ies.ed.gov
The Basic Components Assump-ons Ini-al State (resources)
Strategies/ Ac-vi-es
Outcomes (proximal & distal)
Context INITIAL STATE: What’s there before the interven-on including things such as resources or student characteris-cs, etc. (some-mes called input) STRATEGIES/ACTIVITIES: The things that happen (e.g., ac-vi-es, events, curriculum) to the people who par-cipate or who are the targets of the interven-on (some-mes called output) OUTCOMES: The proximal and distal changes for people involved in or that are the targets of the interven-on ASSUMPTIONS: What you believe to be true of the resources, the people involved, the people targeted, and the interven-on in general CONTEXT: The environment (internal or external) in which the interven-on takes place (e.g., other policies or prac-ces occurring simultaneously)
ies.ed.gov
Example
4-‐year-‐old pre-‐K children Demographics Eligibility/need
Exposed to intervenUon
1-‐hour lesson Wrap-‐around services PosiDve supports Focus on early literacy PD for providers
PosiUve aVtudes to school Improved pre-‐literacy skills
Increased school readiness
Learn appropriate school behavior
Setting/context Personal and family characteristics Prior experience Teacher/provider experience
ies.ed.gov
Greater gains in K literacy
Detailed Components Now that you have the general aspects, you should specify the details. These will lead into your research design, measures, etc. in your narra-ve.
Indicators: Opera-onalizing the outcomes (e.g., what indicates success?) PopulaUons: Iden-fying who should show which change (could be mul-ple groups) Thresholds: Se`ng your expecta-ons for change for each outcome (e.g., how much is good enough?) Timeline: Determining when we should meet the threshold for each outcome
(Connel & Klem, 2000)
ies.ed.gov
Example OUTCOME = Improved pre-‐literacy skills Indicator: Correctly iden-fies le_ers and sounds PopulaUon: 4-‐year-‐olds, especially those with li_le print exposure Threshold: 1.5 effect greater than comparison Timeline: Ajer full interven-on, 16-‐weeks
ies.ed.gov
Another Example (Slightly different layout ) InstrucUonal Context Text and task characteris-cs Nature of instruc-onal support
Reader CharacterisUcs Decoding and fluency proficiency Verbal knowledge A_en-on and behavior
ies.ed.gov
MoUvaUon to Read Decision to engage in reading and task persistence Expectancies for success Self-‐efficacy Perceived control
Value Achievement goals Intrinsic mo-va-on Usefulness Social mo-va-on
Text Comprehension Reading Engagement Physical Cogni-ve: Building and maintaining coherence
Adapted from a model presented by Dr. Sharon Vaughn (R305F100013).
Don't Do This 1. Be overly simplis-c 2. Overwhelm the reader 3. Use color as a key (audience awareness!)
ies.ed.gov
Example: Too Simple
ies.ed.gov
Students with Disabili-es
The Increasing School Success Program
Increased Academic Achievement
[Processes 3-17 repeat twice in Year 2]
13-‐15b. Observe SLTs, document implementa-on 15a.Coaches assist SLT in implemen-ng PD with faculty
16. interview coaches, SLTs, and PLTs about PD and WL processes
PLT
Profile Analysis Set instrucUonal goals & WL focus
WL Debrief
[4 weeks at end of prior year or beg of current year]
PLT
= Begin Process = Feedback for next process = Delega-on of PD = Primary Leadership team
WL Debrief
PLT
WL Debrief
PLT
PLT
[Processes 4-11 WL repeat to mid-year] Debrief 12. Coaches & PLTs choose 2-‐4 teachers (based on WLs) to become Dynamic Leadership Team 1]
11. Coaches share/ implement revisions with PLTs
10. reviews/ revises model based on findings
approx. 9 wks after school begins
= Secondary Leadership team
22
4. Coaches work with PLT
WL Debrief
WL Debrief
PLT
5. guide Coaches and PLTs in analyzing data collected
PLT
[PLT appoints SLT1 to address PD Topic 1]
WL Debrief
PLT
WL Debrief
SLT 1
6a. Coaches facilitate PLT iden-fica-on of annual goal
WL Debrief
PLT
PLT
Research
ies.ed.gov
PLT
13a. researchers train SLT to research best-‐ prac-ces in PD area
SLT
approx. 3 wks after school begins
WL Debrief
PLT
PLT
6-‐8b. observe PLTs, documents implementa-on 7a. Coaches teach PLT to conduct WL
Begins weekly meeUngs
WL Debrief
Implement Prof Devt
PLT
PLT
SLT 1
8a2. Coaches assist PLTs in weekly WLs and Debriefing through mid-‐year
WL Debrief
WL Debrief
WL Debrief
8a1. Coaches assist PLTs in using all data
PLT
PLT
14a. SLT and Coaches create PD unit
18a. submit Annual Report to schools
17. reviews/ revise model based on findings
9. interviews coaches and PLTs about WL process
1a. PI Recruits and Trains Coaches 1b. develops data collec-on tools
2a. PI matches Coaches to schools (n=5)
3b. develops electronic tool 3a. Coaches collect 3 yrs. stud. ach. & demo. data per school
2b. trains coaches to use all data tools
Draw your Theory of change Take a few minutes and write a couple words about the following: 1. What are the iniUal states (inputs)? 2. What are the strategies/acUviUes (outputs)? 3. What are the outcomes? Proximal, Intermediate, Distal… 4. What is in the context?
ies.ed.gov
So Now What? Your Theory of Change acts like a roadmap that can help lead you to research ques-ons, a proposal, and so much more.
ies.ed.gov
ToC & Research Ques-ons QuesUons about iniUal states or inputs – Were resources provided and used?
QuesUons about strategies/acUviUes or outputs – Did the ac-vi-es occur as planned?
QuesUons about outcomes – Were there changes in the proximal/distal outcome rela-ve to the control/comparison?
QuesUons about context – Did the context suppress or reinforce effects? ies.ed.gov
(Milanowski & Kimball, 2009)
ToC & Your Applica-on(s) In IES applica-ons, there are four major parts of the applica-on: Significance, Research Plan, Personnel, and Resources. The first two of them can draw heavily from the ToC.
Significance – Draws heavily from the outcomes, context, and assump-ons Research Plan – Draws heavily from the strategies/ ac-vi-es and the outcomes ies.ed.gov
ToC & Your Research Plan In your research plan, you need to specify exactly what it is you’re exploring, crea-ng, valida-ng, or tes-ng. You also need to specify how you will do these things. Strategies/AcUviUes What are the pieces that you’ll be exploring, crea-ng, tes-ng, etc.? Outcomes Indicators: What will you measure, and how you will measure it? PopulaUons: Who and where (both in treatment and control/ comparison)? Thresholds: What effect (size) should you expect? Timeline: When should you be collec-ng what data? ies.ed.gov
ToC & Your Applica-on(s) Other components of your narra-ve and your applica-on can also be informed by the ToC:
Personnel – For example, you want someone with experience in each of the outcome areas and with knowledge of how to assess the strategies/ac-vi-es. Budget – For example, how many observa-ons will you need to take (# of researchers and % effort) and when (during what grant year) and what assessments you will need (how many of which one should you buy). ies.ed.gov
ToC & Your Statement of Purpose Your Theory of Change also contains the core elements of a good statement of purpose, mission statement, elevator speech, etc. You have the pieces. Put them in a short, diges-ble format that conveys the problem’s significance. And now you have your statement. (It actually requires a bit more than that.) ies.ed.gov
An Example Now we’re going to walk through an example of how you might take a ToC and generate an “elevator speech”, a statement of purpose, even an opening paragraph. Note: Some-mes people have their statement and then create their ToC. You need both, and they inform one another. You do not need to have a ToC to write a statement.
ies.ed.gov
From ToC to Statement of Purpose Distal Outcome: Increase student achievement in science Proximal Outcome: Young children (preK – K) with greater ability to think scien-fically Strategies/AcUviUes: § § § § §
Professional development curricula Four 1-‐hour long workshops for teachers Training on science content and early childhood pedagogy Current, prac-cing educators PreK students
IniUal State: Early childhood educators who lack science content and pedagogical knowledge, lack of curricula to help teachers AssumpUon: Teachers are essen-al to student outcomes. Teachers need both content and pedagogical knowledge and exper-se. Teaching teachers will improve student outcomes. ies.ed.gov
…con-nued Speech: Being able to think scien-fically at a young age predicts long-‐ term academic success in science. However, many children coming into elementary school lack this ability. Even if they are exposed to science in PreK programs, they are s-ll not acquiring the necessary skills. One reason PreK students may be struggling is that early childhood educators ojen do not have the background knowledge of science content or instruc-on to help their students. Because teachers are so important to the learning process, we need to help be_er prepare them for their role. So I am developing an interven-on that will increase early childhood educators’ content and pedagogical knowledge so that they can be_er prepare their students.
ies.ed.gov
Components of a Good Statement • Story! Characters, goals, drama, resolu-on. • Facts – but not too many. This will vary based on audience and use. • Short – shoot for a paragraph or < 90 seconds.
Make them care. ies.ed.gov
Now you try • What is the problem? The tension? The drama? • What evidence do you have that it’s important? • How can you help solve it?
ies.ed.gov
Uses of Your Statement • Opening paragraph(s) of your applica-ons or your purpose paragraph – Note: This is true for all documents, such as disserta-on proposals, not just IES applica-ons.
• Job talks • Communica-ng with the general public (e.g., when you give your NPR interviews) ies.ed.gov
Importance of Opening Paragraph • Opening paragraph sets the scene for readers. – Iden-fies the significance of the work to be done and what actually will be done – Readers use it to organize informa-on in rest of the applica-on – You can lose your readers right off with an unclear opening
ies.ed.gov
NCER/NCSER Abstracts As an example of an opening/purpose paragraph, we’d like to use examples from some of the abstracts we have posted online: h_p://ies.ed.gov/funding/grantsearch ies.ed.gov
Example of Opening/Purpose Purpose: Teaching children how to think scien-fically in the preschool years has the poten-al to address an exis-ng achievement gap in early science and provide children with the skills necessary to con-nue learning and thinking cri-cally throughout the school years.
Background
Problem Unfortunately, early childhood teachers typically lack content and pedagogical knowledge in science and are not prepared to provide developmentally appropriate experiences that support children's early science learning and readiness. In order to address these challenges, the researchers will develop Goal: the Cul$va$ng Young Scien$sts (CYS) intervenUon, which includes How will you a professional development program for early childhood address this educators, science curricular content, and a set of formaUve problem? assessment tools. The interven-on is intended to lead to an increase in teachers' use of science instruc-onal content and prac-ces in preschool se`ngs and improvements in young children science content knowledge and scien-fic thinking skills. ies.ed.gov
Example of Opening/Purpose Core diagnos-c features of au-sm include deficits in social-‐communica-ve Background func-oning. Two pivotal skills for young children with au-sm include joint a_en-on and pretend play, which cons-tute early founda-ons upon which later social-‐communica-ve skills are built. Joint a_en-on (characterized by behaviors to share a_en-on toward such as poin-ng, showing, and coordinated looking objects or events with another person) and symbolic play (characterized by the ability to pretend), play important roles in language development and social engagement with peers. Children with au-sm show deficits in these skills, and as a consequence may lag Problem behind their peers academically and socially. Advancing Social-‐Communica-on and Play (ASAP) is an interven-on that has recently been developed to help preschool children with au-sm learn and prac-ce these important skills. However, we do not know how effec-ve it is. Goal: The purpose of this research is to conduct a cluster randomized trial to How will you evaluate the efficacy of ASAP. The major goals of the project include inves-ga-ng whether children experiencing the interven-on demonstrate address this greater gains in the proximal child outcomes of social-‐communica-on and play problem? skills as well as the more distal outcomes of language development and engagement with classroom objects and peers. The study will also examine whether child-‐level and teacher-‐level (i.e., teacher burnout, general classroom quality) characteris-cs moderate the impact of the interven-on.
ies.ed.gov
Final Comments: Helpfulness of the Program Officer • Share your framework and statement of purpose with the Program Officer – Ensure you are submi`ng to the correct compe--on/topic – Springboard for further discussion
ies.ed.gov
Final Comments: Importance of Clarity of Wri-ng • Readers (e.g., applica-on reviewers) ojen complain about lack of clarity. – Significance too general – Lack of detail regarding interven-on, development cycle, or data analysis – Use of jargon and assump-ons of knowledge – Poor wri-ng (e.g., grammar), awkward construc-ons, etc.
ies.ed.gov
Final Comments: Importance of Prac-cing All forms of communicaUon require pracUce. Things only other people can tell you:
– Is your descrip-on of your theory too long or complex? – Is your statement of purpose too long or complex? – Is your idea engaging, and does it invite ques-ons that build off of your ideas (rather than those that try to figure out what your idea really is)?
ies.ed.gov
Pulling It All Together Recall Why You’re Here: You want to write winning grants and build a solid research career. But How? By ge`ng people interested in your work and the ques-ons you feel are important and by building trust that you have a plan for addressing an important issue. ies.ed.gov
Pulling It All Together • Have a theory of change – Should be as fully explicated as possible – Is always being refined – Should be made with and informed by others
• Have a clear statement of purpose at all -mes – Your “elevator speech” – Simple, elegant (hard to do, actually) ies.ed.gov
Connec-ng Research, Policy and Prac-ce
Thank you.
[email protected] [email protected]
ies.ed.gov
More info Remember: Some scholars dis-nguish between a Theory of Change and a Logic Model. Others do not. We use the term “Theory of Change” for the purposes of this presenta-on as a way to discuss how people can present the causal rela-ons among elements. Please use the term most appropriate for your field. (Note that the IES RFAs use “Theory of Change.”) This presenta-on was informed in part by the following resources: Connell, J., & Klem, A. (2000). You can get there from here: Using a theory of change approach to plan urban educa-on reform. Journal of EducaDonal and Psychological ConsultaDon, 11(1), 93-‐120. Center for Theory of Change: h_p://www.theoryofchange.org/. Retrieved May 1, 2013. Harris, E. (2005). An IntroducDon to Theory of Change (Issue Topic: Evalua-on Methodology, vol. 11). Retrieved May 1, 2013, from h_p://www.hfrp.org/evalua-on/the-‐evalua-on-‐exchange/issue-‐archive/evalua-on-‐methodology/an-‐ introduc-on-‐to-‐theory-‐of-‐change. Kimball , S. & Tony Milanowski, T. (2009). Establishing a Theory of Ac-on and Logic Model for Your Project Session 1 [PowerPoint slides]. Retrieved on May 1, 2013 from Center for Educator Compensa-on and Reform website: h_p://www.cecr.ed.gov/pdfs/september2009mee-ng/TheoryofAc-on.pdf. Reisman, J., Gienapp, A., Langley, K., & Stachowiak, S. (2004). Theory of change: A prac-cal tool for ac-on, results and learning. OrganizaDonal Research Services for Annie E. Casey FoundaDon.
ies.ed.gov