Preparing forModel DemonstrationImplementationApril 2013Debra ShaverMary Wagner333 Ravenswood Avenue Menlo Park, California 94025-3493 650.859.2000

Overview of the Model Demonstration Coordination CenterBeginning as early as 1970 and continuing through the reauthorization of IDEA2004, Congress has authorized the U.S. Department of Education’s Office ofSpecial Education Programs (OSEP) to conduct model demonstrations in earlyintervention and special education to improve results for children and youth withdisabilities [Sec. 661 (a)]. The purpose of model demonstration projects (MDPs)is to develop new practice, procedure, or program models on the basis of theoryand/or evidence-based research. Each project then implements its model intypical settings, assesses impacts, and, if the model is associated with benefits,may go on to disseminate it or scale it up. Since 2005, OSEP has funded eightcohorts of MDPs, each focused on a single new and promising (or perhaps poorlyunderstood or implemented) practice, procedure, or program that is deemed tohave high potential for improving child outcomes.To better inform OSEP’s model demonstration program, SRI Internationalwas awarded contracts in 2005 and 2010 to collect consistent data acrossMDPs—both within each cohort and across cohorts over multiple years and topicareas. The Model Demonstration Coordination Center (MDCC) works with eachcohort to establish consistent design elements such as sample definition andselection, data collection methods and timing, and instrumentation; and for somecohorts, MDCC staff members also synthesize cross-MDP data. Comparingand contrasting implementation experiences within and across cohorts enablesMDCC to distill from MDP data factors that have hindered and those that havepromoted full implementation of their models. With this information, MDCC helpsOSEP better understand what makes an effective model demonstration.Contact InformationMary Wagner, MDCC Principal InvestigatorSRI [email protected] Shaver, MDCC Project DirectorSRI [email protected] Gonzalez, Project OfficerOffice of Special Education ProgramsU.S. Department of [email protected] Website: Model Demonstration Coordination Center has been funded with Federal funds from the U.S. Department ofEducation, Office of Special Education Programs, under contract number ED-CFO-10-A-0133. The content of thispublication does not necessarily reflect the views or policies of the U.S. Department of Education nor does mention oftrade names, commercial products, or organizations imply endorsement by the U.S. government.

ContentsIntroduction. 1Defining the Preparation Stage. 3Reflections by MDP Leaders on Preparing for Implementation. 4Building Relationships. 4Understanding Site Conditions, Priorities, and Needs . . 5Building the Capacity of Model Implementers. 7Building the Organizational Capacity of Sites. 9Building MDP Team Capacity. 10Conclusions.11References. 12

Preparing for Model DemonstrationImplementationIntroductionThe field of education has experienced growing interest inevidence-based practices to improve outcomes for children and youth. Largefederal investments have supported the identification of educational practicesand programs with rigorous scientific evidence demonstrating positive effects.There is a gap, however, between identifying effective practices and successfullyimplementing them and replicating effects in real-world settings (Cook & Odom,2013; Domitrovich et al., 2008; Fixsen, Naoom, Blase, Friedman, & Wallace,2005; Taylor, Nelson, & Adelman, 1999). Implementing promising or evidencebased practices in schools and other organizations involves changing theattitudes, beliefs, and behaviors of individual practitioners and support personnel,shifting organizational structures and cultures, and sometimes reformingsystems. Even under supportive conditions, contextual factors and outsideinfluences can thwart implementation efforts and diminish results.Model demonstration projects (MDPs) provide a valuable opportunity to testpromising practices and programs in real-world settings, and through its ModelDemonstration Coordination Center (MDCC), the U.S. Department of Education’sOffice of Special Education Programs (OSEP) has a unique opportunity to learnabout the factors that promote or hinder implementation of evidence-basedpractices across a range of interventions, populations, and settings.MDCC’s work is guided by literature from the fields of implementation science,organizational change, and diffusion of innovations (e.g., Adelman & Taylor,2003; Fixsen et al., 2005; Rogers, 2003; Weiner, 2009), which suggests thatthe changes required for successful and sustained implementation emergeover time. For example, research on school change indicates that adoptingnew practices or programs takes a significant amount of time and requires arange of conditions and supports (e.g., Fullan, 2007; Hall & Hord, 2011). Muchof this research focuses on the change process of organizations independentlychoosing to adopt new practices or programs. MDPs, however, typically do notoriginate as part of an organization’s natural improvement process; rather, theyare initiated by an external entity (i.e., the MDP grantee) looking to evaluate theimplementation of a specified set of practices in multiple natural settings withina time period dictated by the MDP funding source. Therefore, the motivationfor change, as well as the change processes and timeline, may be differentfor MDPs than for organizations independently opting to adopt an innovativepractice or program.Nonetheless, establishing conditions for successful model demonstrationimplementation typically evolves through multiple stages. We identify 6 stagesof MDPs (Figure 1): initiating collaboration with model demonstration sites,preparing for implementation, initial implementation, full implementation,sustained implementation, and dissemination. MDP teams work with host1

The goal of thisstage is to buildthe capacityof individualimplementersas well as theorganizationsor systemsin which theywork to supportimplementationand optimizeconditions forsuccess.organizations (sites) to create conditions for success at eachstage. In the first stage, MDP leaders look for and secure thecommitment of sites where there is a reasonable chance ofimplementation success. A previous MDCC brief1 presentedinformation from OSEP-funded MDPs about assessing siteconditions for factors that may affect model implementation, suchas administrative support, buy-in among implementers, and model compatibilitywith the site, among other factors (Shaver, Wagner, & Lenz, 2011). In additionto providing valuable information for site selection, initial site assessments helpMDPs tailor their capacity-building activities to the strengths and needs of theselected sites.Figure 1. Stages of Model Demonstration ProjectsInitiationInitiation. Exploringthe degree of matchbetween modelrequirements and theneeds, resources, andcapacities of potentialdemonstration sites.Selecting and invitingsites that have thepotential to be goodcollaborators andsecuring theircommitment to theproject.PreparationPreparation. Workingwith sites to identify ordevelop the resources,administrative support,and policies requiredfor implementation.Developing theknowledge and skills ofthose who willimplement the model.Refining modelcomponents based onlocal hingimplementation ofsome or all modelcomponents.Evaluating earlyimplementationexperiences to identifyadditional training,support, or resourcesneeded. Adaptingprocedures andpractices to siteconditions.Full implementation.Ensuring all modelcomponents are fullyoperational and beingimplemented withfidelity. Integratingmodel practices andprocedures into staffand organizationalpractices and routines.Continuing to adjustmodel componentsbased on ng the fullimplementation of coremodel componentsafter the end of modeldemonstration projectsupport. Integratingmodel practices andprocedures into thesite’s way of documentation of theimplementationprocesses, resources,and support needed forcore components thathave shown a positiveimpact. Creating anddisseminating productsand resources that willenable otherorganizations tounderstand, adopt,implement, and sustainthe model.The current brief addresses the second stage, in which MDPs prepare forinitial implementation. The goal of this stage is to build the capacity of individualimplementers as well as the organizations or systems in which they work tosupport implementation and optimize conditions for success. Much of theimplementation literature highlights the importance of preparation in achievinghigh-quality implementation and ultimately improved outcomes for the targetpopulation. As noted by one research team, “Many implementation efforts failbecause someone underestimated the scope or importance of preparation”(Barton & Krause, 1985, p. 103).The purpose of this brief is to help future MDPs and others successfullyprepare for implementation by sharing the reflections of leaders from OSEPfunded MDPs about their experiences in the preparation stage. MDCC staffgathered information from five cohorts of MDPs that were in different stagesof implementation, from one cohort that was planning for but had not begunimplementation, to cohorts that had completed several years of implementation,to a cohort that had completed implementation and conducted a follow-up studyon model sustainability. Across the five cohorts, 14 MDPs worked in about50 sites. Information for this brief came from two primary sources: qualitative1The MDCC brief, Assessing Sites for Model Demonstration: Lessons Learned from OSEPGrantees is available at serv.html2

It is importantto understand“what it reallytakes toget schoolpersonnelto commit,recommit, andrecommit againto makingorganizationalchanges anddeveloping newsets of skills.”—Model DemonstrationProject Leadertemplates completed by MDPs in all five cohorts about theirimplementation experiences, including their experiences duringthe preparation phase, and discussions held in the summer of2012 with leaders of MDPs in three cohorts. MDPs focused on avariety of interventions and implementation settings, from earlyintervention programs to elementary and secondary schoolswere represented.This brief begins with the MDP leaders’ thoughts on how the preparation stageis defined, followed by their insights into how to use this stage to effectivelyprepare for model demonstration implementation.Defining the Preparation StageThe timeline and activities required for the preparation stage are likely to vary,depending on several factors. First, the readiness of selected sites to implementa new program can vary widely, with implications for the length of time neededfor preparatory activities. Furthermore, factors related to the model itself suchas its state of development and complexity may dictate the preparation timelineand activities. Finally, the 3- or 4-year life span of OSEP-funded MDPs requires afairly rapid implementation schedule, which may limit time for preparation. Giventhese likely variations, we asked MDP leaders how they defined this phase.Many respondents commented on the continual nature of MDP preparation.When asked to define the time period for the preparation stage, one MDPprincipal investigator (PI) responded that preparing for implementation “ wasa work in progress. The first year we were still learning to work with eachother, and we were still developing the model.” Other MDP leaders agreed thatpreparing for implementation is an ongoing process, especially during the firstyear of implementation.Leaders of MDPs in later stages of implementation noted that implementationstages are not necessarily linear. Even when full implementation is reached,activities associated with the preparation stage may need to be repeated. Forexample, high staff turnover may require offering professional development tonew staff members to build the implementation skills that other personnel hadalready acquired. One MDP leader concluded that it is important to understand“what it really takes to get school personnel to commit, recommit, and recommitagain to making organizational changes and developing new sets of skills.”The leader of another MDP posited that helping sites develop the capacities formodel implementation begins at the site selection stage and continues throughfull implementation.For the purpose of this brief, however, MDP leaders were asked to reflecton the time between the selection and commitment of partnering sites and thelaunching of model implementation. These leaders asserted that the suggestedstrategies and activities benefit implementation at all stages.3

“Relationshipsare key” inlaying thegroundworkfor successfulmodelimplementation.—Model DemonstrationProject LeaderReflections by MDP Leaders on Preparing forImplementationThe reflections of MDP leaders on the preparation stage ofimplementation address five primary objectives: Building relationships Understanding site conditions, priorities, and needs Building the capacity of model implementers Building the organizational capacity of sites Building MDP team capacity.Building RelationshipsMost MDP leaders recognized the importance of building and nurturingrelationships with stakeholders in their implementation sites. As one MDP PIstated, “Relationships are key” in laying the groundwork for successful modelimplementation. The goal is to build a foundation of trust between members ofthe MDP team and site personnel. For some MDPs, this foundation had alreadybeen established through prior collaborations. In contrast, MDP staff workingin new sites had to begin at the ground level to build trust. Regardless of thehistory with their sites, MDP leaders used such intentional relationship-buildingstrategies as being present on site, developing relationships with organizationalleaders and those directly involved in model implementation, and demonstratingrespect for the skills, experience, and perspectives of site personnel.Conducting site visits was frequently identified as a valuable way to developrelationships and understand conditions at the sites. In addition to MDP staffproviding information about their model and answeringquestions, listening to site personnel was describedas a very important relationship-building strategy bya number of MDP leaders. One MDP PI reported that“interviewing people and just talking to them abouttheir professional lives as teachers” helped the MDPteam develop a rapport with them. Her team membersasked such questions as, “What do you like about yourjob?” “What are the difficulties?” and “What are yourhopes and fears regarding participating in a project likethis?” Members of another MDP team attended schoolevents such as open houses, parent meetings, and school competitions “to bea presence and provide support” as they worked on building relationships withinthe schools.Several MDP staff members highlighted the importance of buildingrelationships with leaders. The PI of an MDP that had completed its projectobserved that garnering the support of high-level leaders early on was animportant contributor to successful and sustained implementation in severalsites. Developing these relationships from the beginning increased the project’s4

“One of thepowerful thingswe do asobservers isthat we reallydo observe—wedon’t judge. Wedon’t go in asif we were theexperts and tellthem what todo.”—Model DemonstrationProject Leadervisibility and made it easier to align the project with other highpriority local initiatives. Another MDP leader observed thatnurturing relationships with “natural leaders” among those whowould be implementing the model was also beneficial. Herteam looked for those leaders and got them on board with theproject early. She stated, “When others see them [the leaders]beginning to implement and being enthusiastic, they will see that it [changingpractices] can be done.”Demonstrating respect for site personnel as professionals is anothercomponent of successful relationship building, according to some MDP leaders.The personnel of one MDP made it clear to teachers that they were not comingin to judge what the teachers were doing but to help them reflect. “One of thepowerful things we do as observers is that we really do observe—we don’t judge.We don’t go in as if we were the experts and tell them what to do.” Leaders ofother MDPs agreed that setting a tone of mutual respect and collaboration helpsbuild strong relationships. One MDP respondent added that being flexible is oneway to demonstrate respect for the needs and perspectives of site personnel:“We flexed and adapted to them” as part of the relationship-building process.This kind of relationship building takes some skill and intentionality, as oneMDP leader noted: “We are not novices at this. We are really thoughtful aboutgoing into the schools, following up with the principal, asking, ‘How are youdoing?’ letting them know we are there, meeting with the secretaries and the staffso they understand what we are doing.” Furthermore, this leader believed thatthe effort and work required to build relationships for successful collaboration andmodel implementation can be easily overlooked in model demonstration.Understanding Site Conditions, Priorities, and NeedsAnother purpose of site visits is to gain an understanding of the capacities,beliefs, and priorities of site personnel as well as of the organizational structureand culture, which is critical to assessing how compatible a model is likely to bewith a site’s realities. In the preparation stage, MDP leaders work to identify bothfacilitative and potentially hindering conditions and opportunities for capacitybuilding. “It is important to understand the baseline, to get a clear understandingof where the school is, what they have, and what they need,” asserted an MDPPI. As MDP staff members identify the gaps between the capacities of theimplementing organizations and those required for successful implementation,they can determine whether they need to build capacities at the site or adjust themodel to adapt to local conditions.MDPs used a variety of methods to obtain information about site conditions.One MDP’s staff members who were working with secondary schools shadowedstudents for a day to get a sense of what was going on at the school. They wereable to observe pedagogy and obtain a snapshot of what students experiencedon a typical day. They noted that this was valuable because it gave them astarting point for discussions with school personnel. Observers shared themesabout what they saw and had school personnel respond to the observations.5

Another MDP leader talked about the importance of seekingfeedback after sharing findings from observations and interviewsby asking, “Are we correct about the way you view this?” Bysoliciting this kind of feedback, “We are making sure we got ourstory straight,” added this MDP respondent.Other methods of gaining an understanding of site conditions includedattending meetings and training sessions at the sites. The leaders of an MDPthat focused on tiered interventions for English language learners reportedthat they attended district professional development sessions on the district’sresponse to intervention program to determine how the model could beincorporated into existing practices and policies. MDP leaders also reported thatthey attended staff meetings to gauge aspects of the organizational climate thatmight affect model implementation.Some MDP teams used formal needs assessmentsas part of their site preparation work. For example,the PI of an MDP on tertiary behavioral interventionsreported that her team had sites examine data alreadyavailable at the site. This MDP team approachedthis task by saying, “Let’s look at your data and seeif you’re satisfied with what you see.” This processhelped site personnel identify where they neededhelp. Surveys of site personnel represented anothermechanism for assessing needs as well as strengths.The preparation stage also is a time to appraise the buy-in of site personnelfor model implementation. Even when site personnel appeared to be verycommitted to a model during the site selection process, MDP leaders assertedthat buy-in could be accurately ascertained only through face-to-face contactwith site personnel. In schools, understanding buy-in among implementingteachers is essential because, as one leader concluded, “Teachers can secretlyopt out, close their doors, and teach how they want.” Some MDP leaders felt thatindividual interviews were important to get “real” information from site personnelabout buy-in, but others acknowledged that time and scheduling constraints didnot always allow for individual interviews.Gaining an understanding of an organization’s key players and their strengthsis another aspect of preparation for model implementation. “We try to get a feelfor the power structure and who are the power people who make things happen,”stated one MDP PI. Understanding other sources of power and influence at a siteis also important. One MDP experienced implementation challenges because ofan unusually powerful union in one district. If the MDP team had realized up frontthat the union was such an important player, it could have worked to build thoserelationships before conflicts arose.Understanding the model-site fit requires two-way communications—bothlistening to and receiving information from sites and communicating to sitepersonnel what is required for successful implementation. The members of6

“We have tobe willing toconstantlymeet them[site personnel]where they are.”—Model DemonstrationProject Leaderone MDP team shared that although they thought they hadclearly articulated what model implementation would require ofteachers, they discovered later in the year that the teachers didnot fully understand what it really would involve. Part of the MDPstaff’s role according to one PI is “to help adjust people’s realityto a new reality and help adjust their expectations.”When MDP teams learn about gaps between model requirements andconditions and capacities at a site, one course of action is to make changesto the model. In fact, most of the MDP leaders reported having to make majoror minor changes to their model to adapt to local conditions. For example, oneMDP team had to rethink its approach for multitiered supports in a school thathad stratified students into different learning communities based on achievementlevels. “There was a lot of work initially in developing a model that would matchthe needs of that school and any other school as we progressed.” Similarly, otherMDP leaders asserted, “We have to be willing to constantly meet them wherethey are,” and, “While we have a model in place, it can’t be rigidly administered.It has to be open to what they [site personnel] say their needs are.” In one MDP,site capacities and needs were accommodated through a collaborative processwhereby early childhood service providers and MDP leaders jointly developedand modified the intervention to address the needs of children and their families.Not all MDP leaders advocated this level of flexibility. Some spoke of thebalance that is required between listening and adapting to the organizationand staying true to the vision of the project team. One MDP team reported thatit negotiated with site personnel in a respectful way, noting, “We need to coconstruct the model but with the critical components the project leaders believeare important.”Through initial visits and contacts, model leaders may determine that therequisite conditions for successful model implementation are not present. Thiswas the case for one MDP. After observing teachers, conducting interviews,and communicating with administrators in a school that had committed to theproject, MDP personnel concluded that poor internalcommunications, tensions between teachers andadministration, and organizational instability werelikely to create significant implementation issues.Given these suboptimal conditions, the MDP leadersmade the difficult decision to pull out of the schooland find a replacement school.Building the Capacity of Model ImplementersMDP leaders reported engaging in several typesof activities to help those who would be implementing the model acquire theskills, knowledge, and attitudes for success. Almost all the MDPs providedformal professional development (PD) activities to prepare site personnel forimplementation. For school-based programs, these typically took place in thesummer. Offering follow-up coaching sessions with individual implementers also7

was a common way to reinforce content from the formal PDsessions and provide individualized support. One MDP teamrecognized the variation in resources, staffing patterns, andtraining needs in its participating early childhood programs andmodified the training format accordingly (e.g., one-to-one andsmall-group format training).Many MDP leaders reported working with site personnel during initial visits toidentify gaps in knowledge that needed to be addressed in formal PD sessions.For example, after seeing teachers struggle with classroom management duringtheir early observations, the leaders of one MDP added this topic to their summerPD. This topic was not directly related to the model; however, the training wasadded to support the model’s implementation.By involving site personnel in identifying training topics, MDP leaders soughtto increase engagement in training and model implementation, but respondentsarticulated other strategies as well. Several MDP leaders pointed to theimportance of engaging training participants in self-reflection, stating that whensite personnel can consider challenges and come up with their own solutions,they deepen their understanding of the model, and new practices are moresustainable. Providing practice opportunities was another strategy to helpparticipants gain knowledge and skills. MDP teams also developed a variety oftraining materials for model implementers, including print and online manuals,video modules, check lists, and other resources.The formal PD sessions were valuable for continuing to gain an understandingof site personnel’s capabilities and for building relationships. Initial PD sessionsoften helped MDP staff identify topics for subsequent training sessions. SeveralMDP leaders reported that they were intentional about using training sessions tobuild trust with participants and garner credibility with the host organizations.Building the capacity of implementers also involved addressing their attitudesand beliefs. Most MDP teams had some reluctant site personnel and had to workto help them see the potential benefits of the model. One MDP leader claimedthat “one of the best ways of changing beliefs is through informal TA [technicalassistance].” Thus, this MDP used informal TA such as one-on-one coaching tolisten to the concerns of those involved in model implementation, help resolveproblems, and gain their trust, in hopes of increasing their buy-in, commitment,and confidence in implementing the model.The preparation stage was just the beginning—MDP leaders acknowledgedthat developing the capacity of implementers does not happen overnight.Nor does it happen without the leadership and structural supports of the hostorganizations, as described below.8

The effortthis [buildingorganizationalcapacity] entailsdepends on thecomplexity ofthe model, howwell the modelfits in with othersite initiativesand practices,and the existingcapacities ofsites.Building the Organizational Capacity of SitesMany of the MDP preparation activities targeted thedevelopment of the leadership, infrastructure, and resourcesof the host organizations to support model implementation.MDP leaders reported that the effort this entails depends on thecomplexity of the model, how well the model fits in with othersite initiatives and practices, and the existing capacities of sites. For example,the leaders of several MDPs indicated that their models did not require majorchanges at their host organizations, with one leader noting, “We are embeddingthis model into the context of what teachers do naturally in their classroom,”adding that the project team did not require the school to do much differently atthe beginning of model implementation. Similarly, other MDP leaders reportedthat especially at the beginning of implementation, they centered their effortson increasing the effectiveness of existing practices atthe sites rather than asking sites to make significantchanges. Nonetheless, almost all the MDPs used thepreparation stage to build organizational support forimplementation, even if major organizational changeswere not required.Developing leadership capacity was a common MDPpriority in this stage. For example, one MDP leaderreported being intentional about building administrativecapacity to support implementation. At this MDP, aswell as others, frequent meetings with administratorsand other leaders were h

these likely variations, we asked MDP leaders how they defined this phase. Many respondents commented on the continual nature of MDP preparation. When asked to define the time period for the preparation stage, one MDP principal investigator (PI) responded tha