Current Issues in Education > Volume 11, 2009 > Number 8

Citation Information

Mohammed, S., Pisapia, J., & Walker, D. A. (2009, May). Optimizing State Policy Implementation: The Case of the Scientific Based Research Components of the NCLB Act Current Issues in Education [On-line], 11(8). Available: http://cie.ed.asu.edu/volume11/number8/

Optimizing State Policy Implementation: The Case of the Scientific Based Research Components of the NCLB Act

By:

Shereeza Mohammed
John Pisapia
Florida Atlantic University

David A. Walker
Northern Illinois University

Abstract

A hypothesized model of state implementation of federal policy was extracted from empirical studies to discover the strategies states can use to gain compliance more cost effectively. Sixteen factors were identified and applied to the implementation of the Scientific Based Research provisions of the No Child Left Behind Act.

Data collected from state program directors countrywide produced a strong predictive model using 10 of the hypothesized factors and the degree of policy compliance. It appears that the key to increasing compliance of state implementation plans was to consider the effects of several significant predictors, some of which had unexpected influence.

Table of Contents

• The Role of the State in Implementing Federal Educational Policy
• Purpose of the Study
• Theoretical Framework
• Method
• Findings
• Discussion, Conclusions and Implications
• Limitations
• Recommendations
• References

Educational governance in the United States has been uniquely decentralized; a loosely-coupled system of federal, state, and district input steeped in local control. Each decade has seen attempts to increase the federal role in local education. This heightened role gradually evolved through the movement from the Elementary and Secondary Act (ESEA) of the late 1960s, Goals 2000 in the 1980s, to the Improving America Schools Act (1994) culminating in the No Child Left Behind Act (NCLB, 2001) which requires that every student be proficient in state standards by 2014 or risk the loss of federal Title I funding. Historically, the response to these incursions, such as those posed by NCLB, has been resistance or mutual adaptation to federal intervention.

The Role of the State in Implementing Federal Educational Policy

As federal educational policy intent evolved, much of the literature on policy implementation focused at the federal and local level, and was less representative of the state level. However, the state does not mindlessly follow orders from the federal or local governments. Since it holds constitutional responsibility for the provision of educational services to its citizens, the state plays more than just the intermediary role traditionally studied in implementing federal education policy. In fact, the implementation of any federal policy requires that the state consider the interests of bureaucrats, politicians, and special interest groups at the federal, state, and local level whose input and posturing provides complexity to the process as they respond to federal educational policy implementation. In 1990, Goggin, Bowman, Lester, and O’Toole viewed the state role as dynamically balancing forces from above and below and translating policy horizontally to enact federal rules and regulations that best suit their own capacity and context. It’s the state’s responsibility to translate not only the procedural aspects of the law, but also the substantive intent so that a state attains the policy’s goals as most appropriate to its needs and ability to implement. It is this balancing role of the state regarding federal policy implementation that is the focus of this study.

The Purpose of the Study

The intent of the study was to identify and test a set of state initiated strategies and tactics that predict the level of compliance achieved during policy implementation by State Departments of Education. This intent was carried out by determining if there was a relationship between and among a state’s capacity to implement, its contextual environment, the implementation strategy it favored, and the degree of compliance of the plan the state submitted to the federal government to implement the Scientific Based Research provisions of the NCLB Act. By identifying those factors that predict compliance states could secure the federal funding that local school districts have come to depend on to provide educational services to its citizens.

Theoretical Framework

The components of a hypothesized model of policy implementation were developed from the empirical studies of policy implementation in education (Goggin, Bowman, Lester, and O’Toole, 1990; Honig, 2004; Leithwood & Anderson, 1988; Matland, 1995; Musella, 1989; Odden, 1991; O’Toole, 2000). From these sources, we identified three constructs to organize the factors we extracted to study: a state’s capacity to implement policy, its contextual environment, and its favored implementation strategy. As seen in Figure 1, we further described each of the three constructs through factors and sub-factors found in the literature. The 10 capacity sub-factors we found are listed more comprehensively in Table 1.

Figure 1. Factors and sub-factors affecting the extent of a state’s policy implementation.

Policy Implementation

For the purpose of this study, policy implementation was defined as the extent to which the plan submitted by states to the federal government complied with all the components in the mandate. A state could choose to implement some, all, or more aspects than was outlined in the law. This variability in the extent of compliance in a state’s plan was used as a measure of policy implementation. We assumed that the compliance of a state’s planned implementation in relation to its plan [see the bottom right hand side of Figure 1] would depend on the state’s capacity to implement, its context, and the implementation strategy it chose (Goggin, Bowman, Lester, and O’Toole, 1990; Honig, 2004; Leithwood & Anderson, 1988; Matland, 1995; Musella, 1989; Odden, 1991; O’Toole, 2000).

State Capacity Factors

In this study, a state’s capacity to implement policy is defined by a number of factors that have shown repeated evidence in the literature to limit and/or facilitate both the rate and extent of implementation. These factors are: institutional factors, personnel, financial resources, and technical assistance (Crosby, 1996; Goggin, Bowman, Lester & O’Toole, 1990; Grindle 1996, Leithwood & Anderson, 1988; Odden, 1991).

Institutional factors. Institutional factors which impact policy implementation include the structures such as vertical and horizontal linkages and cross-functional teams that support both top-down, and bottom-up program implementation (Anderson, 2002; Bardach, 1977; Huberman & Miles 1984; Pressman & Wildavsky, 1984; Stewart & Bullock, 1981). A top-down approach that is highly structured horizontally and hierarchical works best when a policy can be simply implemented as it is stated without any adaptation and does not require the technical expertise of a variety of governmental and non-governmental organizations (NGOs).

Cross-functional teams also impact policy implementation. For example, in their study of curriculum reform efforts of 10 states, Anderson, Odden, Farrar, Fuhrman, Davis, Huddle, Armstrong and Flakes-Mosqueda (1987) found that local capacity for change increased where there were collegial relationships linking the districts to the state department. This was further supported by Marsh and Odden’s (1990) finding that both top-down and bottom-up strategies needed cross-functional teams involving teachers, department chairs, curriculum specialists, and administrators at both the school and district sites to blend the two approaches to attain successful outcomes. They reported that cross-functional teams, communicating in both directions created supportive linkages and structure that enabled implementation of the new state curricular frameworks and attained the most appropriate outcome for students.

Personnel Development and Technical Assistance. The capacity to implement depends on the knowledge and skills of those managing, planning, and evaluating the implementation at the state level as well as those at the site of implementation (Anderson, 2002; Corcoran, 2003; Desimone 2002; Goggin et al., 1990; Leithwood & Anderson, 1988). The Corcoran (2003) study on the use of evidence based practices in three urban school districts verifies that lack of competent staff was one of the downfalls leading to faulty implementation of evidence-based practice. These districts tried to implement evidence-based practice in too many areas and relied on one professional development team to attend to both district staff and teacher needs. This directly impacted the level of technical expertise available to Local Educational Agencies (LEAs) for successful program implementation.

Technical assistance is often cited as a factor for successful implementation (Desimone, 2002; Fuhrman, Clune and Elmore, 1988; Corcoran, 2003; Odden, 1991; and Leithwood & Anderson, 1988). Corcoran’s (2003) study on introducing evidence-based practice cited the importance of the appropriate technical expertise, the lack of which resulted in failure. Corcoran found that district committees lacked the knowledge to assess and interpret studies with significant findings, contradictory conclusions, small sample sizes and claims from various reform developers. Thus, they were incapable of vetting appropriate programs for district and school use because they lacked the necessary technical expertise of the research community.

Hence, states with an abundance of professional staff and expertise should have a greater capacity to identify and facilitate the use of relevant scientifically based strategies and programs from the state to the classroom level. Further, such staff will use these skills as well as technical assistance from federal, state and research institutions. Such expertise and assistance, if continuous and intensive, will sustain its effect on implementing SBR programs and strategies at the classroom level. All of these sub-factors were examined from the state’s perspective during this study.

Financial Resources. Funding for both the implementation process and the actual program is a frequently cited factor influencing policy implementation (Garn, 1999; Goggin et al., 1990; Leithwood & Anderson, 1988). However, regardless of the amount of federal funding attached to a program, states may still be wary of the associated conditions in accepting to implement it. For instance, state budgets play an important part in the implementation of federal policy. In fact, “according to the National Council of State Legislatures, 19 states faced FY2004 budget shortfalls of more than 10% of their general fund” (Rotherham, 2003, p. 35). Therefore, the role of financial resources is great when considering the degree of compliance of a state’s plan to implement policy.

State Context

Contextual factors create a state’s unique environment to implement policy. A state’s ability to generate political and inter-organizational support for a policy affects its rate and extent of implementation (Goggin et al., 1990; Odden, 1991).

Political context. At the state level a network of elected and appointed government officials, and interest groups can either work cooperatively and build advocacy coalitions or work on opposite sides of an issue to obstruct a policy’s extent and rate of implementation (Goggin et al., 1990). In fact, the political context can produce a wide variation in approaches used by LEAs ranging from resistance to adaptation. Fuhrman et al. (1988) found that active LEAs and interest groups were proactive in creating policies and networks with other districts and the state to engage in “amplifying policies around local priorities” instead of simply adapting them. Fuhrman and her colleagues used the term “strategic interaction” to describe the means by which LEAs were able to “seize policy opportunity, coordinate and expand state policies to meet their needs, and anticipate and actively shape state policy” (1988, p. 255).

Inter-organizational support. Inter-organizational support from external agencies is thought to be positively associated with successful policy implementation (Goggin et al., 1990; Odden, 1991; and Leithwood & Anderson, 1988). A complex network of various organizations can provide repeated and overlapping of functions. They serve as a second level of observers to find missed issues before a problem arises and also to provide extra expertise and alternate perspectives (Goggin et al. 1990).

Federal administrators used this assumption of a positive relationship between inter-organizational networks and policy implementation in creating several reinforcing structures to support the Education Sciences Reform Act (2002). In particular, federal funding through the U.S. Department of Education’s (ED) Institute of Educational Sciences must be aligned with the established criteria of scientific research. The establishment of the What Works Clearinghouse (WWC) provides a source of programs and strategies that meet the criteria and the 10 regional education laboratories that are available to support states in distinguishing criteria-relevant research products. Further, states can use the services of research institutions and universities to develop the scientifically-based research products they need. All of these avenues illustrate the variety of networks that a state and district can build to support the shift to scientifically-based practice.

Implementation Strategy

The capacity to implement federal policy seems to be directly affected by the strategy used by each state (Musella, 1989). Several different strategy models are found in the literature (Moore, Goertz, & Hartle, 1983; Murphy, 1981; Musella, 1989). Musella proposed that states often choose the classical, political, or cultural strategies to implement policy.

The classical strategy is noticeable when policy is implemented exactly as prescribed. It results in high fidelity to the policy, or in the policy being implemented beyond the stated requirements. However, with the scarcity of financial resources facing many states, some may use a political strategy and bargain with regulators to reshape certain aspects of the policy. Such negotiations result in concessions and tradeoffs being made between states and federal regulators that can vary from changes in rules to enforcement or evaluation strategies. The cultural strategy suggests that the federal policymakers and state policy implementers view a federal policy as a guide to implementing measures to attain similar, but reduced goals. In this case that which is implemented may be very different from the original policy as it is reformulated to meet local needs and desires.

Implementation Strategy

A predictive model of policy implementation was drawn from the theoretical framework. We expected that:

1. A state’s level of capacity to implement policy stemming from its institutional factors, personnel factors and financial resources, affects its ability to achieve high levels of policy implementation.
2. The political and inter-organizational factors that comprise a state’s contextual factors can promote or hinder a state’s ability to achieve high levels of policy implementation.
3. The implementation strategy a state pursues affects its ability to achieve high levels of policy implementation.

These three assumptions guided the development of a predictive model of policy implementation composed of the 16 sub-factors seen in Table 1.

Table 1. The Predictive Model of Policy Implementation.

As seen on Table 1, we extracted 9 factors and 16 sub-factors from the theoretical framework that have been found to impact state policy implementation. Theoretically, the high presence of these factors found in a state’s policy culture leads to high degrees of compliance in 13 of the sub-factors and a potentially negative response in the other three factors.

Scientific Based Research and State Policy Implementation

The extracted model of policy implementation was tested by applying it to state implementation of the SBR provisions of NCLB. These provisions introduced a different paradigm, and in fact, criteria for making practice based choices that have been proven to be successful using scientific research. The goals of the NCLB Act that every student be proficient in state standards by 2014 or risk the loss of federal Title I funding has produced a broad and deep impact (Rentner, 2005; 2006). It affects state and local decisions regarding the state tests, textbook adoption, expenditure of funds, and assignment of teachers and principals.

SBR is the result of a dynamic debate about the quality of educational research which has encompassed the education research community and those who regulate it at the federal level. Educational research was compared with medical research as well as that in other social sciences. According to Whitehurst (2001), the increase of randomized field trials from 1950 to the present in the study of criminology, social policy and psychology showed rapid growth in these fields, which can be replicated in education, if randomized experiments are emphasized. According to advocates, the aim was to ultimately deliver the same success seen in the medical field from using the scientific model. The detractors suggested that in medicine the concern is for the wellbeing of an individual who is treated with the singular goal of better health that is easily recognizable and directly measurable. In education, however, the concept of a well-educated person varies according to values and the definitions of knowledge needed at various stages of a child’s development which does not enjoy consensus (Hammersley, 2002).

This debate culminated in the belief that SBR would fill in the gaps in improving student learning as well as in the knowledge needed for improvement to meet the goals of the NCLB Act. The Act contained the following narrow definition of SBR (Olson & Viadero, 2002, p. 14).

Table 2. The Definition of SBR according to the NCLB Act

The NCLB Act is prolific in its use of SBR to determine the receipt of federal funding. For example, in Title I part A of the NCLB Act, the programs, methods and instructional strategies used by schools must be grounded in SBR as a condition of receiving federal Title I funds. Further, schools identified as needing improvement must have their improvement plans, curricula, instruction and technical assistance also grounded in SBR (NCLB, 2001).

In effect the SBR provisions require states and schools accepting federal dollars to use programs proven by scientific evidence to raise student achievement cost effectively. Theoretically, a state could reject federal funds and thereby not have to implement the provisions of the law. However, most states try to implement the SBR provisions and receive the federal fiscal support it previously relied upon. To do this each state’s capacity, context and strategic intent factors become important to indicate the extent of SBR compliance they can achieve.

This paradigm shift SBR practices presented several problems. The most significant is the lack of adequate numbers of studies that meet the SBR criteria that relate to programs, methods and instructional strategies. This is further exacerbated by the current culture of teachers, administrators, and district personnel using programs, curricula, and strategies based on philosophical reasons. Moreover, most research on educational interventions has either been grounded in qualitative research which falls outside of the NCLB definition of SBR or has not yet been conducted. Add to this that many states are experiencing fiscal as well as other challenges and this increases the problems of implementing SBR according to the NCLB Act.

In summary, the change to evidence-based forms of practice created a cultural revolution demanding much time, consistent effort, and intensive monitoring to implement. This revolution challenged the state to maximize its resources and capacity to plan its implementation carefully to be successful at the local level. Therefore, this study focused on the strategies and tactics the state - through its State Department of Education - used to respond to the SBR provisions of NCLB. At the time of this study the three programs met the SBR standard: Title IA, Reading First, and Comprehensive School Reform and were offered to states for implementation. While all three programs were initially studied, only Reading First produced a viable predictive model the results of which are reported in this article.¹ The layer of NCLB with its top-down control of standards, testing and accountability has made the balancing role of the state more complex and requires a strategic approach to navigate the unique implementation environment existing in each state (Cooper, 2004).

Method

The present study was a non-experimental correlational design with the four major variables being degree of policy implementation compliance (i.e., criterion variable), state capacity, state contextual factors and implementation strategy (i.e., predictor variables). The review of the three predictor variables yielded 16 sub-factors to test for ability to predict. These factors were described in the paragraphs above. The predictive nature of the policy implementation model was tested by applying it to the SBR provisions of NCLB Act as they related to implementation of the Reading First program.

The survey target population consisted of state program directors involved in the implementation of the Reading First programs in the 50 State Departments of Education. Thirty five of the 50 states directors responded resulting in a response rate of 70 %. The profile which emerges of survey respondents indicates that most of them (95.6%) had a graduate degree in education, more than half of which were at the Masters level. Half of the respondents majored in Educational Leadership, Administration or Management, while most of the other half majored in Teacher Education. Most (73.5%) were employed at a State Department of Education just prior to this position as program director. Just over half had between 1 and 5 years experience as program director and 42.6% had more than 5 years experience on the job. The level of research expertise at the state level was limited; fewer than 3% of program directors had their highest degree in the field of Educational Research. Similarly, most of the personnel responsible for implementing the research aspects of their programs are administrators (43%). Only 7% were consultants and another 7% were specialists directly involved with programmatic functions. Hence, there does not seem to be a wealth of expertise on educational research at either the program administrator level or at the technical level for these three programs. Moreover, in 29% of states, respondents indicated that there were no assigned persons overseeing the implementation of the research aspects of the program. This would imply that there is a large need for technical assistance external to state departments to comprehensively implement the SBR components of each program. This target population was considered a universal sample and was representative of expert public policy implementers in education at the state level. Hence this study yielded results that are not generalizable beyond this population.

Data describing the use of the predictor factors were collected from the program directors on a validated survey instrument. This instrument consisted of sixty-six questions separated in sections each probing a different predicator sub-factor to be answered from alternatives presented in a Likert-type scale.² This survey was validated by a state liaison from Palm Beach County School District using a practitioner’s perspective, a university professor of educational policy in the United Kingdom and a leading policy analyst from the Education Commission of the States. Since this study utilized a universal sample there was no appropriate sample to pilot the instrument. Therefore a reliability study was conducted after the survey responses were returned to determine the instrument’s internal consistency. The survey was managed online using the Perseus Survey Solutions Enterprise 5.2 survey program and hosted on a university-based data gathering website.

The financial data were collected to determine the presence of federal grant programs and were collected from federal databases. The source for the amount of the federal grant per program came from the ED website where tables for each federally funded program are represented by fiscal year.

The criterion variable was measured by numerically scoring each state’s implementation plan submitted to the federal government using the components of SBR in the law for the Reading First program. The items chosen were categorized to be discrete and mutually exclusive as recommended by Gall et al. (2003). The aim, therefore, was to create the checklist so that each item or component coded only for one idea and did not overlap with any other.

Prior to use, the compliance checklist was validated by two raters who were trained in its use. Each expert was independently required to identify and list only the SBR components in each of the three programs in the law. Each list was then compared with the author’s checklist and found to be identical. Reliability was assessed as each expert was asked to rate two state plans per program using the appropriate checklist. The inter-rater reliability score was calculated and values were adjusted using Cohen’s Kappa. The results demonstrated very high reliability as rater agreement was 100 %. The checklist was then applied to the state plans for all three programs using the scoring scheme found on Table 3.

Table 3. Component Scoring on each Checklist

The resulting score for all of the components in the state plan were then divided by the score of components that explicitly matched those in the law and multiplied by 100 to create a percent of compliance. This resulting score became a measure of the extent of implementation (degree of compliance) for each program in each state.

Through the steps of data analysis, the data were found to be normally distributed and lay within acceptable limits for skewness and kurtosis values. A reliability analysis on the original 16 sub-factors identified through the literature review calculated using Cohen’s Alpha. Only sub-factors with an alpha value which approximated to 0.70 or above were chosen as reliable enough to be used for further analysis. This analysis resulted in only the ten sub-factors found in Table 4 being retained for further analysis of their ability to predict policy implementation compliance.

Table 4. Constructs and Sub-factors with Acceptable Reliability Values

Once pre-processing of the data was completed, correlational and multiple regression analyses were used to determine the relationships between the predictor and criterion variables to determine if a viable, stable, and replicable predictive model of policy implementation could be produced. Correlational analyses were employed to detect the controlling effects of variables. Multiple regressions, squared semi-partial correlations were used to determine the unique variance a predictor contributed to the criterion variable. The size of the effects of each relationship being examined using Cohen’s index of effect sizes which categorized each value into small, medium and large effects (1992).

Findings

The analysis of the Reading First (RF) program plans produced a viable predictive model. The sample size of 26 responding state Reading First program directors initially appears to be small. However, using Green’s (1991) formula to determine an appropriate sample size, it was determined, that an n of 11 would have been suitable in this case:

The overall model, had a high multiple correlation coefficient (R = 0.925) and an adjusted R² value of 0.759, which was a large effect size and indicated that nearly 76% of the variance in percent compliance could be explained by the ten sub-factors found in the model (see Table 5).

Table 5. Model Summary

Further, a power analysis using the sample size of 26, with 10 predictors and an R value of 0.925 indicated the power of this model to be high (0.99). The presence of 10 predictors also prompted the need to examine the extent of multicollinearity among them. The variance inflation factor for each predictor was less than 10 (lying between 2.00 and 4.41), which signified that there were no high intercorrelations among the predictor variables and hence, the existence of multicollinearity was minimal.

To investigate the influence and size of the effect of each sub-factor, the following statistical measures were examined: the standardized coefficient,?; the squared semipartial correlation, (ΔR²); and the associated effect size. The standardized coefficient β indicated the influence of the predictor on the criterion variable. The squared semipartial correlation was considered to be the change that occurred in R² (ΔR²) for the whole model when a particular predictor was removed. In essence, it is related to the portion of the variance in the criterion explained by that predictor in the model and, hence, its effect size. This study employed Cohen’s (1992) cutoff values for effect sizes using ΔR² (i.e. 0.02 (small), 0.15 (medium) and 0.35 (large)). As was first discussed by Glass, McGaw, and Smith (1981), and reiterated by Cohen (1988), about these effect size target values and their importance:

… these proposed conventions were set forth throughout with much diffidence, qualifications, and invitations not to employ them if possible. The values chosen had no more reliable a basis than my own intuition. They were offered as conventions because they were needed in a research climate characterized by a neglect of attention to issues of magnitude. (p. 532)

Further, the critical effect size of 10% was applied to the ΔR² value to determine if the effect size was of practical significance. Medium effect sizes, where ?R2 values were equal to or greater than 15%, were considered to be of high practical significance. Practical significance relates to the real world multi-dimensional context in which state planners operate to help them prioritize their resource allocations, especially where many variables cluster between the small and medium effect sizes as defined by Cohen (1992). All of these measures are itemized in Table 6 for each predictor.

As seen on Table 5, cross-functional linkage had the most influence on percent compliance (β = 0.958). Cross-functional linkage, which refers to teams serving various functions among practitioners and administrators, also caused a 21% change in R² when it was removed from the equation. This means that cross-functional linkage uniquely accounted for almost 21% of the variance in Y resulting in an effect size that was above the medium cut point and was of high practical significance. These results suggest that the most influential factor in implementation planning was cross-functional linkage.

Table 6. Statistical Measures for Each Predictor and Percent Compliance

Finally, the relative practical significance of all factors demonstrating statistical significance is displayed in Figure 2. All the significant factors are displayed in order of their influence on percent compliance. The first level of three factors demonstrated effect sizes of high practical significance, the second level of three factors established average practical significance, and the third level of four factors displayed effect sizes of low practical significance.

Figure 2. The practical significance of ten significant factors found in this study on Implementation Plan Comprehensiveness for Reading First

Discussion, Conclusions and Implications

A strong predictive model of the relationships among a state’s capacity to implement the SBR components, its contextual environment, the implementation strategy it favored, and the comprehensiveness of its planned policy implementation was produced. Cross-functional linkage, finances and negotiation were the most influential positive predictors of the compliance and comprehensiveness of the state’s plan to implement SBR. As the presence of these three factors increased so did the level of compliance with the provisions of SBR. On the other hand, partnerships and coordination, as well as horizontally linking with other institutions such as private nonprofits agencies; for-profit consultants; national, regional, and state research labs; in- and out-of-state universities; and faith base organizations, were negatively correlated with the level of compliance. As the presence of these two factors increased the level of compliance found in the plan decreased.

The key to comprehensive state implementation plans appears to be found in the relationships established with professionals at the school, district, and state levels. These make up the cross-functional linkages found in multifunctional teams that include practitioners and administrators. By increasing the input of these teams at the state, district, and school levels, and providing funding and negotiating with the federal government, states enhanced their level of compliance with the federal SBR mandate. These findings echo recommendations by Rentner (2006) for more transparency in state accountability. She affirms that state departments should provide more information about the process for considering state changes to accountability plans. The findings indicate that those states adhering to this recommendation find their state plans are approved faster than those states that place less emphasis on such relationships.

Finances, in the form of consequences of loss of funding and federal program grants, provided the second most positive influence (behind cross functional linkages). This relationship demonstrated that as the level of threat of losing federal funding increased in a state, the implementation planning comprehensiveness also increased. For policymakers this confirmed the influence of fear that has been historically used as the “stick” in the well known carrot and stick analogy. Furthermore, federal program grant was the fifth most influential factor and has practical significance for implementers and policy funders alike. In this relationship the amount of federal funding of grant programs such as Reading First appeared to be an important consideration when states planned the comprehensive magnitude of their implementation strategies. Thus, as expected, the provision of withholding of finances should be continued as a policy instrument to ensure full compliance with federal policy.

The ability of the state to negotiate with the federal government on the conditions specified for SBR in NCLB also led to more comprehensive plans. Negotiation and reformulation formed the sixth most influential factor that had a positive and practical significance to implementers. As leverage points in the negotiations, states considered citing funding restraints or intended on engaging in recommendations to the ED to increase provision effectiveness. Such a strategy appeared to be influential and to have an effect of practical significance among state strategies when considering level of implementation planning.

On the other hand, partnerships, coordination, and linking with other institutions such as private nonprofits agencies; for-profit consultants; national, regional, and state research labs; in- and out-of-state universities; and faith base organizations, also proved to be statistically and practically significant. Partnership and coordination and horizontal linkages were the second and third most influential factors identified in the analysis. However, these factors demonstrated a negative effect on the comprehensiveness of the incorporation of SBR components into the state plan submitted to the federal government. This means that as their level of presence increased, the implementation plan comprehensiveness decreased. Therefore, for a program such as Reading First with established criteria and interventions, little can be gained from connecting with either research institutions or other state level agencies. Thus, resources may be more optimally allocated to other areas to build capacity where there is greater need.

We concluded that state departments of education would be well served to place their energy, emphasis, and money on developing strong cross-functional linkages, complying with federal grant requirements and employing negotiation in developing their plans to implement federal mandates that resemble programs such as Reading First and minimize efforts related to other factors the degree that it is politically feasible. This order of importance is especially appropriate for mandated programs such as Reading First where most of the preparations in terms of research and identifying interventions pre-existed the planning phase. Only after states place their priority on cross-functional linkages, as well as finances and negotiations, it is recommended that they consider using the four other strategies that have a statistically significant relationship with planning comprehensiveness, but when further analyzed yielded lower levels of practical significance (i.e., personnel, internal technical assistance, political support, and technical assistance from external agencies). Although they produced small effects on planning, their influence was still statistically significant and funding can be allocated to them once the first six influential factors have been considered. While these factors seem to be less significant in the state’s response to federal policy implementation, there is some evidence (see Rentner 2006) that they are important in local implementation of federal policy.

Finally, the review of literature led to the development of a hypothesized predictive model of 16 sub-factors. During the analysis six factors were discarded and 10 retained for study. These 10 sub-factors produced a predictive model of policy implementation that we sought as the aim of the study.

As seen in Table 7, 3 of the 10 hypothesized relationships were confirmed by the statistical model produced in this study: cross-functional linkages, consequences of the loss of funding, and federal program grant. If their presence in a state’s plan is high, their impact on the degree of compliance with will also be of high or of average practical significance and impact. From the initial theoretical relationship discovered in the review of literature, the high presence of negotiation and reformulation strategies were expected to have a negative impact on the degree of compliance with federal mandates. The findings of this study surprisingly indicated evidence of a positive impact.

Moreover, the high presence of six of the predicted influencers in the state plan - horizontal linkage, amount and knowledge of personnel, internal technical assistance, political support of stakeholders, technical assistance from external agencies, partnership, and coordination were found to reduce the level of state compliance with federal mandate studied. They either had a negative or low impact.

In summary, Table 7 compares the hypothesized model to the actual model of factors that influence policy implementation. The hypothetical model was useful in providing initial guidance in preparing for this study. However, the results of this study did not confirm the ability of some factors (i.e., horizontal linkages, amount of knowledge of implementers, technical assistance internal and external to the system, partnership and coordination, and political support), as well as a classical strategy as being useful in producing a comprehensive state plan that met SBR provisions and that was approved by the federal government.

Table 7. Expected Versus Actual Relationships with respect to Practical Significance Based on Evaluative and Planning Perspectives

Limitations

The main limitations of this study arise from the three federally-funded programs: Reading First, Title IA, and Comprehensive School reform that were studied. Only the Reading First program produced a viable regression model and was used as the program from which this article and major conclusions were drawn. Some reasons for the prominence of Reading First included the established, but more relaxed guidelines from the U.S. Department of Education regarding scientific-based research, proven scientifically based interventions, and the luxury of a year and a half to produce its implementation document. The other two programs were not presented with guidelines from the federal government in a timely manner that could be used in the production of state plans for implementation. Furthermore, these two programs (i.e., Title IA and Comprehensive School Reform) had a shorter planning window and fewer interventions available that were scientifically-based than Reading First. These conditions led to less compliance and a small sample size of programs to analyze. Finally, because the sample was universal in nature and limited to state program directors, the results are confined to state level public policy implementers of this genre. These factors therefore limit and impact the generalizability of the final conclusion to all federal mandated programs chosen to implement the scientific based research provisions of NCLB.

Recommendations

The following recommendations were drawn for states planning to implement federal policies such as Reading First, where pre-existing programs and interventions based on scientific research data exist. First, the study provides state level implementers a clear and predictive set of strategies to establish effective and efficient planning decisions in the implementation of a program such as Reading First. State policy planners and implementers should prioritize their efforts and resources on establishing cross-functional linkages, maximizing their funding, and considering negotiation in terms of their contextual needs as a strategy to use with the federal government. Additionally, resources spent on establishing horizontal linkages with state agencies and partnerships with research agencies can be reallocated to other areas of need in order to make more efficient planning decisions to save both time and money.

Second, federal level policymakers now have an empirically tested set of policy instruments at their disposal that can positively influence state compliance, such as providing grants and constructing mandates which use loss of federal funding as a consequence of non compliance. Further, by creating a preparatory phase that preceded the enactment of the Reading First program, they can ensure that the products of this phase would boost the extent of implementation. In so doing, this preparatory phase enhanced the ability of the state to implement the program and so the mandate format of the policy is more easily translated into comprehensive implementation planning.

Third, this study was exploratory, yet produced a strong predictive model between capacity, contextual, and strategic factors and the extent of implementation planning by states. This is important because the paradigm shift toward scientifically-based research has profoundly impacted the worlds of educational research, policy implementation, and practice. Yet, this study pointed out some success and failure of current research-proposed factors that should be considered by state policy makers and implementers. The predictor and non-predictor factors should be further tested to confirm these findings since they can certainly benefit all those involved in evidence-based practice, as well as those in the field of policy implementation.

Further, study of programs with and without preparatory phases is recommended to replicate the findings of this study and to also distinguish the differences among the planning, implementation, and evaluative phases of a program’s implementation. These three phases might be strengthened through the addition of milestones to clarifying expectations, responsibilities, and accountability.

Finally, and most importantly, the policy factors that influence state compliance with federal mandates should also be used to study local education agency compliance with state mandates. Such studies should target if presence and ordering of the factors are similar in both arenas. Similarly, future studies are necessary to ascertain if certain provisions of a state’s response to federal mandates such as evaluation, monitoring, and funding lead to greater local degrees of compliance and eventually to increased student achievement.

Footnotes

¹ See Mohammad, S. (2005). State planning strategies to implement the scientifically based research components of the NCLB Act. Unpublished doctoral dissertation. Boca Raton, FL: Florida Atlantic University for the full survey and validation techniques.

² See Mohammad, S. (2005). State planning strategies to implement the scientifically based research components of the NCLB Act. Unpublished doctoral dissertation. Boca Raton, FL: Florida Atlantic University for the full survey and validation techniques.

References

Anderson, B., Odden, A., Farrar, E., Fuhrman, S., Davis, A., Huddle, E., Armstrong, J., & Flakes-Mosqueda, P. (1987). State strategies to support local school improvement. Knowledge: Creation, Diffusion, Utilization 9(1), 42-86. 

Bardach, E. (1977). The implementation game: What happens after a bill becomes a law. Cambridge, MA: MIT Press.

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates, Publishers.

Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155-159.

Cooper, B. S., Fusarelli, L.D., & Randall, E. V. (2004). Better policies, better             schools: Theories and applications. Boston, MA: Pearson/Allyn and Bacon.

Corcoran, T. (2003). The use of research evidence in instructional improvement. Consortium for Policy Research in Education Policy Briefs. Retrieved December 15, 2003, from http://www.cpre.org/Publications/rb40.pdf

Crosby, Benjamin, (1996). Policy Implementation: The organizational challenge. World Development. 24(9) 1403-1415

Education Sciences Reform Act of 2002, H.R. 3801 107d Cong. (2002).

Farrar, E., Desanctis, J., & Cohen, D. (1980). Views from below: Implementation research in education. Teachers College Record, 82(1), 77-100. Retrieved June 21, 2003, from http://www.tcrecord.org/PrintContent.asp?ContentID=1001

Fuhrman, S., Clune, W., & Elmore, R. (1988). Research on education reform: Lessons on the implementation of policy. Teachers College Record, 90 (2) 237-257.

Glass, G. V., McGaw, B., & Smith, M. L. (1981). Meta-analysis in social research. Beverly Hills, CA: Sage.

Goggin, M., Bowman, A., Lester, J., O’Toole, L. (1990). Implementation theory and practice: Toward a third generation. Glenview, IL: Scott, Foresman/Little, Brown.

Green, S. B. (1991). How many subjects does it take to do a regression analysis? Multivariate Behavioral Research, 26, 499-510.

Grindle, M. (1996). Challenging the state: Crisis and Innovation in Latin America and Africa. Cambridge: Cambridge University press

Honig, M. (2004). The new middle management: Intermediary organizations in education policy implementation. Policy Analysis,26:65-87.

Huberman, M., & Miles, M. (1984). Innovation up close: How school improvement works. New York: Plenum.

Leithwood, K., & Anderson, S. (1988). Evaluating policy implementation: The case of special education legislation. Toronto, Canada: The Ontario Institute for Studies in Education.

Lusi, Susan Follett, (1997). The role of state departments of education in complex school reform. New York: Teachers College Press, Columbia University.

Marsh, D.D., & Odden, A. (1990). Policy implementation and current curriculum reform: An analysis of the implementation of the California mathematics and science curriculum frameworks. In A. Odden (Ed.) (1991), Education policy implementation (pp. 219-239). Albany, NY: State University of New York Press.

Matland, R. (1995). Synthesizing the implementation literature: The ambiguity-conflict model of policy implementation. Journal of Public Administration Research and Theory, 5(2): 145-174. 

McLaughlin, M.  W. (1987). Learning From experience: Lessons from policy implementation.  Educational Evaluation and Policy Analysis, 9(2), 171-178

Moore, M., Goetz, M., & Hartle, T. (1983). Interaction of federal and state programs. Education and Urban Society, 15(4), 453-478.

McDermott, K. (2003). Changing conceptions of federalism and education policy implementation: Where did the “No Child Left Behind Act” come from, and where is it going?  Prepared for the Annual Meeting of the American Political Science Association, Philadelphia, PA, August 27-31, 2003

Murphy, J. (1981). Differential treatment of the states: A good idea or wishful thinking? Prepared for The School Finance Project, Washington, DC: U.S. Department of Education. In Moore, M., Goetz, M. & Hartle, T. (Eds.) (1983). Interaction of federal and state programs. Education and Urban Society, 15(4), 453-478.

Musella, D.F. (1989). Problems in policy implementation. In M. Holmes, K.A. Leithwood, & D.F. Musella (Eds.) Educational policy for effective schools (pp. 93-111). Toronto, Canada: The Ontario Institute for Studies in Education.

No Child Left Behind Act of 2001, Pub. L. No. 107-110. Odden, A. (Ed.). (1991). Education Policy Implementation. Albany, NY: State University of New York Press.

O’Toole, L. (2000). Research on policy implementation: Assessment and prospects. Journal of Public Administration Research and Theory, 10(2): 263-288.

Pressman, J., & Wildavsky, A. (1984). Implementation (3rd ed.). Berkeley, CA: University of California Press.

Rentner, D. (2005). From the capital to the classroom: Year 3 of the No Child Left Behind Act. Center on Education Policy.

Rentner, D. (2006). From the capital to the classroom: Year 4 of the No Child Left Behind Act. Center on Education Policy.

Rotherham, A.J. (2003, Summer). Hopes and hazards: No Child Left Behind & low performing public schools. The State Education Standard. National Association of State Boards of Education, pp.30-35.

Stewart, J., & Bullock, C.S. (1981). Implementing equal education opportunity policy. Administration & Society, 12, 427- 446.

U.S. Census Bureau data. Retrieved June 15, 2004, from website http://www.census.gov/hhes/income/income03/statemhi.html.

U.S. Department of Education data. Retrieved June 15, 2004, from website http://www.ed.gov/about/overview/budget/statetables/index.html