On Thursday, May 28, the Center on Standards, Alignment, Instruction, and Learning (C-SAIL) hosted a virtual conference to present and discuss five years of findings on the implementation and impact of college- and career-readiness standards. We invited attendees to submit questions during the event which we would answer following the conference. Below is a selection of attendee questions along with answers from the C-SAIL team.

Opening Plenary Session

Question: How is the study accounting for 1) the NAEP assessments do not measure attainment of the Common Core or any particular state's standards; 2) implementation of the new standards varies in quality and depth?

Answer: Admittedly, NAEP is not perfectly aligned with the Common Core State Standards (CCSS) or other college- or career-ready (CCR) standards, and therefore it may not fully capture the effect of the CCR standards that have been at the center of the current wave of standards-based reform. Although not perfect, the alignment between NAEP and CCR standards is still substantial. A 2015 study conducted by the NAEP Validity Studies Panel (Daro, Hughes, & Stancavage, 2015), for example, found that 79% of the items on the Grade 4 NAEP mathematics assessment were clearly matched to content that appears in the CCSS at or below grade 4, and the alignment rate was even higher for grade 8 (87%). Therefore, although NAEP was not designed to assess specifically the knowledge and skills targeted by the CCSS or CCR standards, it is a reasonable outcome measure for an analysis of the impact of the CCR standards on student achievement. Further, a test perfectly aligned with a particular set of standards may not provide the best test of the impact of the standards, if the hope is that teaching would not focus narrowly just on what is specified in the standards, but aim to improve students’ knowledge and skills more broadly.  

Regarding the implementation of the new standards, we acknowledge that the rollout of the CCR standards across states has been slow (typically 3–5 years) and hindered by a multitude of implementation challenges.  Implementation challenges may be one potential explanation for the lack of positive impact of the new standards on student achievement. Given data constraints, we were unable to take the quality of implementation of the CCR standards into account in our analyses. Thus, findings from our study should be interpreted as evidence for the impact of the CCR standards as implemented, rather than the impact of well-implemented CCR standards.

Reference: Daro, P., Hughes, G. B., & Stancavage, F. (2015). Study of the Alignment of the 2015 NAEP Mathematics Items at Grades 4 and 8 to the Common Core State Standards (CCSS) for Mathematics. Washington, D.C.: American Institutes for Research.

Question: To what degree does LEA local control impact standards and curriculum implementation. The degree of LEA local control varies widely from state to state, and from even from region to region within states. If SEA roles for implementation vary based on local control law, policy, and culture, would this impact LEA implementation within differing states?  If so, how do we account for this?

Answer: You are right, there is considerable local and state variation. This helps explain why LEAs within and across states may differ considerably. Our paper [link ER Trends paper here] addresses how this local control impacts curriculum, professional learning, and other aspects of standards implementation.

Question: Would you speak more about the slide on page 30 re: alignment, specifically the negative numbers in Elem ELA?

Answer: The negative numbers there mean that teachers of ELA are not emphasizing the content emphasized in their new standards (to the extent that mathematics teachers are). One notable finding there is that the ELA teachers’ alignment moves more toward zero in 2019 – indicating that ELA teachers are shifting toward focusing more on emphasized content. Typically, ELA has been less tractable than mathematics in terms of moving teachers to focus on the standards. This finding is consistent with that idea.

Question: When you talk about "results" is it results as shown in standardized tests and graduation rates alone? Maybe there is a problem with the measuring tool as well. 

Answer: In the FAST study, we estimated impact on the alignment of teachers’ instruction to their state standards, as measured by the Survey of Enacted Curriculum (SEC), and student achievement as measured by state tests. We found a significant, positive impact on the alignment of teachers’ instruction to state standards in math, but not ELA. We didn’t find a significant impact on student achievement.

We used measures that have been widely used to examine alignment and impact on student learning.  Although teachers’ alignment on the SEC was not positively associated with student achievement in our study, other studies have found a link between the SEC and student achievement. State tests are intended to be aligned to state standards, but we didn’t analyze the state tests in our study to assess the extent to which that is the case.

Question: What about microlearning in professional development? Is there any research regarding this approach?

Answer: Thanks for asking about microlearning. We aren’t familiar with research on the impact of this approach but will keep looking.

Breakout Session: Curriculum

Question: What role does "Passing on Failure,” if any, have to do with the core curriculum and standards implementation? How important is it for students to be prepared for the content standards?

I work with a specific type of student who did not get these skills in their younger school experiences. Every one of them learns differently and finds it very difficult to understand some of the curriculum lessons because they don't have the basics.  Do you look at supplementing resources based on individual learning styles?

Answer: I think clearly student preparation is one of the key drivers of teachers’ curricular decision-making. For instance, in a six-state study of teachers’ curriculum use in elementary mathematics, “textbook is too hard” was the third biggest reason teachers gave for using materials other than their main textbook, and my interviews with teachers in California confirm this. In particular I know that math teachers supplement often because they say that their core materials do not provide sufficient “practice,” which I take to mean that they think that students lack foundational skills that are not well covered in textbooks. All that said, students are who they are, and teachers to at least some extent have to do the best they can with the students they have and the resources they have/can find. I think the best recommendation is to work very hard to ensure that there is a vertically (i.e., across-grades) coherent curriculum that is implemented well and that students are promoted when they can demonstrate that they have mastered the curriculum. But of course, that is easier said than done, and of course also some students will always need remediation. A recent brief by Alan Schoenfeld also touches on these issues in the area of mathematics.

Question: What evidence do we have that districts are promoting "curriculum by Pinterest" approaches vs well designed research-based curriculum.

Answer: There isn’t any good data that I am aware of that would answer this question (see this report for a discussion of state data around curriculum). Anecdotally I can tell you that this isn’t much of a phenomenon in at least math and ELA in California, but I cannot really say much beyond that about other states or subjects. I think it’s probably more common to adopt curriculum materials of some kind but not really support teachers to use them well or in ways that are intended.

Question: Is there evidence that state, school and district policies are beginning to recognize the needs for time that teachers have for planning lessons, selecting curricula, planning for individual student needs, and improving their practice rather than the current few minutes each day in most school schedules.

Answer: Again, I don’t really know of good evidence on this specific question – tracking teacher time use and school/district policies is not something we do very well. A report I have recently written that is currently under review finds that common planning is moderately common—35% of teachers in a three-state sample (LA, MA, RI) report receiving it weekly, and another 18% report receiving it monthly. But that means just under half of teachers receive it less often than that. There is also, anecdotally, a growth in PLCs, which I think often include these kinds of collaborative planning opportunities (though often are also more focused on analyzing student data). I also cannot say whether these represent increasing trends. But certainly, I agree with the implication of the question that those kinds of supports are important to drive good curriculum implementation.

Question: Can you speak about what specific recommendations should be made when looking at supplementing, especially for striving readers?

Answer: This is a great question, and one that I am not well equipped to answer. My broad suggestion is twofold. First, I think there needs to be coherence. Whatever curriculum striving readers get needs to be coherent with what they’re getting in other classes and across grades. Second, I think especially struggling readers need to be engaged, and that engagement probably comes in large part from the relevance of the readings to themselves and their identities. But I recognize that these are very general suggestions, and I would need to do more research myself to offer more.

Question: We have just created a much-needed Director of Curriculum and Instruction position in our district.  Do you have a recommended reading list or social media connection for learning and support?

Answer: This is great to hear! I would say that probably the best strategy is going to be to find other districts that are doing good things with curriculum and with offices similar to this one and to draw from their experiences. It’s a bit academic, but my C-SAIL colleagues and I recently wrote about our case studies of five partner districts around their curriculum efforts, and we came away with some findings that I think may be relevant to district leaders.

Question: There are many advocates for culturally relevant curriculum and instruction. What are best practices for classes with students from diverse cultural backgrounds, and perhaps multiple home languages?

Answer: I would turn to the experts on this! A few resources that seem highly useful to me include this, this, and this, but there are so many more great resources available online on this topic. You should also reach out to authors who have expertise in this area.

Question: Several panelists seem to be using "curriculum" and "materials" interchangeably. I'd love to hear people talk about how they define curriculum and its relationship to curricular materials, both commercially published and locally created. How do you see the connection between curriculum frameworks, or standards implementation guidance, and instructional materials?

Answer: There is a lot of looseness and overlap in how people talk about curriculum. I’ve pasted at the end of this document a portion of a chapter I recently wrote about curriculum materials that I think lays out these distinctions. As for curriculum frameworks or implementation guidance, my strong preference would be for states to just lay out what curriculum materials they think are good and should be adopted as opposed to creating yet another layer in between standards and instruction. 

Question: Do any of you have a teacher/coach friendly tool for evaluating task and standard alignment? Thinking this might give teachers thinking points as they review curricula for adoption. Thank you!

Answer: My personal opinion is that for the most part I would rely on outside expert ratings for alignment, especially those of EdReports. That said, two tools that I am familiar with are the EQUIP and the IMET.

Question: I am in a district in which no core curriculum resource was purchased for ELA, Science, or SS and teachers were allowed to "build" the curriculum map.  This has been incredibly frustrating to teachers.  Did this approach appear in your research, and if so, what were to academic outcomes to this approach?

Answer: Thankfully none of our districts in the C-SAIL project took this approach (which I agree with you is misguided and results in a lot of tough labor for teachers). In a separate (not yet published) study where we looked at 30 districts across the state of California, we did find two districts that had taken this approach. Teachers were universally negative about it. But I don’t know of research on the achievement effects of this approach (again because getting data on who is doing this is very difficult).

Question: What does effective differentiated instruction look like in curriculum resources?

Answer: This is a really great question. I think differentiation is one of the main reasons why teachers supplement, and I think teachers often will tell you that their core materials are not very well positioned to help them differentiate. But I cannot say what materials do this better than others. It’s a great area for future work – how can we design core materials that ALSO provide teachers the differentiation support they need?

Excerpt from Polikoff, M. S., Campbell, S. E., & Korn, S. A. (2018). Using quantitative and qualitative methods to study the content and effects of curriculum materials. In C. R. Lochmiller (Ed.), Complementary research methods for educational leadership (pp. 193-212). New York, NY: Palgrave MacMillan.

Much of the research on curricular materials is grounded in the opportunity to learn literature (see McDonnell, 1995). Carroll (1963) introduced the concept of opportunity to learn (OTL) as one of several factors that affect student outcomes, along with aptitude, perseverance, and quality of instruction. Students’ opportunity to learn a given topic is determined by teacher- and school-level decision-making. Factors contributing to OTL include the length and content of classes, the order in which lessons are taught, the amount of time given to teach a specific topic, and the content to which students are exposed through curriculum materials (Kurz, 2011).

Curriculum materials have a direct effect on student learning through exposure, and they have an indirect effect on student learning through teacher decision-making, which is often influenced by the content and sequence of textbook lessons. OTL represents a guiding framework when researchers consider things like the equity of access to high-quality materials across socioeconomic groups (e.g., Elmore & Fuhrman, 1995). OTL has been used as a central argument in court cases ruling that students from low socioeconomic backgrounds should have access to the same quality of materials as their higher-SES peers (e.g., Eliezer Williams et al., v. the State of California et al., 2000). In these court decisions, curriculum materials have been cited as a key policy lever associated with students’ OTL.

Opportunity to learn can be defined and measured across multiple levels of curriculum. Some research (e.g., Porter & Smithson, 2001) differentiates between the intended curriculum, the enacted curriculum, and the assessed curriculum. The intended curriculum refers to the skills that students are expected to know as determined by factors such as state standards, district pacing guidelines, or the content of a teacher’s guide. The enacted curriculum refers to the curriculum that is actually delivered to students in class via instructional materials and teaching methods. The assessed curriculum refers to the skills and content that students are held accountable for learning through direct or indirect assessment. Curriculum materials such as textbooks form a bridge between the intended and the enacted curriculum because they offer a publisher’s interpretation of the intended curriculum, for which teachers are then given guidance for enacting in class. The Common Core State Standards represent the current intended curriculum in many states, which publishers have used as a guideline for producing a new set of materials they claim to be aligned to the standards.

Curriculum materials offer teachers suggestions regarding the sequence and pacing of lessons, the scope of the subject matter covered in a year, and the strategies to be used for instruction. The choices that teachers make regarding the implementation of the lessons shape the enacted curriculum. Textbooks and other curriculum materials are an important factor in the relationship between the intended and the enacted curriculum, and therefore are a contributing factor to differences in student opportunity to learn.

Not every teacher uses traditional textbooks as their primary source of instruction, but it is clear that teachers do consider these materials in shaping their lessons. A recent representative sample of teachers from the American Teacher Panel indicated that, while they use a variety of sources to plan and implement lessons, the majority of teachers still use traditional textbooks for at least some, if not all, of their planning (Opfer, et al., 2016). This survey evidence suggests teacher use of materials is changing during the Common Core era. One of the factors contributing to this is the proliferation of non-traditional materials such as open online educational resources, interactive or hybrid textbooks (with features online and in print), and websites where teachers share materials with other teachers (either for free or for a charge).

District leaders and policymakers recognize that curriculum materials are a potentially important policy lever in improving student achievement. Much of the concern over adopting a good textbook is related to the emphasis placed on the curriculum as a primary tool for helping to improve instruction (e.g., Remillard, 2005). However, the textbook itself does not do most of the teaching; it is the role of the teacher to interpret the lessons and implement them. A teacher may implement the textbook with varying degrees of fidelity, and researchers must recognize the participatory stance that a teacher takes with regard to the book. The teacher may implement the lessons with complete fidelity; may use the textbook as one of many available resources; or may interpret and interact with the text in a co-constructive relationship (Remillard, 2005).  The relationship between teacher and curriculum materials is one that requires further research, especially in the context of the emergence of open online resources and other technological advances. Teacher implementation of materials is an important variable for researchers studying the effects of textbooks on student outcomes.

In spite of the fact that teachers exercise substantial discretion when implementing materials, recent rigorous quantitative evidence suggests that simply using one book over another can produce meaningful direct effects on student achievement. One recent large-scale study (Agodini, et al., 2010) randomly assigned elementary mathematics textbooks to schools, finding statistically significant differences in their effects on student learning. These differences were .10 to .15 standard deviations in magnitude, enough to move students from the 50^th to the 57^th percentile, depending on grade. Three recent studies use quasi-experimental matching methods (which we describe below) to investigate the impact of elementary mathematics textbooks in California, Florida, and Indiana; again, each study found that at least one textbook outperformed the others (Bhatt & Koedel, 2012; Bhatt, Koedel, & Lehmann, 2013; Koedel, et al., 2016). Together, these four recent studies provide compelling evidence that textbooks matter for student learning, irrespective of teacher use.

Breakout Session: Leadership & Governance

Question: Meeting expected standards of performance in Standards-based teaching and learning (SBTL), is dependent on the adequacy, equity, and access to aligned resources to support the implementation of the standards. This is often compounded by variations in the contexts of schooling (demographics, geographical, socioeconomic locale, etc.). How do we get policies to address these considerations while simultaneously implementing and measuring outcomes of SBTL?

Answer: In the early 2000’s, standards were treated as a panacea for large resource deficits between schools with different funding streams and socioeconomics. Recent research by Kirabo Jackson, Barbara Biasi, and others demonstrates the importance of school finance equalization. Without material changes in the conditions of schools, we cannot expect fidelity to implementation of curriculum standards, as more immediate concerns (such as lead-free healthy learning environments, teacher turnover, and manageable class sizes, to name just a few) will impede instructional change.

It is necessary for policies to take into account local context and resources. This needs to be built into policies in an explicit way, and state education agencies should consider this as they determine how to best support their local education agencies.

Question: How you decide what students can achieve with support and students with substantial cognitive deficits who are basically tortured by making them try to reach standards that they can't possibly reach.  I'm not talking about students who can function at or near average but those who are seriously cognitively disadvantages?

Answer: IEPs play a key role in this. These are case-by-case considerations that should involve special education experts, families, and educators to determine how best to meet the students’ needs. We should be wary, however, of excluding too many students from performance-based accountability, as it can allow schools to avoid providing adequate accommodations as has been the case historically in special education.

Question: How does the panel recommend we move forward for next academic year...given the deficit thinking about more gaps in learning? How can we get districts, teachers to move to an acceleration, aggressive model of addressing opportunity gaps?

Answer: Holding up and sharing models of successful approaches may help inspire and motivate districts and teachers to take a more active approach. We should also be mindful of the first question – resource deficits can explain learning deficits, and districts need the resources to accelerate instructional change and move entire curricula online during the time of COVID-19.

Question: Local control can mean that districts and schools have the ability to differentiate based on student need, which I fully support.  It can also, though, mean that there is a completely laissez-faire attitude in which practice has little to do with the needs of students. How can states do a better job of differentiating policymaking and supports based on district willingness and capacity to support high quality student learning?  We've talked about this for a long time, but few if any policies achieve it.

Answer: This is a challenge. States might create a system of supports and accountability, which uses trends in metrics to determine the extent of capacity districts have to serve their students, and state supports/accountability can be calibrated to that. There are states, such as California, that use Local Control Accountability Plans (LCAP) to create a formalized approval process by which the state can provide feedback without prescribing solutions.

Question: What is leadership's role in ensuring a holistic approach to all standards of disciplines and preventing a school from becoming top-heavy in English and Math standards focus, possible to the detriment of other subjects?

Answer: Leadership must create a culture of valuing multiple subjects, the arts, athletes, etc., and follow through with supports such as resources and time for all subjects. Instructional leaders can also identify synergies across content areas, such as reading subject area texts (e.g., nonfiction science) in ELA.

Question: How does a leader ensure standards are being followed in every course and class?

Answer: Distributing leadership through lead teachers is one approach. Another is allotting time during the school day for collaboration and co-planning so that teachers can generate standards-aligned lessons collectively.

Question: I would agree with a framework, but what do you do for educators who do not have buy-in into a consolidated framework such as Danielson, when research says many NBCT don’t follow it?

Answer: Building buy-in is critical. One approach is to create opportunities for teachers to observe how such a framework can be helpful for their own practice. Peer observation opportunities are one way of doing this.

Question: How can we leave out physical education, art, music, and others?

Answer: We shouldn’t. Integrating these areas across academics is one way of ensuring they are a fundamental part of schooling.

Breakout Session: Students with Disabilities

Question: More often than not we, special education educators, are torn between writing a standards-based IEP goal when the student may not be able to read at the grade-level intended. Are you able to give an example as to an IEP goal for a student that cannot read at grade-level, but given a standard that requires grade-level reading?

Answer: Skilled reading, at any grade level, comprises multiple skills. As posited by the Simple View of Reading, these can be divided into two broad categories: (1) word recognition and (2) language comprehension. Many college- and career-readiness standards fit well within this model. In determining a student's instructional goals, this model can be helpful. For example, some students may demonstrate significant word recognition difficulties that prevent them from independently accessing grade-level texts. In those situations, I often use selections from my students' grade-level texts for fluency building (e.g., repeated readings) and word learning (e.g., morphological analysis) exercises. In addition, I will often pair these efforts with systematic phonics instruction. To develop my students' comprehension, I will often begin by providing a read aloud accommodation (e.g., print-to-speech software, peer or adult tutor). With this support, our focus turns to building the knowledge and skills needed to understand and learn from grade-level texts (e.g., determining key ideas and details, attending to the craft and structure of a text, integrating knowledge across multiple sources). For additional information, Diane Browder and colleagues have written several books and produced many other great resources on designing and implementing standards-based individualized education programs. 

Question: How do we change the mindset of general educators that SWD education are ALL of OUR responsibilities and adopt the "inclusive body that works"?

Answer:  Mindshift will occur when we see enough evidence to challenge the dominant beliefs.  Every pilot I taken to scale followed a similar pattern: a) research the “inclusive body that works” b) identify an innovation to target, and c) follow implementation science stages.  First, pitch the innovation to all stressing “the why” (what need does it fill for all students and how will it help the teachers/staff).  Include SWDs in this, but do not make that population the only “why”.  Pilot/start out small with early innovators and make supporting the implementation a priority.  At evaluation cycle pause points (e.g. winter and spring screening), share out with the entire staff to build buy in: innovation classroom growth/outcomes over business as usual classrooms (with SWDs data disaggregated as a second data point to move along that mindshift), video of implementation, the implementing teacher sharing out about the experience.  If these things are all positive, my experience has been other teachers/staff will ask to come on board.

Question: If a student's IEP goals are grade level standards, these goals are not individualized.  I understand that this prompts individualized instruction. However, if a student has not mastered their independent level standards, don't we need to focus on mastering these? How will a student master grade-level standards without foundational skills from previous grades?

Answer: Right. Prerequisite skills, often foundational skills, are necessary for SWD to meet state standards. The importance of attending to these skills must be explained in the context of state standards. 

Question: You touched on the issue of "choosing" IEP goal vs. Standards in the regular classroom.  How do Spec. Ed teachers approach G.E. teachers to buy into this especially consider time available each day?

Answer: Special education teachers need to explain the logic of attending to prerequisite skills in the context of addressing state standards. The “selling” of this will be easier if the special educator is willing and able to assist in the teaching of the prerequisite skills.

Question: How do we put more pressure on states to implement UDL?

Answer: I don’t know of any credible research to support UDL as an inclusionary strategy. I would urge caution especially before buying expensive UDL programs.