ID Bar
Feature headline
Volume I, Number 6, June, 2003

Not One Dime!

RON PAIGE: We won't spend a dime on programs that won't work. If it's not working, we won't spend money on it.

This comment from the Secretary of Education was made during a PBS interview first broadcast June 2, 2003.

REPORT CARD: READING FIRST
In the first of a series of reports on education, John Merrow looks at "Reading First," a cornerstone of the Bush administration's major education initiative "No Child Left Behind."

The Secretary's claim may not be entirely accurate. The DOE is funding a summer reading program - a pilot that may be extended nationally - for 40,000 students in the Atlanta school district, the primary element of which is independent summer reading. This article looks at the DOE's involvement in the creation of this program and questions whether the design of this program included the kinds of science based research methodology the DOE is insisting local districts employ.

  • Are the strategies employed by this program based on a thorough search of the best research evidence as to what works and what does not work when launching summer reading programs?
  • Were the specific elements of this program such as reading lists, reading logs, recognition strategies and reading partnership strategies carefully selected to optimize results?
  • Is there an evaluation design that meets the high "scientific standards" that the Secretary often mentions as being essential before expanding such a program or "spending a dime on programs that won't work?

1. First - A Commendation to Atlanta and the DOE for a Great Start

The more I looked into this project, the more impressed I became by the energy, enthusiasm and activity the school district has been able to mobilize to encourage summer reading. There are many aspects of this program that are admirable and exciting.

Partnerships - With the support of the DOE, the district has attracted an impressive collection of groups to support the effort. Click here for a complete list of the 16 major sponsors. A few examples . . .

First Book
First Book has committed to provide an age appropriate book for all elementary and middle school students enrolled in Atlanta Public Schools.

Target Stores
Target Stores has committed to provide volunteer tutors to support students participating in the Summer Reading Achievers program.

Scholastic Inc.
Scholastic, the world's largest publisher and distributor of children's books, has committed to provide books for students and school libraries in the Atlanta Public Schools.

Atlanta-Fulton County Library System
The Youth Services Librarians of the Atlanta-Fulton Public Library system will visit schools in their service areas to promote Summer Reading and invite students to get registered for the Library's Summer Reading program.

According to Elizabeth L. Matthews, Director of Communications, First Book gave 37,000 books to this program in May of 2003. At the beginning of the summer, an additional 20,000 books were provided - 10,000 additional to the Public Schools and 10,000 to the Boys and Girls Clubs of Metro Atlanta. A national program devoted to putting brand new books in the hands of children from low income families, First Book's Web site is at http://firstbook.org/.

Commitment - It soon became evident that the school district is solidly behind this summer reading effort, as everybody from Superintendent Dr. Beverly Hall through the ranks of the administration and schools themselves seems intent on making this project a success.

2. Some Design Issues

Because this program is being heralded as a pilot program that might be extended nationally, this article will take a look at some aspects of the DOE program that relate to the Secretary's promise of scientifically-based programs and evidence of success before spending money.

We would expect a national demonstration program to include and test various best practices in a way that would identify the relative merits of each.

The Reading List - Any Old Book will do?

The reading list for this program just had to be withdrawn because of errors. According to the AP story, some librarians have also criticized the list as being archaic.

We would expect that the DOE would have consulted with librarians and experts in the field of reading before publishing an archaic list. Good lists with current titles designed to entice young readers could be a crucial factor in a program's success.

Summer Reading List Pulled From Web Site
Wednesday, June 4, 2003

By VANESSA PALO, Associated Press Writer

"I don't know if someone pulled out a really old bibliography from a file cabinet somewhere," Nancy Margolin, a media specialist at McDougle Elementary School in Chapel Hill, N.C., said Wednesday. "These don't seem like the kinds of lists that would be provided by librarians."

Margolin added that because of the dated nature of many books, the suggested titles "are not as reflective of a diverse population."

Department spokesman Dan Langan said "proofreading and clerical errors" led to the misprints.

Related Articles from EducationNews.org

"The DOE can't even proof its materials!" - May 2, 2003

Response from John McGrath, Senior Director, Office of Intergovernmental & Interagency Affairs, U.S. Department of Education

More comments directly from Nancy Margolin . . .

I reviewed the elementary lists which are rife with errors. In addition to the incorrect titles and misspelled authors' names, which are rampant throughout all the lists, there are other issues as well:

  • The lists are very weak with regard to current titles. Although there are indeed many "old classics" which are still widely read by children, there is no excuse for these lists to be so heavily weighted with so many old, dated titles. For example, the 2nd grade list of 25 titles or series includes only six written since 1990. Seven of the titles were written prior to 1970.
  • Is there a commitment to including multicultural titles? It is often the more current titles that do a better job of reflecting a more diverse population.
  • Is the information in very old non-fiction titles still accurate? The 3rd grade list includes six non-fiction titles; four were written in the early 1980s. Subjects include space and endangered animals; is this information still correct? In addition, many of the titles on several of the lists are currently "out of print". (Available in some libraries, but not readily available in bookstores.)
  • Are these titles really appropriate for the specified grade levels? The lists are short, approximately 20 - 25 titles per grade level, and while it is important to acknowledge and provide for a range of "reading levels" at each grade, the titles should be books that would interest children at that particular age and grade. In the 1st grade list, there is a very high proportion of books much more appropriate for a preschool audience.

All of us who are involved in public education work hard to encourage and inspire children to read throughout the year. Thoughtfully and carefully prepared summer reading lists can provide a great service to students and their families. However, these lists are an embarrassment. Rather than encouraging reading, these lists, with their outdated titles and pervasive errors, will frustrate those who try to use them.

Nancy Margolin, Media Coordinator
McDougle Elementary School
890 Old Fayetteville Rd.
Chapel Hill, NC 27516

The Department's attempt to identify appropriate reading for these children belies the claims of science and expertise made by the Secretary. Here is a case where money should have been spent on the development of a current and effective reading list.

ROD PAIGE: Reading First says that teaching reading is a science, and we've been acting like it's an art. (Another quotation from the PBS interview.)

In this case, it seems the Federal architects of this reading program failed spelling, science, and art. Making wise choices of books for children to read is certainly not a simple scientific choice. How much was spent on this reading list? On this program?

The Secretary draws a false dichotomy by putting so much stress on science.

The Research Base

There is a substantial body of research documenting what is called "Summer Reading Loss." One example would be Cooper, H., Nye, B., Chariton, K., Lindsay, J. & Greathouse, S. (1996). "The effects of summer vacation on achievement test scores: A narrative and meta-analytic review." Review of Educational Research 66, 227–268.

These studies help to describe the problem and some of its causes, but they do not provide strong data on how to change the patterns. Most of the writing about reversing summer reading loss is speculative at best.

The logic goes like this . . . Affluent students and good readers with books at home do not suffer summer reading loss. If we buy books for disadvantaged students, reading loss will disappear as it does for good readers.

One pair of researchers promotes putting books in the hands of disadvantaged students but stresses the importance of identifying the "right books," especially for struggling readers. If the book is too hard or not matched to the child's interests, the program may not succeed.

"Lost Summers - For Some Children, Few Books and Few Opportunities to Read" by Anne McGill-Franzen and Richard Allington, ASCD's Classroom Leadership - August 2001 | Volume 4 | Number 9 http://www.ascd.org/publications/class_lead/200108/mcgillfranzen_2.html

Effective intervention to make students successful is a greater challenge than getting students to read books in the summer. Allington provides a comprehensive bibliography. Click here.

Program Assessment

According to the press release issued by the Department, the research design for this program does not match the Secretary's own insistence upon scientific design:

To help determine the pilot's effectiveness and applicability, the following accountability measures have been incorporated:

* Students are required to submit reading logs detailing the number of books read and what they have learned by the end of August 2003.
* Principals will certify that 70 percent of their students completed the program.
* Data from Atlanta Public Schools reading assessments will help to determine if the campaign helped mitigate the loss of K-8 reading skills that occurs over the summer and improve scores district-wide.

When I interviewed the architect of this program to gain a clearer picture of the data analysis planned, John McGrath indicated they planned to compare the reading progress of those who participated (read 10 books) with those who did not.

Investing in pilots makes lots of sense, but this particular program fails the test of scientific research because it does not include control groups. A full experimental design would have tried out a half dozen different types of summer reading programs and compared results before trying to expand the program nationally.

"Reading is the one skill upon which all others depend, and making sure that every child in our schools learns to read is a mission of the President's and mine," Secretary Paige said. "Research tells us that children who don't read during the summer may lose a month or more of the reading progress they had made in school.

"I'm pleased to announce the launch of the No Child Left Behind Summer Reading Achievers program. We have joined forces with Atlanta Public Schools in a pilot program to make sure that when students show up for the first day of school, they will be ready to learn. (Emphasis added.)

While we know that students who don't read in the summer often loose ground compared to those who do, where are the DOE studies documenting the success of programs comparable to the Atlanta program?

The Secretary's claim for this program is ungrounded . . . "to make sure that when students show up for the first day of school, they will be ready to learn." (Emphasis added)

Asking students to read good books is a great idea, but where is the data to support his high hopes for this particular strategy? Is this an example of scientific decision-making? Hardly. More like wishful thinking.

The data on the impact of summer school interventions with more rigor than a summer reading program is not encouraging.

For years school districts have sent children to summer school believing that this intervention would result in increased reading and mathematics achievement. Yet, five independent reviews of the research indicate that the effects of summer school on disadvantaged students' achievement levels are negligible.
"Preventing Summer Reading Loss During Summer Break"

By John Schacter in The Achiever, June 1, 2003 • Vol. 2, No. 10

Some Options Worth Assessing

A scientific research design for a national prototype should consider the relative value of various options so that the national expansion of the prototype would include only the best practices.

1. Does student reading performance improve more when the independent reading is done with a partner/tutor than when it is done completely independently?

2. Does student reading performance improve more when the independent reading is done with a partner/tutor who has been trained to emphasize comprehension when discussing what the child has read?

3. Do different kinds of reading partners tend to have different impacts? Are parents generally more effective than community volunteers, for example? Are adult tutors more effective than teen tutors?

4. Does the design and focus of a reading list influence the enthusiasm of the readers? If there is a focus on issues and topics that are particularly intriguing to urban youth, does that enhance the learning?

5. Are some reading logs more effective than others? Do rigorous and intriguing questions improve student success?

6. Are there effective ways to engage students in thoughtful analysis of the books they have read to improve their skills in analysis, interpretation and synthesis? Do weekly book clubs and discussions help? Online discussions? Other approaches?

7. Are some recruitment strategies more effective in winning the participation of poorly performing, reluctant readers than others?

8. Can we identify types of children who fail to participate or drop out without reading all ten books? Are there different strategies that might work with them?

9. Did our program reach those who need it the most?

10. What school factors contributed to the success or failure of this program?

A scientifically designed pilot program would investigate these issues and options in a scientific manner so that the resulting findings could provide insight to guide the development of future programs, but interviews with the DOE revealed that these issues had not been considered and are not built into the study or its assessment model.

To take a closer look at one of the options . . .

Some reading programs make use of quizzes to verify the quality with which students did their reading. How did the DOE decide to use a simple reading log in the Atlanta pilot program instead of investing in this strategy or others? Was it a decision based on scientific data, cost or convenience?

Quoting from Scholastic . . .

Scholastic Reading Counts! is a reading motivation and management program that helps you encourage and monitor independent reading.

Here's how it works:


Students select books from a database of over 23,000 titles from a wide variety of publishers.

Students read the books at their own pace.

Students take short quizzes on the computer to verify comprehension.

Click here for a tour of the program.

Would students achieve more growth with the use of a program such as Scholastic's Reading Counts which includes verification than they would using only a self-reporting reading log?

For a copy of the reading log used in the DOE program, click here for PDF form from the DOE Web site.

The reading log asks the student to list the title, the author and what they liked about the book.

According to Seth Coleman, Public Information Officer for the Atlanta Public Schools, the district was not granted additional funds by the DOE to launch this summer reading program. Perhaps a more robust and effective program would have emerged if there had been funding to add some of the elements mentioned above.

Here is what the Department Web site says about the program. The words "if successful" have been stressed to show that this is an experiment.
No Child Left Behind
Summer Reading Achievers Pilot Program
The No Child Left Behind Summer Reading Achievers program is designed to encourage Atlanta students in grades K-8 to read during the summer months and help prevent fall-off in reading skills during the vacation. The program is simple. Students who read 10 books over the summer will receive a variety of prizes, including free books and a Summer Reading Achievers certificate. This year's program is a pilot conducted by the U.S. Department of Education and the Atlanta Public Schools. If successful, the program will expand next year to schools nationwide.

No specific "scientific" research to support this program's strategies is cited on any of the DOE pages devoted to this program.

How did the Department settle on a program that had a focus primarily on the reading of ten books when elsewhere it cites research on summer reading that argues for very different strategies? The Federal "No Child Left Behind" Web site features an article by John Schacter of the Milken Foundation , "Preventing Summer Reading Loss During Summer Break," that outlines five conditions for successful programs.

Schacter bases his conditions on research he conducted in Los Angeles. A summary of his findings is available at http://www.nochildleftbehind.gov/Newsletter/#2.

1. Implement a research-based reading curriculum. Three large-scale reviews of kindergarten to third-grade reading research have demonstrated that the following five instructional elements lead to increases in young children's reading achievement: (1) explicit phonemic awareness instruction, (2) explicit systematic phonics instruction, (3) guided repeated oral reading, (4) direct and indirect vocabulary instruction, and (5) explicit teaching of comprehension skills.

2. Start early. Most summer school programs across the country start too late in the child's academic career (e.g., third grade).

3. Make summer learning fun.

4. Intervene for eight weeks instead of four. Most summer schools are four or six weeks. This is too short of a time to develop consistency and make headway with young learners.

5. Tutor summer learners the following school year. Do not stop working with students who need it most just because they are back in school.

"Reducing Social Inequality in Elementary School Reading Achievement: Establishing Summer Literacy Day Camps for Disadvantaged Children." By John Schacter, Ph.D.
http://www.mff.org/pubs/reading_camp_study_2001.pdf

Does this DOE project pass the SBR Test? (SBR = Scientifically Based Research)

The DOE project seems to ignore the scientific design principles the Department advocates for research and decision making because it does not include control groups or compare the effectiveness of various interventions. It falls into the research trap of comparing DOING SOMETHING with DOING NOTHING.

Predictably, DOING SOMETHING often beats DOING NOTHING, but we don't learn much from these studies. If we compare the effectiveness of six different strategies, we learn much more that is useful in guiding future projects. Comparing strategies allows us to get the most return on our investments - whether they be dimes or millions.

To prove a strategy effective in scientific terms, one has to set up a controlled study to guard against the Hawthorne Effect and to eliminate confounding variables.

Scientifically based (Experimental) Research - "The Gold Standard"

* Is quantitative in nature
* Determines potential links between practice or program and student achievement
* Controls all the:

* Environment
* Intervention (program or practice)
* Subject selection

Source: The School Improvement KnowledgeBase
Developed by the Region VII Comprehensive Center & Northrop Grumman Information Technology

http://www.helpforschools.com/sikb/reference/levels_research.shtml

To prove a strategy effective in scientific terms, one has to control for SES (socioeconomic status).

The Importance of Artful Reading Strategies.

The Secretary's disdain for the art of teaching reading is unsettling, for it shows disregard for the creative aspects of appealing to students' interests, wishes and dreams. The reading list fiasco shows what happens when inventors translate this disregard for the art of teaching reading into actual program design. If we want to invent a great summer reading program, we hire a team of reading specialists and school librarians who have many years of experience and familiarity with best practices.

Current attempts to reduce reading experiences to carefully scripted events show disdain for the craft of teaching and the strategic judgments made by excellent teachers to change the performance of students. Matching good books to students is not a simple matter of science. It takes empathy, skill, art and imagination. Making sure that the summer reading is done by students with comprehension and quality requires artful design.

"What did you like about this book?" is the only question asked of the Atlanta readers.

The Three Questions

The article began with three questions . . .

  • Are the strategies employed by this program based on a thorough search of the best research evidence as to what works and what does not work when launching summer reading programs?
  • Were the specific elements of this program such as reading lists, reading logs, recognition strategies and reading partnership strategies carefully selected to optimize results?
  • Is there an evaluation design that meets the high "scientific standards" that the Secretary often mentions as being essential before expanding such a program or "spending a dime on programs that won't work?

While there is much to applaud in this summer reading program, the current research design does not produce the kinds of data required to support thoughtful program expansion.

The answer to all three questions is negative.

There are critically important strategic issues, elements and possibilities that deserve consideration and scientific study before the prototype becomes the main model.

© 2003, Jamie McKenzie, all rights reserved. This article may be e-mailed to individuals by individuals, but all other duplication, distribution, publication and use is prohibited without first receiving explicit permission. Contact for information.
What can you do to change this law before it does great damage to the schools and children in your state and town?
  1. Subscribe to "No Child Left" to stay informed about efforts to repeal NCLB. Click here.
  2. Speak with the school board members, administrators and teachers in your community to learn how NCLB will change schools and learning in your town.
  3. Start communicating with your Senators and Representatives to let them know you want this law changed to put more emphasis on capacity building and support rather than testing and punishment.
  4. Write letters to the editor of your local newspaper expressing your concerns. Illustrate the dangers of this law with specific and compelling examples.
  5. Emphasize concrete alternatives that would do more to improve the futures of disadvantaged children.

A List of ESEA (NCLB) Amendments

1. Fund social programs that impact school readiness so that all children actually enter school ready to learn as the first President Bush promised long ago.

2. Fund capacity building (enhanced teaching and learning) in districts and districts for several years before engaging in punishing labels and reckless choice provisions. Capacity building might mean providing hundreds of hours of training in effective reading strategies, for example. But it does not mean training everybody in a single highly scripted program endorsed by the administration for pseudo-scientific reasons.

3. Devote public money to truly public schools. Be careful not to divert funds to reckless experiments or diploma mills.

4. Fund enough construction of new schools within public systems so parental choice is real.

5. Support informed school choice within public systems.

6. Emphasize rewards and incentives rather than sanctions.

7. Hold all publicly funded schools to standards for performance and quality, whether actually private, charter or truly public. Be careful about simplistic notions of high stakes testing.

8. Fund recruitment and preparation of effective teachers and aides from all racial and economic groups to close the gap between current staffing levels and what is desirable.

9. End the insulting, broad brush assaults on teachers and administrators struggling against difficult challenges.

10. Capitalize on the good research conducted to discover what works best in schools and avoid simplistic panaceas and platitudes imported from the world of business and medicine.

11. Enrich the options available to all children. Forswear tightly scripted, robotic programs and the fast food approaches to school improvement.

12. Build school improvement on a richly defined foundation of alternatives and strategies.

13. Eliminate Trojan horses, hidden agendas and shameful politics from ESEA.

14. Stop using Madison Avenue techniques to hide the harsh realities of so-called compassionate conservatism.