Citation Information

Goodrich Andrade, H. (2001, April 17). The effects of instructional rubrics on learning to write. Current Issues in Education [On-line], 4 (4). Available: http://cie.ed.asu.edu/volume4/number4/.


The Effects of Instructional Rubrics on Learning to Write

Heidi Goodrich Andrade
Ohio University

 

Abstract

This study examines the impact of instructional rubrics on eighth grade students' writing and on their knowledge of the qualities of effective writing. Students in the treatment group were given instructional rubrics that articulated the criteria and gradations of quality for three assigned essays. Students in the control group wrote the same three essays but did not receive the rubric. Students in the treatment group received, on average, higher scores on one of the three essays. Questionnaires administered at the end of the study revealed that students in the treatment group tended to identify more of the criteria by which their writing was evaluated.

 

Table of Contents

 

Arrow Up

Introduction

Rubrics are currently among the most popular innovations in education (Goodrich Andrade, 2000; Goodrich, 1997a, 1997b; Jensen, 1995; Ketter, 1997; Luft, 1997; Popham, 1997), but little research on their effectiveness has been undertaken. Moreover, few of the existing research efforts have focused on the ways in which rubrics can serve the purposes of learning and thinking as well as meet the demands of evaluation and accountability. The study described in this paper investigates the impact of instructional rubrics on students' written compositions and on their knowledge of the qualities of effective writing.

A rubric is usually a one- or two-page document that lists the criteria for a specific assignment and describes varying levels of quality, from excellent to poor. "Instructional rubrics" are rubrics that have been explicitly designed to support as well as to evaluate student learning (Goodrich Andrade, 2000). Instructional rubrics have several features that support learning:

  • they are written in language that students can understand;
  • they define and describe quality work;
  • they refer to common weaknesses in students' work and indicate how such weaknesses can be avoided, and;
  • they can be used by students to assess their works-in-progress and thereby guide revision and improvement.

Although the format of an instructional rubric can vary, most rubrics have two features in common:

  1. a list of criteria, or what counts in the evaluations of a project or assignment, and
  2. gradations of quality, or descriptions of strong, middling and problematic work.

Table 1 contains one of the instructional rubrics used in this study. Like each of the rubrics used, it draws on district, state and national standards as well as on feedback from teachers and researchers. It accompanied the following persuasive essay assignment:

The State of California has a law that all students must be educated until 16 years of age. This law passed after some debate. Some people thought it was a good law, some didn't. Put yourself in these lawmakers' shoes and argue either for or against this law. In a 5-paragraph essay, be sure to:
  • form an opinion on this issue and support it with strong arguments and relevant information, and
  • use your knowledge of democracy to explain how having or not having such a law would affect a democratic society like ours.


Table 1. Persuasive Essay Rubric

  4 3 2 1
Makes a claim Makes a claim and explains why it is controversial. Makes a claim but doesn't explain why it is controversial. A claim is made but it is buried, confused, or unclear. Does not make a claim.
Gives reasons in support of the claim Gives clear and accurate reasons in support of the claim. Gives reasons in support of the claim, but overlooks important reasons. Gives one or two weak reasons which don't support the claim well, and/or irrelevant reasons. Does not give reasons in support of the claim.
Considers reasons against the claim Thoroughly discusses reasons against the claim and explains why the claim is valid anyway. Discusses reasons against the claim, but leaves out important reasons, and/or doesn't explain why the claim still stands. Acknowledges that there are reasons against the claim but doesn't explain them. Does not give reasons against the claim.
Relates the claim to democracy Discusses how issues related to democracy can be used both in support of and against the claim. Discusses how issues related to democracy can be used to support the claim. Says that democracy is relevant but does not clearly explain how or why. Does not mention democracy.
Organization Writing is well organized, has a compelling opening, an informative body, and satisfying conclusion. Has appropriate paragraph format. Writing shows organization through a clear beginning, middle and end. Generally uses appropriate paragraph format. Writing is usually organized but sometimes gets off topic. Has several errors in paragraph format. Writing is aimless and disorganized.
Conventions Uses correct grammar, mechanics and spelling. May use complex sentences, sophisticated vocabulary, analogies, etc. Generally uses correct grammar. Some minor errors do not distract or confuse the reader. Shows some control of conventions but frequent errors are distracting or confusing to the reader. Writing shows little control of conventions. Serious and numerous problems distract and confuse the reader.


This rubric was designed to promote the development of writing skills by describing effective, genre-specific writing as well as the kinds of problems that students commonly experience as they write. Genre-specific criteria are helpful as guidelines for student writers because they "announce what is to be achieved in clear and useful language" (Cooper, 1999, p. 31). For example, the Considers Reasons Against the Claim criterion reminds students to acknowledge a perspective opposite their own and explain how this perspective is lacking. Research shows that students (as well as adults) tend not to consider contrary arguments (Perkins, Jay & Tishman, 1993), not because they can't do it but rather because they don't think about doing it. Including this criterion cues students to attend to an important component of a persuasive essay. The overarching principle here is that a rubric which reflects and reveals problems that students commonly experience provides more informative feedback than one that either describes mistakes they do not recognize or that defines levels of quality so vaguely as to be meaningless (e.g., "poorly organized" or "boring").

The Persuasive Essay instructional rubric was also designed to support the use of reasoning skills. The second and third criteria, Reasons in Support of the Claim and Reasons Against the Claim, give the rubric an emphasis on critical thinking—an emphasis missing from many rubrics. These two criteria inform students that critical thinking must be demonstrated in their essays and attempt to guide them in how (and how not) to do it.

 

Arrow Up

Theoretical framework

The hypothesis for this study is that instructional rubrics can have positive effects on students' writing and learning about writing. This hypothesis draws on several areas of cognitive and educational research, including authentic assessment, self-regulated learning, and the teaching and evaluation of writing. Perspectives on authentic assessment provide a guiding definition of assessment as an educational tool that serves the purposes of learning as well as the purposes of evaluation (Gardner, 1991; Hawkins et al., 1993; Shepard, 2000; Wiggins, 1989a, 1989b; Wolf & Pistone, 1991). The literature on self-regulated learning and feedback suggests that learning improves when feedback informs students of the need to monitor their learning and guides them in how to achieve learning objectives (Bangert-Drowns et al., 1991; Butler and Winne, 1995). Similarly, the literature on assessing writing recommends distinguishing between evaluation and grading by having students engage in a process of ongoing evaluation that provides precise and detailed information about what is expected for a particular assignment, as well as guidance on how students can improve their writing on that assignment (Cooper, 1999; Cooper & Odell, 1999; White, 1994; White, 2000).

Taken together, theory and research on assessment, self-regulation, and feedback suggests that instructional rubrics have the potential to scaffold students' writing if the rubrics and the writing assignment have certain characteristics. They must:

  • articulate clear, genre-specific criteria for the assignment;
  • provide guidance in meeting the criteria;
  • provide opportunities for improvement through revision;
  • be sensitive to students' developmental readiness by referring to appropriate grade level standards.

In this study, these principles for effective assessment were implemented by giving students instructional rubrics like the one in Table 1. The other rubrics that were used and their accompanying assignments can be found in Appendix A.

 

Arrow Up

Research Questions and Methods

This study was motivated by two research questions. The first question is: Does providing students with instructional rubrics affect their knowledge of the qualities of effective writing? A written questionnaire was used to uncover students' beliefs about "what counts" when evaluating an essay. The questionnaire consisted of one question; "When your teachers read your essays and papers, how do they decide whether your work is excellent (A) or very good (B)?" The question, which was borrowed from a study conducted by Dr. W. Haney of Boston College (personal communication, July 29, 1996), allowed for an examination of students' knowledge of the criteria by which their writing was evaluated and, by extension, of the qualities that define effective writing.

This study's second research question is: Does providing students with instructional rubrics affect the quality of their writing? This question was investigated by creating two groups of students—those who received an instructional rubric and those who did not—and comparing the average scores received on the essays written by each group. In this way, it was possible to determine, at least in broad stroke, whether or not rubrics can have a measurable effect on student writing.

 

Arrow Up

Sample

This project was supported by the Edna McConnell Clark Foundation, which asked that the work be carried out in schools with which the foundation collaborates. As a result, the research was conducted in nine eighth-grade classes in two very different middle schools in Southern California. One of the schools (School A) is located in an upper middle class, largely professional, suburban neighborhood with little ethnic diversity. Many of the non-White students that attended School A were bussed in from adjacent communities and tended to be placed in lower level classes. The language arts teachers with whom I worked in School A designed their curricula independently of each other. School B, in contrast, is located in an ethnically and linguistically diverse, working class, urban community. The teachers with whom I worked at School B collaborated on an integrated curriculum that combined history and language arts. Their shared Humanities curriculum drew explicitly on the district's standards and an experimental new portfolio process.

The combined sample from both schools included 242 students. Half of the students were boys and half were girls. One hundred and twenty-one (50.0%) were Latino, 86 (35.5%) were White, 31 (12.8%) were Black, and 4 (1.7%) were of Asian descent (Filipino, Chinese, Vietnamese, Cambodian or Laotian). Approximately 8% of the students were considered to have special educational needs, and 6% were identified as students for whom English was a second language (ESL). The average Humanities/language arts grade for the term prior to this study was 75.9% (or a C). One hundred and forty-one of the students in the sample attended School A and 101 attended School B.

 

Arrow Up

Procedure

The study spanned the 1996-97 school year. Students were asked to write three different essays approximately one month apart. Each assignment was designed to meet the individual school's curriculum and evaluation needs, so students in the two schools were twice assigned different essays. The first assignment for both schools was a persuasive essay. The second assignment was an autobiographical journey essay at School B, and an essay entitled "Oh, The Places We'll Go" in School A. School A's second essays were not included in this study because of problems in the implementation of the assignment. The third and final assignment was a historical fiction essay in School B, and an essay about a personal challenge in School A.

Before writing a first draft of each essay, students in the treatment classes were given an instructional rubric. As principle investigator, I introduced the rubric to students during one class period in one of the treatment classrooms while the teachers observed. The teachers of the treatment classes then introduced the rubric to their own classes while I observed. Students in both the treatment and control classes were asked to write first and second drafts of the essays, but the students in the control group were not given a rubric.

Approximately three weeks after the completion of the third essay, all students were asked to write a narrative response to the one-question questionnaire.

 

Arrow Up

Dependent Measures

Data were collected on two dependent variables: 1) students' responses to the written questionnaire, and 2) students' scores on the essays written for this study. Three of the four classes at School A and all five classes at School B filled out and returned the questionnaires at the end of the study, for a total of 196 complete questionnaires. I analyzed students' narrative responses to the questionnaire by noting all of the qualities of writing, or criteria, to which students referred, such as spelling, neatness, organization, "good ideas," and "whether [the teacher] likes me or not."

Three research assistants and I scored the essays. None of the research assistants had a background in research or writing and none were able to score more than one collection of essays, necessitating rather lengthy training periods and extended attempts to reach reliability for each assignment. We always began by reviewing the rubric for the assignment in order to come to agreement on the precise definition of terms and to "unpack" overlapping criteria. The resulting adaptations did not significantly change the meaning of the rubric, but aided in achieving scoring reliability. See Appendix B for the rubrics that we used to score the essays.

Each essay was scored on each criterion, then an average score was calculated. A total of one hundred and six persuasive essays were scored. Because of implementation problems in School A, the second essay written for this study was not used. Thirty-seven autobiographical incident essays from School B were scored. One hundred and sixty historical fiction/personal challenge essays were scored.

Reliability was checked by testing the correlation between the average scores assigned by two raters, and by determining the Cohen's kappa for the scores given for each criterion on the rubric. (Cohen's kappa is a measure of percentage agreement corrected for chance agreement: k = Po - Pe / 1 - Pe. ) Comparing only the average score would have produced higher rates of agreement but would also have masked disagreements about how students performed on the individual criteria. For the persuasive essay, a correlation of .93 and a Cohen's kappa of 70% were achieved by the two raters on the first twenty-six essays scored. For the autobiographical incident essay, the raters scored twenty-two essays together. A correlation of .73 and a Cohen's kappa of 60% agreement were achieved for the last six essays. For the historical fiction and personal challenge essays, the raters achieved a Cohen's kappa of 67% and a correlation of .74 after scoring thirty-five essays together. At the conclusion of this study the research assistants and I revised the scoring process, and a subsequent study (in preparation) had far higher rates of agreement. For the purposes of this study, however, the above rates of agreement were considered low but adequate.

 

Arrow Up

Independent Measures

Data were also collected on several independent measures, including school attended, teacher, grade level, gender, ethnicity, previous performance in English as measured by standardized test scores and grades, and identification as ESL or a student with special needs.

 

Arrow Up

Analysis

I analyzed the questionnaires by noting the criteria to which students referred, including academically relevant qualities like content and spelling, and academically irrelevant influences such as whether or not the work was turned in on time. I compared the kind of criteria referenced by students in the treatment and control groups to each other and to the criteria contained in the rubrics used in the study. The responses from students in School A and School B were analyzed separately because the students in the control group at School B had had previous exposure to rubrics used by their teacher.

The equivalence of the treatment and control groups on each of the independent variables was assessed using chi-square tests for categorical variables and t tests for continuous variables. The two groups were equivalent in terms of gender (x2 = .002, p = .96), ethnicity (x2 = 6.76, p = .24), number of students with special needs (x2 = .05, p = .82), number of ESL students (x2 = .62, p = .43), and previous grades in English/language arts (t = .02, p = .99). Because School A and School B used different standardized tests with different scales, equivalence in terms of standardized test scores was determined by comparing the treatment and control groups in each school to each other. At School A, the treatment and control groups were equivalent (t = -.34, p = .74). At School B, the control group had, on average, higher scores and the difference approached statistical significance (t = 1.91, p = .06). As a result, the sample at School B was biased against the treatment and the findings were likely to represent a conservative estimate of the treatment effect. For this reason, standardized test score (Test) was included as a high priority control variable in the multiple regression model building process.

Multiple linear regression was used to understand the relationship between the treatment, the independent variables, and the essay scores. The main effect of each predictor and its interaction with the treatment and with gender were tested. The effect of a predictor was considered statistically significant if its p value was < .05. Residual plots from the multiple regression models were inspected throughout the model building process to ensure that the assumptions of linearity, normality and homoscedasticity had not been violated.

 

Arrow Up

Questionnaires

The analysis of students' responses to the questionnaire revealed striking differences between the treatment and control groups. The students in the control group at School A tended to mention fewer and more superficial criteria such as spelling, punctuation, and neatness, if they mentioned any specific criteria at all:

Well, they give us the assignment and they know the qualifications and if you have all of them you get an A and if you don't get any you get a F and so on.

Note that this student knows that the teacher has her standards or "qualifications" but he does not suggest that he knows what they are. Students in the treatment group, in contrast, tended to mention the same criteria to which the control group referred plus a variety of others, including criteria contained in the rubrics used in this study:

Student 1: The teacher gives us a paper called a rubric. A rubric is a paper of information of how to do our essays good to deserve an A. If they were to give it an A it would have to be well organized, neat, good spelling, no errors and more important, the accurate information it gives. For a B it's neat, organized, some errors and pretty good information but not perfect.
Student 2: An A would consist of a lot of good expressions and big words. He/she also uses relevant and rich details and examples. The sentences are clear, they begin in different ways, some are longer than others, and no fragments. Has good grammar and spelling. A B would be like an A but not as much would be on the paper.

With the exception of "neat," all of the criteria referred to by these two students were included in the rubrics used in this study. Some of the criteria are quoted exactly as they were written in the rubric (e.g., "... sentences are clear, they begin in different ways..."), while others are paraphrased (e.g., "big words").

Table 2 is a list of the criteria from the rubrics that were mentioned by students in the treatment group at School A but not by students in the control group. The numbers to the left represent the number of times each criterion was mentioned by students in the treatment group. Students in the control group at School A did not refer to any of these eleven criteria, even by chance.


Table 2. Criteria contained in rubrics and referenced by students in the treatment group but not by students in the control group at School A (n = 74).

No. of references Criterion
20 Word choice, e.g., "words give [the reader] a vivid picture in her mind"
8 Voice, reveals feelings and emotions
7 Interesting, not boring
3 Has accurate information
3 Provides details
2 Is descriptive
2 Uses proper paragraph format
2 Includes ideas, thoughts and opinions
2 Makes a point
2 Is well-organized, e.g., ";has a beginning, middle and end"
1 Sentence structure


The results from School B are a little different because the students in the control group were accustomed to using rubrics. Seven students in the control class referred to the use of rubrics in their responses, even though they were not given the rubrics used in this study. Nonetheless, small differences in the treatment and control groups at School B were found. Table 3 is a list of the criteria contained in the rubrics used in this study and mentioned by students in the treatment group but not by students in the control group at School B.


Table 3. Criteria contained in rubrics and referenced by students in the treatment group but not by students in the control group at School B (n = 122).

No. of references Criterion
4 Word choice, "powerful words," "vividness"
4 Organization
3 Length, five paragraphs
3 Gives details
2 Tells about action and events
2 Is easy to understand
2 Ideas and content
1 Setting
1 The way the writing flows
1 Makes a point
1 Voice
1 Sentence fluency
1 Tells about lessons learned
1 Contains correct information


Arrow Up

Discussion of Questionnaires

When compared to the responses of students in the control group, students in the treatment group tended to refer to a greater variety of academically relevant criteria for effective writing. These differences suggest that the students who received the three instructional rubrics had more (if not complete) knowledge of what counts in writing and of the criteria by which their essays were evaluated. It appears that instructional rubrics have the potential to at least broaden students' conceptions of effective writing beyond mechanics to include qualities such as word choice, voice and tone. However, the results of the essay scores discussed in the following section suggest that, predictably, transferring students' new knowledge about effective writing to the composition of written essays is more difficult.


Arrow Up

Essay Scores

Table 4 lists the final regression models for each of the essays. The parameter estimates and p-values for the treatment condition reveal that there was a positive effect of treatment on the second essay (the autobiographical journey) but not the first or third essays. Interestingly, the negative parameter estimate for the interaction between treatment and gender for the third essay (historical fiction/personal challenge) indicates that there may have been a negative effect of treatment on girls' scores but no effect for boys.

Table 4. Parameter Estimates from Final Regression Models (See also Appendix D)

  Essay 1
n = 106
Essay 2
n = 37
Essay 3
n = 160
Intercept 1.57*** 2.18** 1.62***
Trt_Cntrl 0.0009 0.49** 0.12
Grades 0.01*** -0.005 0.009*
Test 0.010* 0.01~ 0.009~
Teacher -0.100**    
School 0.300~ (N/A) 0.22*
Gender   -1.78~ 0.51*
Grades*Gender   0.02~  
Ethnicity     0.20~
Trt*Gender     -0.43~
R2 % 25 40 19

~ p < .10
* p < .05
** p < .01
*** p < .001


Essay 1. There was no measurable effect of the treatment on students' scores on the persuasive essay. The only statistically significant effects come from variables with traditionally robust predictive power: previous performance in English, teacher, and school attended.

Essay 2. Because of implementation difficulties at School A during the writing of the "Oh, The Places We'll Go" essay, only the autobiographical essays from School B were scored. The results show that, controlling for grades, test scores, gender, and an interaction between grades and gender, students in the treatment group are predicted to score, on average, almost half a point higher on a 4-point scale than students in the control group. Figure 1 summarizes the effect of treatment graphically.

figure1

Figure 1. Relationship between essay scores, ASAT scores and gender for Essay 2, autobiographical incident (n=37)


The essays in Appendix C reveal some of the differences in the autobiographical journey essays written by students in the treatment and control groups. In general, the treatment essays attend more carefully to the purpose of the journey, character development, dialogue, action, paragraph breaks, and conventions. That is not to say that the treatment essays always deal with these criteria effectively; some attempts to meet the requirements of the rubric are limited and/or clunky. For example, the use of dialogue is usually minimal ("Then my dad said, 'Shut up or I'll turn around and drive straight home'"), and the lessons learned are often tacked on to the end of the essays as afterthoughts ("I learned a valuable lesson that day which was, 'Never do anything, that you now you're going to get in trouble for and regrete.'"). Nonetheless, the student writers in the treatment group were clearly attending to the criteria on the rubric and, by attempting to meet them, learning about writing. Although the teacher of the control group had had her students write down the criteria for the autobiographical essay, they did not have the full rubric at their elbows as they wrote, and their writing reveals fewer explicit attempts to fulfill the criteria.

Essay 3. The analyses of last two essays, historical fiction and personal challenge, were collapsed because the effect of treatment did not differ by school. Since the main effect of treatment is not statistically significant, there are no measured overall differences in essay scores between the treatment and control groups, controlling for the other variables (t = .72, p = .47). The main effect of gender is statistically significant (t = 2.22, p = .03), which shows that, on average, girls are predicted to score .51 points higher than boys, controlling for grades, test scores, and ethnicity. However, the interaction between treatment and gender approaches statistical significance (t = -1.76, p = .08), suggesting that the effect of treatment may be different for girls and boys. For boys, there was no statistically significant difference between the average essay scores for the treatment and control groups. Girls in the treatment group, in contrast, tended to score .31 points lower than girls in the control group, controlling for grades, test scores, school and ethnicity. Thus, it appears that there may be a negative effect of the treatment on girls' scores for this essay. Figure 2 represents this relationship graphically.

Figure 2

Figure 2. Relationship between essay scores, grades and gender fr Essay 3, historical fiction/personal challenge (n=160)


Arrow Up

Discussion of Essay Scores

Findings from the analysis of the essay scores paint an uneven but intriguing pattern of results. In general, it appears that simply handing out, reviewing and explaining instructional rubrics can orient students toward the criteria for writing as communicated by the rubric and can help students write to those criteria, but that a more intensive intervention may be necessary in order to help all students perform at higher levels consistently.

The lack of a treatment effect for the first assignment—the persuasive essay—may be due to several factors. For one, it was many teachers' and students' first exposure to a rubric. Only one of the eight teachers participating in this study had previously used rubrics: they may not have been adequately prepared to support students in their use. This is also a likely explanation for the fact that the teacher variable had an effect on scores on the first essay but not on the second or third essays: by the second essay, each of the teachers and their classes had at least some familiarity with rubrics.

A second reason for the lack of an effect of the treatment on the first essay may be that the rubric itself was not written in particularly student-friendly terms. The second and third rubrics were written in more accessible language. A third reason for the lack of an effect may be that the students did not have enough time to revise the essays. Several teachers reported that the three days the students were given to write and revise was inadequate. Students were given five days to write essays two and three. Finally, a power calculation suggested that this sample (n = 106, control group n = 30) only had a power of 31% to detect a small effect of treatment even at the relaxed alpha level of .10. A larger sample size may or may not have detected an effect.

Findings from the second assignment—the autobiographical essay—are positive yet conditional. On the one hand, the magnitude of the between-group differences for the second essay appears to be educationally as well as statistically meaningful. An average of a half-point difference on a 4-point scale is a 12.5% difference. This effect is all the more meaningful because of the minimal amount of classroom time taken by the intervention: less than forty minutes was spent on introducing and reviewing each rubric. On the other hand, the sample size for the second essay is limited (n = 37, treatment = 26, control = 11), so firm conclusions are not warranted.

The findings from the third assignment—the historical fiction and personal challenge essays—stand in partial contrast to the findings from the second assignment. Assignment 3 results indicate that instructional rubrics may actually be related to a detriment to the performance of girls but not boys. However, it is possible that the results of the last essay were confounded by end-of-the-year pressures. Teachers at both schools reported that the third essay assignment came just as their students were attempting to meet new, district mandated portfolio and exhibition requirements for graduation. One teacher referred to the time period when students were writing the third essay as:

a last ditch effort to complete their graduating exhibitions. Although the third essay would have been awesome to put in an exhibition, most kids were trying to take the easy way out (which was to revise something they already had rather than create something new). When push came to shove—finish exhibition and go to high school or finish the essay—high school won out.

Nonetheless, the possibility of gender differences in the ways students respond to the use of rubrics needs further investigation.

It is conceivable that the different results for each essay could also be explained in part by the fact that students were asked to write different kinds of essays, and different kinds of writing require different kinds of skills. Autobiographical essays, for instance, may be easier for students to write and to revise according to a rubric than persuasive essays or historical fiction in part because students are more practiced at telling their own stories. In addition, historical fiction and especially persuasive essays require a writer to decenter, or argue her point from a perspective other than her own. Although many junior high students can be expected to have the cognitive maturity needed to decenter, Moffett (1983) notes that writing from multiple points of view is a difficult skill that develops over a lifetime. It is possible that students' preferences for their own perspectives made the autobiographical essays easier to write. If that was the case, it is also likely that they found the autobiographical essay rubrics easier to understand and to use, hence the positive effect of the treatment. The implication, not surprisingly, is that instructional rubrics scaffold writing within students' zones of proximal development and no further. Like all instructional materials, rubrics should be designed with regard to students' cognitive development and skill level, aiming just beyond what students are able to do without assistance.

The relationship between gender and writing may come into play as well. For example, another study (Goodrich Andrade & Delamater, in preparation) also found that girls tended to earn lower scores on historical fiction essays than boys. It has been suggested (M. Donahue, personal communication, February 16, 2000) that girls have more difficulty writing historical fiction because most textbooks provide little information about the lives of women. Future studies of the effects of rubrics should be careful to assign writing assignments that give male and female students equal opportunities to succeed.


Arrow Up

Conclusion

Taken together, the analyses of the questionnaires and the essay scores indicate that simply handing out and explaining instructional rubrics can increase students' knowledge of the criteria for writing as communicated by the rubric, but that translating that knowledge into actual writing is more demanding. Although instructional rubrics show promise even in a minimalist intervention like the one applied in this study, positive effects on writing are not a given. The literature on teaching and assessing student work, as well as my own teaching experience, indicate the need for sustained attention to the process of writing, with the provision of instructional rubrics playing a key part—but not the only part—in providing helpful feedback to students. For example, I have found it useful to involve students in the design of rubrics, based on their own critiques of effective and ineffective sample work, but controlled studies of this approach do not exist. Research is needed on the most effective role for rubrics in the writing process and on the effect of rubrics on the performance of female students if the promises and pitfalls of this popular approach are to be understood and applied in ways that promote learning and development.

Notes: This study was conducted while the author was a principle investigator at Project Zero, Harvard Graduate School of Education. Correspondence should be addressed to Heidi Andrade, Ohio University, College of Education, 340 McCracken Hall, Athens, OH 45701. The author would like to thank Norma Jimenez and Beth Delamater for their assistance in analyzing the results of this study. Thanks also go to the Edna McConnell Clark Foundation for its financial support. The opinions expressed in this paper are the author's and do not necessarily reflect those of the Foundation.


Arrow Up

Author

Heidi Goodrich Andrade is an Assistant Professor at the College of Education, Ohio University. She took a position at OU after eleven years at Harvard University, where she received her doctorate and conducted research at Project Zero. Her teaching focuses on educational psychology and student assessment. Through her research she investigates the development of high-level thinking, with an emphasis on the relationships between thinking, learning and classroom assessment. She has worked with educators and businesses nationally and internationally on the design of thinking-centered instruction and assessments for a variety of contexts, including classrooms, after school programs, television shows and CD-ROMs. She has written numerous articles, including an award-winning article on rubrics (Goodrich, 1997), and is co-author of two books, Teaching through Projects: Creating Effective Learning Environments (Goodrich, Hatch, Wiatrowski & Unger, 1995) and Thinking Connections: Learning to Think and Thinking to Learn (Perkins, Goodrich, Tishman & Mirman Owen, 1994). Dr. Andrade can be reached via email at andradeh@ohio.edu.


Arrow Up

References

Bangert-Drowns, R., Kulik, C., Kulik, J., & Morgan, M. (1991). The instructional effect of feedback in test-like events. Review of Educational Research, 61, 213-238.

Butler, D. & Winne, P. (1995). Feedback and self-regulated learning: A theoretical synthesis. Review of Educational Research, 65(3), 245-281.

Cooper, C. (1999). What we know about genres, and how it can help us assign and evaluate writing. In C. Cooper and L. Odell (Eds.), Evaluating writing: The role of teachers' knowledge about text, learning and culture. National Council of Teachers of English: Urbana, IL.

Cooper, C. & Odell, L. (1999). Evaluating writing: The role of teachers' knowledge about text, learning and culture. National Council of Teachers of English: Urbana, IL.

Gardner, H. (1991). Assessment in context: The alternative to standardized testing. In B. R. Gifford and M. C. O'Connor (Eds.), Changing assessments: Alternative views of aptitude, achievement and instruction. Boston: Kluwer.

Goodrich Andrade, H. (2000). Using rubrics to promote thinking and learning. Educational Leadership, 57(5), 13-18.

Goodrich Andrade, H., & Delamater, E. (in preparation). Gender and the role of rubric-referenced self-assessment in learning to write.

Goodrich, H. (1997b). Thinking-centered assessment. In S. Veenema, L. Hetland, & K. Chalfen (Eds.), The Project Zero classroom: New approaches to thinking and understanding. Cambridge, MA: Project Zero, Harvard Graduate School of Education.

Goodrich, H. (1997a). Understanding rubrics. Educational Leadership, 54(4), 14-17.

Hawkins, J., Frederikson, J., Collins, A., Bennet, D., & Collins, E. (1993). Assessment and technology: New ways to assess student learning. Technology and Education, 36(5).

Jensen, K. (1995). Effective rubric design: Making the most of this powerful assessment tool. Science Teacher, 62(5), 34-37.

Ketter, J. (1997). Using rubrics and holistic scoring of writing. In Tchudi, S. (Ed.) Alternatives to grading student writing. Urbana, IL: National Council of Teachers of English.

Luft, J. (1997). Design your own rubric. Science Scope, 20(5), 25-27.

Moffett, James (1983). Teaching the universe of discourse. Boston: Houghton Mifflin Co.

Perkins, D. N., Jay, E., & Tishman, S. (1993). Beyond abilities: A dispositional theory of thinking. The Merrill-Palmer Quarterly, 39(1), 1-21.

Popham, W. J. (1997). What's wrong—and what's right—with rubrics. Educational Leadership, 55(2), 72-75.

Shepard, L. (2000). The role of assessment in a learning culture. Educational Researcher, 29(7), 4-14.

White, E. (1994). Teaching and assessing writing: Recent advances in understanding, evaluating, and improving student performance, 2nd edition. Portland, ME: Calendar Islands Publishers.

White, E. (2000). Bursting the bubble sheet: How to improve evaluations of teaching. The Chronicle of Higher Education, XLVII(11), B11.

Wiggins, G. (1989b). Teaching to the (authentic) test. Educational Leadership, 46(7), 41-47.

Wiggins, G. (1989a). A true test: Toward more authentic and equitable assessment. Phi Delta Kappan, 70(9), 703-713.

Wolf, D., & Pistone, N. (1991). Taking full measure: Rethinking assessment through the arts. New York: College Board Publications.


Arrow Up

Appendix A

Essay 2: Autobiographical Journey
Write about a journey you have taken. It could be a long trip, a short ride, a walk, or even a fantasy journey you experienced in your mind. Tell your readers a story that lets us enter your real or imaginary journey and understand what it means to you.

Instructional Rubric for Essay 2: Autobiographical Journey

  4 3 2 1
The purpose of the journey Tells where you went and why going there was interesting or important Tells where you went and why you went there Either where you went or why you went there is unclear Does not tell where you went or why you went there
The scene Vividly sets the scene, describing important sights, sounds, smells, and/or tastes along the way Describes the scene in detail, but not vividly Describes the scene at some point (usually the beginning) but some scenes are not described well Does not describe the setting of the journey
The cast of characters Creates complex characters by showing them in action, using dialogue, letting the reader overhear their inner thoughts, describing their appearance, personality, behavior, etc. Creates characters by describing them and using dialogue Describes characters but does not show how they speak, behave, feel, etc. Does not introduce a cast of characters
The action Tells one or two specific exciting, funny, unusual, or sad things that happened during the journey and why they were important Tells one or two specific things that happened but it isn't clear why they were important Tells one or two specific things but they aren't clearly written No specific events or actions stand out
Feelings, insights, lessons learned Reveals feelings about and insights gained from the trip, and draws a general lesson learned from it Reveals feelings about and insights gained from the trip Describes a variety of feelings and ideas, but doesn't have a central "vibe," insight or reflection Doesn't share any of the writer's insights or lessons learned
Organization Story has an interesting beginning, a developed middle and satisfying end. Correct paragraph format, at least 5 paragraphs Story moves through the beginning, middle and end in a logical order. Correct paragraph format, at least 5 paragraphs The story is usually organized but sometimes gets off the topic. Some problems with paragraphs and/or less than 5 paragraphs The story is aimless and disorganized. Incorrect paragraph and/or less than 5 paragraphs
Conventions Uses first person form, correct grammar, mechanics and spelling. Uses complex sentences, sophisticated vocabulary, etc. Generally uses correct grammar, mechanics and spelling. Frequent errors are distracting but do not interfere with meaning Numerous problems with fragments, run-ons, grammar, spelling, etc. make the story hard to read


Essay 3, School A: Personal Challenge

Write a 5 (or more) paragraph essay about a time you faced a challenge. Perhaps this challenge seemed impossible, discouraging, or scary. Perhaps you were helped through it by another person, or maybe you faced it on your own. Tell about this incident in detail, including who was involved, what the people and surroundings looked like, exactly what happened, your thoughts about the alternatives or ways you could have handled the challenge, and the final outcome. Try to make your readers understand why this particular event is memorable.

Instructional Rubric for Essay 3, School A: Personal Challenge

Criteria 4 3 2 1
Ideas and Content My paper tells of a difficult, discouraging or scary challenge; shows growth or change in the main character and how s/he coped; uses relevant and rich details and examples My paper tells of a challenge but it isn't made to sound very gripping; growth of the main character may be hard to see; uses relevant details and examples My paper tells of a challenge but it may not always be in focus; it may get off topic; it's not clear how the character grows; some details or examples may not matter or don't go together The challenge is murky; it is hard to tell what the topic is; it seems a little like random thoughts on paper
Organization My paper has a lead that establishes the challenge, a developed middle that builds tension, and a satisfying ending that resolves the problem, all in an order that makes sense, flows, and hangs together I have a beginning, middle and end in a logical order but without flair. My paper takes the reader on a walk but on a sidewalk, not a high wire My organization is rough but workable; my writing may drag its feet then race ahead; my ending may stop suddenly or drag on too long My writing is aimless and disorganized; there is little sense of a beginning or ending; it is probably confusing to a reader
Voice My writing has personality & sounds like a real person wrote it; it shows how I think and feel & sounds like it was written to be read My writing voice is engaging but may come and go, fading in and out My writing is bland or mechanical, sounds like I have not found my own way to say things There are no hints of the real me in my writing; it may sound like I don't like what I have written
Word choice The words I use are striking but natural, e.g., I use "terrified" instead of "scared," or "gut" instead of "stomach"; I use powerful verbs My paper has some fine word choices, but is often routine. My word choice is uninspired, colorless, and dull or sounds like I am trying too hard to impress; some words may be used incorrectly The same words are repeated over and over and over and over; some words may be bewildering and confusing to a reader
Sentence Fluency My sentences are clear; they begin in different ways; some are longer than others; no fragments; my paper is a delight to read out loud My sentences are well constructed; some minor errors in sentence structure; my paper marches along but doesn't dance My sentences are often awkward or mechanical; little variety in length; may have many sentences that begin with the same word My paper is tough to read because of incomplete sentences, run-ons, and awkward phrasings
Conventions I use the correct paragraph form, grammar, capitals, spelling, and punctuation I made some errors, mostly by taking risks and using interesting words or sentences My spelling is correct on common words; several errors in conventions are distracting Many errors in paragraph form, grammar, caps, spelling and punctuation make my paper hard to read


Essay 3, School B: Historical Fiction

Please write a letter from or journal entry of a 14 year-old American living in some year between 1491 and 1979. Take on the persona of your fictional character and write about a day in her or his life. Your 5 (or more) paragraph entry should tell what year it is and use historically accurate phrases and language to describe your living situation, clothing, hygiene, education, roles in family and community, work, food, etc. Also refer to relevant political, social, and/or religious events.

Instructional Rubric for Essay 3, School B: Historical Fiction

Criteria 4 3 2 1
Ideas and Content My paper brings the time and place in which my character lived alive; vividly describes her/his experiences and values; uses only historically accurate language and events My paper tells the time and place my character lived; describes a day in her/his life; most or all language and events are historically accurate The time & place my character lived is relatively clear, but his/her experiences are more like a list than a letter or diary entry; some language or events may be historically inaccurate The setting is murky; the character's language and experiences are often historically inaccurate; the paper may stray off topic or just ramble
Organization My letter/diary has a strong lead, a developed middle, and a satisfying ending, all in an order that makes sense, flows, and hangs together I have a beginning, middle and end in a logical order but without flair. My organization is rough but workable; my writing may drag its feet then race ahead; my ending may stop suddenly or drag on too long My writing is aimless and disorganized; there is little sense of a beginning or ending; it is probably confusing to a reader
Voice My writing sounds like a real person wrote it; it has personality; shows how I think and feel; sounds like it was written to be read My writing voice is engaging but may come and go, fading in and out My writing is bland or mechanical, sounds like I have not found my own way to say things There are no hints of a real person in my writing; it may sound like I don't like what I have written
Word choice The words I use are striking but natural, e.g., I use "terrified" instead of "scared," or "gut" instead of "stomach"; I use powerful verbs My paper has some fine word choices, but is often routine. My word choice is uninspired, colorless, and dull or sounds like I am trying too hard to impress; some words may be used incorrectly The same words are repeated over and over and over and over; some words may be bewildering and confusing to a reader
Sentence Fluency My sentences are clear; they begin in different ways; some are longer than others; no fragments; my paper is a delight to read out loud My sentences are well constructed; some minor errors in sentence structure; my paper marches along but doesn't dance My sentences are often awkward or mechanical; little variety in length; may have many sentences that begin with the same word My paper is tough to read because of incomplete sentences, run-ons, and awkward phrasings
Conventions I use the correct paragraph form, grammar, capitals, spelling, and punctuation I made some errors, mostly by taking risks and using interesting words or sentences My spelling is correct on common words; several errors in conventions are distracting Many errors in paragraph form, grammar, caps, spelling and punctuation make my paper hard to read


Arrow Up

Appendix B

Scoring Rubric for Persuasive Essay (adaptations to instructional rubric in bold)

Criteria 4 3 2 1
Makes a claim Makes a claim and explains why it is controversial. Makes a claim but doesn't explain why it is controversial. A claim is made but it is buried, confused, or unclear. Does not make a claim.
Gives reasons in support of the claim Gives clear and accurate reasons in support of the claim. e.g., pro: leadership, informed voters, productive workers, learning from history, self-esteem, decision-making and other tht. skills, jobs / $, economy, delinquency / risk;
con: freedom of choice, extra-curricular pursuits self-education, waste of $, bad students ruin it for others, jobs / $, having kids of one's own.
Gives reasons in support of the claim, but overlooks important reasons. Gives one or two weak reasons which don't support the claim well, and / or irrelevant reasons and / or confused reasoning. Does not give reasons in support of the claim.
Considers reasons against the claim Thoroughly discusses reasons against the claim and explains why the claim is valid anyway. Reasons, as listed above, should come from whichever side was not taken as the claim. Discusses reasons against the claim, but leaves out important reasons, andor doesn't explain why the claim still stands. Acknowledges that there are reasons against the claim but doesn't explain them. Does not give reasons against the claim.
Relates the claim to democracy Discusses how issues related to democracy can be used both in support of and against the claim. Discusses how issues related to democracy can be used to support the claim. Says that democracy is relevant but does not clearly explain how or why. Does not mention democracy.
Might mention freedom or choice, but without connecting beyond self to country or world.
Organization Writing is well organized, has a compelling opening, an informative body, and satisfying conclusion. Has appropriate paragraph format. Writing shows organization through a clear beginning, middle and end. Generally uses appropriate paragraph format. Writing is usually organized but sometimes gets off topic. Has several errors in paragraph format, and / or middle is disorganized. Writing is aimless and disorganized.
Conventions Uses correct grammar, mechanics and spelling. Generally uses correct grammar. Some minor errors do not distract or confuse the reader. Shows some control of conventions but frequent errors are distracting or confusing to the reader. Writing shows little control of conventions. Serious and numerous problems distract and confuse the reader.
Words and Sentences Words are striking but natural, varied, and vivid. Sentences are clear, defined, fluent, and diverse. May use sophisticated vocabulary and analogies. Fine but routine word choice. Well-constructed but somewhat flat sentences. Some minor errors. May attempt analogies. Word choice is dull, uninspired, or overly self-conscious. Some words may be used incorrectly. Sentences are redundant, possibly awkward or mechanical. No analogies or strange ones! Minimal variety in vocabulary, and some words may be bewildering or confusing to the reader. Sentences are poorly crafted and difficult to read, e.g. run-ons, fragments, awkward phrasing. No analogies.

 

Scoring Rubric for Autobiographical Journey Essay

  4
The Wow factor
3 2
An attempt
1
The purpose of the journey (Can I say where they went & why?) Tells where you went and why going there was interesting or important to you Tells where you went and why you went there Either where you went or why you went there is unclear Does not tell where you went or why you went there
The scene (Do I have a picture?) Vividly sets the scene, describing important sights, sounds, smells, and/or tastes along the way Describes the central scene(s) in detail, but not vividly Describes the scene at some point but some central scenes are not described well or only unimportant details are given Does not describe the setting of the journey
The cast of characters


        1








        2
Creates complex characters by showing them in action, describing their appearance, personality or behavior,




using dialogue, letting reader "overhear" their inner thoughts.
Creates central characters by describing who they are, what they look like, gestures, expressions,




and using relevant dialogue
Tells who is in the story by giving names, ages, or older/younger references, but does not show how characters behave, feel, or only describes one of several characters

or uses little or only irrelevant dialogue
Only vaguely refers to characters or leaves significant characters out,






does not use dialogue
The action Tells one or two specific exciting, funny, unusual, or sad things that happened during the journey and why they were important Tells in detail one or two specific things that happened but it isn't clear why they were important Tells one or two specific things but without enough detail to let a reader understand what's going on No specific events or actions stand out
Feelings, insights, lessons learned Reveals feelings about and insights gained from the trip. A general lesson learned draws on a thread that runs through the essay. Reveals feelings about and insights gained from the trip, but insights may be tacked on at the end. Describes feelings and ideas, but doesn't have a central insight or reflection, or it isn't well connected to the story Doesn't share the writer's feelings, insights or lessons learned
Organization


        1






        2
Story has an interesting beginning, a developed middle that builds tension, and satisfying end.

Correct paragraph format, at least 5 paragraphs
Story moves through the beginning, middle and end in a logical order.



Generally correct format, at least 5 paragraphs
Organization is rough but workable. Story may get off topic.





Some problems with paragraphs and/or less than 5 paragraphs
The story is aimless or disorganized, lacks direction.





Incorrect paragraph (maybe only 1) and/or less than 5 paragraphs
Conventions Uses first person form, correct grammar, mechanics and spelling. Uses complex sentences, sophisticated vocabulary, etc. Generally uses correct grammar, mechanics and spelling. Frequent errors are distracting but do not interfere with meaning (3 or so errors per paragraph) Numerous problems with fragments, run-ons, grammar, spelling, etc. make the story hard to read

 

Scoring Rubric for the Historical Fiction Essay

Criteria 4 3 2 1
Ideas and Content My paper brings the time and place in which my character lived alive; vividly describes her/his experiences and values; refers to historically accurate events. My paper tells the time and place my character lived; describes a day in her/his life; most or all events are historically accurate. The time & place my character lived is relatively clear, but his/her experiences are more like a list than a letter or diary entry; some events may be historically inaccurate. The setting is murky; the character's experiences are often historically inaccurate; the paper may stray off topic or just ramble.
Organization [Stand back and think with holistic perspective] [Imagine paragraph breaks] My letter/diary has a strong lead, a developed middle, and a satisfying ending, all in an order that makes sense, flows, and hangs together. I have a beginning, middle and end in order; some minor organization problems such as a superfluous or out-of-place sentence. My organization is rough but workable; my writing may drag its feet then race ahead; my ending may stop suddenly or drag on too long. My writing is aimless and disorganized; there is little sense of a beginning or ending; it is probably confusing to a reader.
Voice (a)









Voice (b)
I use only historically accurate language, consistently using terms, phrases and slang from the period.

My writing sounds like a real person wrote it.
My language is mostly historically accurate but without distinction.




My writing voice is engaging but may fade in and out.
I use basic English, avoiding 90's slang but not using language of the period.




My writing is bland or mechanical in many places.
I make no discernable attempt to use historically accurate language.




There are no hints of a real person in my writing; it may sound like I don't like what I have written.
Word choice The words I use are striking but natural, e.g., I use "terrified" instead of "scared," or "gut" instead of "stomach"; I use powerful verbs. My paper has some fine word choices and generally good language; some parts may be routine. My word choice is uninspired, colorless, and dull or sounds like I am trying too hard to impress; some words may be used incorrectly. The same words are repeated over and over and over and over; some words may be bewildering and confusing to a reader.
Sentence Fluency My sentences are clear; they begin in different ways; some are longer than others; no fragments; my paper is a delight to read out loud. My sentences are well constructed; some minor errors in sentence structure; my essay marches along but doesn't dance. My sentences are often awkward or mechanical; little variety in length; may have many sentences that begin with the same word. My paper is tough to read because almost all of my sentences are incomplete, run-ons, and/or awkward.
Conventions I use the correct paragraph form, grammar, capitals, spelling, and punctuation. I made some errors, perhaps by taking risks and using interesting words or sentences. My spelling is correct on common words; several errors in conventions are distracting. Many errors in paragraph form, grammar, caps, spelling, punctuation make my paper hard to read.

 

Scoring Rubric for the Challenge Essay

Criteria 4 3 2 1
Ideas and Content My paper tells of a difficult, discouraging or scary challenge; shows growth or change in the main character and how s/he coped; uses relevant and rich details and examples My paper tells of a challenge but it isn't made to sound very gripping; growth of the main character may be hard to see; uses relevant details and examples My paper tells of a challenge but it may not always be in focus; it may get off topic; it's not clear how the character grows; some details or examples may not matter or don't go together The challenge is murky; it is hard to tell what the topic is; it seems a little like random thoughts on paper
Organization My paper has a lead that establishes the challenge, a developed middle that builds tension, and a satisfying ending that resolves the problem, all in an order that makes sense, flows, and hangs together I have a beginning, middle and end in order; may have minor organizational problems such as an out-of-place sentence or two. My paper takes the reader on a walk but on a sidewalk, not a high wire. My organization is rough but workable; my writing may drag its feet then race ahead; my ending may stop suddenly or drag on too long. [You may find yourself rearranging paragraphs.] My writing is aimless and disorganized; there is little sense of a beginning or ending; it is probably confusing to a reader
Voice My writing sounds like a real person wrote it; it has personality; shows how I think and feel; sounds like it was written to be read My writing voice is engaging but may come and go, fading in and out My writing is bland or mechanical, sounds like I have not found my own way to say things There are no hints of the real me in my writing; it may sound like I don't like what I have written
Word choice The words I use are striking but natural, e.g., I use "terrified" instead of "scared;" I use powerful verbs My paper has some fine word choices [You can count several], but is often routine. My word choice is plain or colorless or sounds like I am trying too hard to impress; some words may be used incorrectly The same words are repeated over and over and over and over; some words may be bewildering and confusing to a reader
Sentence Fluency My sentences are clear; they begin in different ways; some are longer than others; no fragments; my paper is a delight to read out loud My sentences are well constructed; some minor errors in sentence structure; my paper marches along but doesn't dance My sentences are often awkward or mechanical; little variety in length; may have many sentences that begin with the same word My paper is tough to read because of incomplete sentences, run-ons, and awkward phrasings
Conventions I use the correct paragraph form, grammar, capitals, spelling, and punctuation I made some errors, mostly by taking risks and using interesting words or sentences My spelling is correct on common words; several errors in conventions are distracting Many errors in paragraph form, grammar, caps, spelling, punctuation make my paper hard to read

 

Arrow Up

Appendix C

Identification number Experimental Condition Score on Rubric
1 treatment 3.22
2 treatment 3.00
3 treatment 4 (teacher score only)
4 control 1.56
5 control 1.44


Autobiographical Journey Essays Written by Eighth Grade Students in this Study

Essay 1: "Vegas Vacation"

My dad had been planning on taking my brother and I to Las Vegas for weeks. The reason it was so important was because my brother and I have never been out of the state. We were going to leave July 10, a Monday, and get back July 14, a Thursday. We left at 3:00 AM so that we would beat the heat. We would get there somewhere around 3:00 AM. The day before we left we picked up a rental car, so we would have more room to move around.

As we were driving through California we saw many important things. We saw landmarks which my dad said we would see. These landmarks represented the distance to our next destination. The closer we got the more the sun came out. It looked beautiful as we were going through all the mountains in California. In Barstow we drove by a slaughter house and it smelled awful for about 30 minutes, then it went away. I drank bottled water and ate chips, that were some of the things we brought on the trip; so we would have something to eat and drink on the long trip.

My dad, my brother and I were the ones who went to Vegas. My brother was getting impatient about something and he started cusing. Then my dad said "Shut up or I'll turn around and drive straight home." My brother and I were in the backseat telling each other jokes and listening to music. My brother is short, skinny, and has brown hair. My dad is tall, strong, and has brown hair too.

The second day we were there we went to an amusement park called Grand Slam Canyon. The park had an indoor roller coaster that went all throughout the park. I didn't go on it because the line was too long, and because I didn't want to wait in line all day. They also have lots of Midway games, and also a big 50 ft. high water slide. I went on the waterslide 3 times. It goes down pretty fast and I also got wet at the end of the slide. The next day we went to a waterpark called "Wet N' Wild." It was very hot 120 degrees, so the water was very refreshing on our bodies. The park had big pools with water shooting all around. You could just sit around, or frolic in the refreshing water. The best part of the park was a long river like thing that went all the way around the park. The current would take you about 5 miles an hour. You could sit on a big doughnut shaped innertube and float all the way around the park. I went on a lot of different slides. At first I was nervous, but once I started going it was very fun. We took a lot of pictures at these two places, and cut a lot of other places. We drove through Glitter Guleh and saw all of the different casinos and lights. We visited a lot of other casinos and hotels. We went inside the MGM Grand, Excaliber, Treasure Island, and many more.

My feelings about this experience is very positive. I had the most fun I ever had in my life. It wasn't the only time I went, I also went the next summer. I found out how crazy about money people can get when there is a lot of it around. I also realized that I shouldn't take San Diego's weather for granted. I learned that good weather is more important than having a lot of fun. Sometimes fun is more important, but most of the time the weather is much more important.


Essay 2: My Trip to the Mall

It all started on a Sunday afternoon, when my friend Mike and I were at my house watching T.V and we were so bored because all there was to watch was Barney. We decided to go rollerblading at the mall because we thought it could have been exciting. It was fantastic at the mall! We had the greatest time there, we saw all kinds of different clothing and tasted all kinds of exotic foods. We arrived at this shop were they sold all kinds of weird things. The shop was small, blue, green, and with weird writing on the walls, it was really dark inside so it was hard to see. We smelled all sorts of things in the shop like: Black Beauty, Strawberry, and Cherry.

A mysterious lady came out of the back room, she was tall, skinny, with dark hair, and freckles. When she spoke to us, she sounded like if she was chocking on something. I saw that she was limping so I took a good look at her leg it looked like a pirate had chopped it off and glued on a wooden one. Her personality wasn't at all good, I mean she said,"Get the hell out of here." So of course we left the old lady and her weird old shop.

After we had left we went to the movies but we had ran out of money so at the time Mike and I thought the best thing to do was to sneak in; which was unusual for us because we like to play by the rules besides it was very important to us to brake the rules because we wanted to now how it felt like. So eventually we snuck in and eventually we got caught! For some reason I was expecting to happen, when it did it felt like a dark cloud had come over us and God himself was going to pass judgment on us! Good thing that was all in my head because in reality the cops let us of in a warning. So of course right after that we left the mall and our journey had come to an end.

I guess when I got home I felt bad for what I had done. I learned a valuable lesson that day which was, "Never do anything, that you now you're going to get in trouble for and regrete."


Essay 3: Untitled

In my journey I was going to Mexico to visit my family. This trip was very important to me because I learned how to get along with-out my parents. Also, I learned how to do my best because two people were depending on me. In this trip I flew from Tijuana B.C to Mazatlan, Mexico with my sister Adriana. This interesting trip occurred when I was thirteen years old.

The airplane I went on was Aero Mexico. This airplane had rows of three and two seats. In the airplane there were lots of people. We were sitting in the front so that's why I saw lots of people. The airplane at first smelled like perfume but later, it smelled like chicken. The airplane also was very cold at first but, as soon as I heard we were getting near Mazatlan I was very nervous. I heard some people laughing, talking, a man snoring hard. It was annoying, I decided to look out the window. The only thing I saw was a lot of white, blue, puffy clouds. I think the best thing I had seen so far that day was the beautiful and interesting sky. When we got off the airplane we were at the Mazatlan, Mexico Airport. Mazatlan is a very hot place. You feel like if there's not any air.

There in the big airport I saw my tio Juan and my cousin Stephanie "Fanny". My tio was chubby and in my opinion ugly. My cousin was wearing a flowered dress and some little sandals. She was skinny and she was an adorable cute kid. Then, as we were out side I saw this old tall man. He had white hair and was wearing a white sombrero. It was my abuelo. After hugging my abuelo we got on his truck. Whey we got to Tepic, Mayarit, Mexico I saw this old small woman. It was my abuela. She had a dress and grey hair. She told me that she loved me.

An important and unforgtable moment was when my abuelo told my fourteen year old cousin to leave the house because we were flirting with my abuelo's worker. When my tio told him he got very mad. I had never seen him that mad in my life. In that moment I felt very sad because my cousin got in a lot of trouble. My abuelo didn't say anything but I knew he was very mad. This is an important event because I learned and felt something that really hurt.

The lesson I learned was not to talk to boys because my relationship with my abuelos is not the same as with my parents. I also learned that flirting with that guy was not right. Looking back on journey, I tought the moments I spent with my family were sad, funny, and fun. I feel that my trip to Mexico without my parents was a good experience for me to see the world and other things as well.


Essay 4: COUNTRYSIDE—My Journey to up state New York

On October 7, 1996 I left with my aunt, uncle, and three cusins. My mom (Clara) was very sad that was leaving then for at least 4 yrs. My brother (Darrold) sad too. When I was leaving the house my mom and brother cried. My brother cried the most. I too cried a little but only for a while. Before I left my moms rm. She told, "listen to your aunt, don't give her a hard and help her up when she needs help". I told her "yes mom". My dad (Greg) droped me off at the Navy Log where my aunt and uncle were staying at for the night. We left the Navy Log at October 8, 1996. The ride to New York was boring because we hardly had anything to do. I had some drawing papers, crayons, markers, and writing papers. That's all I had to do. The only stops we made were to restraunts, gasoline stations, stores, and hotels. Oh I forgot to say that we traveld to New York by car. One of the worse places I would never want to pass by was the desert. It was so hot even rolling down the window didn't help. The trip to New York took us at least one week. When we got there, we stay with my grand parents place. My grant parents welcomed us and my aunt, an and two other male cusins too. There were more boys than girls I had to live with. Five boys and one girl me. Well that's my journey to New York. Hope you injoyed it too.


Essay 5: My journey to Disneyland

At 8:30 a.m. my mom had woken me up. My mom told me to wake up and take a shower. So I did. At 9:39 I was done. I was putting close on and when I was done my mom told me and everyone else to get a jaket because we are going somewhere. My mom also had told me to stop horse playing with everyone. When everyone got ready it was 9:01. My mom told me to put my stuff in the trunk of the car. So I put all my stuff in the trunk and so did everyone else. When my step dad put the ice chest in the trunk. I grabbed a soda out from it.

Then we all went off together. My sister, my mom, my step dad, brothers, and sister all went off together. My sister and my step brother and I sat in the back messing around. My sister started to play with my step brother my mom and step dad, brother, and sister sat in the front were they all talked to each other.

We stopped at my step dads work where we all got drinks and went to the restroom. We stayed there talking for 30 min. We all got back in the car and took off to Disneyland.

Everyone was falling asleep. I was awake talking to my mom that where we were going. I just stopped asking my mom because I was falling asleep.

I woke up at Disneyland parking lote. We all got out and got something to drink then we left to go on rides.


Arrow Up

Appendix D

Interpretation of the final regression model for the persuasive essay (Essay 1)

Controlling for grades, standardized test scores, teacher, and school, there is no statistically significant effect of treatment on essay 1 scores. This model accounts for 24.49% of the variation in essay 1 scores.

No interactions were statistically significant, suggesting that the effects of treatment did not differ for School A and School B, nor for females and males.

ESL, special education, gender, and ethnicity were dropped from the final model because they were not statistically significant through the model building process nor were any interactions between them and the question predictor or other control variables. This model is the most parsimonious model that accounts for significant covariates.

Power calculation
For a multiple linear regression model which already includes 4 covariates (control variables) with a squared multiple correlation R2 of 0.245 (the R2 obtained from a model including the four control variables only), a sample size of 106 will have 31% power to detect at alpha= 0.100 an increase in R2 of 0.010 due to including 1 additional variable.


Interpretation of the final regression model for the autobiographical journey essay (Essay 2)

Controlling for grades, standardized test scores, gender, and the interaction between grades and gender, there is a statistically significant effect of treatment on essay 2 scores. Controlling for the aforementioned variables, treatment students are predicted to score, on average, .49 points higher on essay 2 than control students. Since there were no interactions present between treatment and the control variables, this means that the effects of the treatment did not differ by gender, test scores, or grades. However, there was an interaction between grades and gender, suggesting that the effect of grades on essay 2 scores differed by gender. In this case, the main effect indicates that for males, there is no effect of grades on essay 2 scores, controlling for experimental condition and standardized test scores. However, for females, there is a positive effect of grades on essay 2 scores, controlling for experimental condition and standardized tests (betagrade = 0.019537, t-statistic = 1.985, p < .0560). This model accounts for 39.92% of the variation in essay 2 scores.

Teacher, school, ESL, special education, and ethnicity were dropped from the final model because they were not consistently statistically significant through the model building process, nor were any interactions between them and the question predictor or other control variables. This final model is the most parsimonious model that accounts for significant covariates.

Power calculation

For a multiple linear regression model which already includes 4 covariates with a squared multiple correlation R2 of 0.176, a sample size of 37 will have 95% power to detect at alpha= 0.100 an increase in R2 of 0.223 due to including 1 additional variable.


Interpretation of the final regression model for the historical fiction and personal challenge essay (Essay 3)

The statistically significant interaction between treatment and gender means that the effect of treatment differs for females and males, controlling for grades, standardized test scores, school, and ethnicity. Since the main effect of treatment is not statistically significant, this means that for males (males are coded as gender = 0) there are no statistically significant differences in essay 3 scores between males who were in the treatment group and those in the control group (betatreatment = .119110, t-statistic = .72, p = .4726), controlling for the aforementioned variables. For females, there was a statistically significant difference in predicted essay 3 scores between females in the treatment and control groups (betafemale= -0.308641, t-statistic=-1.736, p<.0845). The negative parameter estimate indicates that, on average, females in the control group are predicted to have essay 3 scores that are .31 points higher than females in the treatment group, controlling for grades, standardized test scores, school and ethnicity. Moreover, the main effect of gender in the above model is statistically significant (betagender = -.507182, t-statistic = 2.218, p < .0280) which indicates that females in the control group are predicted to have scores on essay 3 that are .51 points higher than males in the control group, on average, controlling for grades, standardized test scores, and ethnicity. However, there was no statistically significant difference on essay 3 scores between males and females in the treatment group (betatreatment = .079431, t-statistic = .784, p < .434), controlling for grades, standardized test scores, and ethnicity. This model accounts for 19.09% of the variation in essay 3 scores.

There was no interaction between school and treatment suggesting that the effects of treatment did not differ for School A and School B.

Teacher, ESL, and special education were dropped from the final model because they were not consistently statistically significant through the model building process nor were any interactions between them and the question predictor or other control variables. This final model is the most parsimonious model that accounts for significant covariates.

Power calculation

For a multiple linear regression model which already includes 6 covariates with a squared multiple correlation R2 of 0.188, a sample size of 160 will have 39% power to detect at alpha= 0.100 an increase in R2 of 0.010 due to including 1 additional variable.


Arrow Up