Writing Assessment

Long over due wrap up and notes

2006-11-03T11:49:00.000-08:00

The group working on this blog in Spring 06 finished up their work within a Blackboard discussion forum.

A new sets of postings to the blog will begin in early December 2006.

See you then.
Carl

Copyright discussion

2006-03-19T15:29:00.000-08:00

I'm not sure if this fits in with the multi-media discussion, but it touches on some of the topics we talked about in our copyright discussion. Specifically, where does the line end between your own work and that which came before? How much can you borrow from other authors and creators?
This is just a link to a Da Vinci Code copyright case that was posted Sunday. Apparently the case (which inlvolves other authors who had espoused many of the same ideas as Dan Brown) should wrap up Monday.
Whatever the outcome, the article makes clear it will have momentous impact on the copyright discussion.
http://news.yahoo.com/s/nm/20060319/en_nm/arts_davinci_court_dc

Multimedia: No Child Left Behind, Camcorders, and Original Poetry

2006-03-15T17:56:00.000-08:00

An article in the online Milwaukee Journal Sentinel describes efforts by four school districts to engage middle school students who were "not inclined to create" by encouraging them to use technology. Using a grant from No Child Left Behind to purchase laptop computers and video-production software, the districts designed a project with goals, "...to use modern technology to research, write and design multimedia projects, boost the students' motivation and self-esteem in the classroom, and have the teachers involved integrate the technology into everyday teaching methods, according to Kevin Messman, the district's coordinator of instructional technology." For example, one district's students illuminated their own original poetry. The results were very positive. Students knew more about the technology than the teachers, which reversed the role of "teacher" in the classroom. Students who did not necessarily "like" writing before enjoyed the process and empowerment: they realized that composition is an emotional process. Their confidence increased in English and other subjects as well."Kirsten Yoder, the Eisenhower teacher assigned to the project, said she's seen her students react positively to the project because it doesn't penalize them for past problems, like being behind peers in a particular field. 'It's more of an equalizer, because the kids are all starting from different backgrounds. It's not so much about how did they do on their last test.'"

A Portfolio Assessment Model for ESL

2006-03-14T09:13:00.000-08:00

http://www.ncela.gwu.edu/pubs/jeilms/vol13/portfo13.htm

I discovered an interesting article which gives practical advice on developing a portfolio assessment model for ESl students in The Journal of Educational Issues of Language Minority Students. To sum up, the proposed model breaks down the development of a portfolio into six main levels as well as subheads.

Identify purpose and focus of portfolio
1. Establish a portfolio committee
2. Focus the portfolio

Plan portfolio contents
3. Select assessment procedures
4. Specify portfolio contents
5. Determine frequency of assessment

Design portfolio analysis
6. Set standards and criteria
7. Determine procedure to integrate information
8. Schedule staff responsibilities for analysis

Prepare for instruction
9. Plan instructional use
10. Plan feedback to students and parents

Plan verification of procedures
11. Establish a system to check reliability
12. Establish a system to validate decisions

Implement the model

While I found the real-time model useful, the authors also offered another reason portfolios may work well for ESL students. "Linguistic, cultural and educational diversity in the ESl classroom are easily addressed in assessment because portfolios can be individualized." While they don't spend anymore time on the topic, it really got me thinking.
What I love about portfolios is that they can be so easily tailored to a student's interest, thereby reaching and hopefully interesting everyone. For example, using the gender issues we've been talking about and trying to get boys more involved in reading and writing--say you assign a story about baseball. You think that the boys are going to all love it, yet if they didn't grow up in a culture that values baseball, they could probably care less about the topic whether they're male or female.
Yet, if you allow students to pick topics meaningful to them, you not only show them their new school values their heritage or culture, you can get them more involved. Additionally, during the peer review section students of other nationalities can hopefully learn something about another culture or value system.

The Premature Obituary of the Book

2006-03-06T12:23:00.000-08:00

http://literature.sdsu.edu/WhyLiterature/
Hi everyone-
This isn't really on my topic but it touches on a lot of stuff we've been discussing: why males don't read as much as females and the replacement of books with computer screens). I'll post on my other topic later this week, but thought this was worth a quick mention.
Published in The New Republic in May of 2001, the author is Mario Vargas Llosa, a Georgetown professor and author.
While the article is a defense of the study of literature, he concludes that most men don't read enough because they are "too busy" dealing with the realities of life. I'll let the author explain in his own words. This excerpt is from his opening paragraph.

It has often happened to me, at book fairs or in bookstores, that a gentleman approaches me and asks me for a signature. "It is for my wife, my young daughter, or my mother," he explains. "She is a great reader and loves literature." Immediately I ask: "And what about you? Don't you like to read?" The answer is almost always the same: "Of course I like to read, but I am a very busy person." I have heard this explanation dozens of times: this man and many thousands of men like him have so many important things to do, so many obligations, so many responsibilities in life, that they cannot waste their precious time buried in a novel, a book of poetry, or a literary essay for hours and hours. According to this widespread conception, literature is a dispensable activity, no doubt lofty and useful for cultivating sensitivity and good manners, but essentially an entertainment, an adornment that only people with time for recreation can afford. It is something to fit in between sports, the movies, a game of bridge or chess; and it can be sacrificed without scruple when one "prioritizes" the tasks and the duties that are indispensable in the struggle of life.
It seems clear that literature has become more and more a female activity. In bookstores, at conferences or public readings by writers, and even in university departments dedicated to the humanities, the women clearly outnumber the men. The explanation traditionally given is that middle-class women read more because they work fewer hours than men, and so many of them feel that they can justify more easily than men the time that they devote to fantasy and illusion. I am somewhat allergic to explanations that divide men and women into frozen categories and attribute to each sex its characteristic virtues and shortcomings; but there is no doubt that there are fewer and fewer readers of literature, and that among the saving remnant of readers women predominate.

This article was great, if for nothing else, as a reminder that reading is important and the study of literature is crucial to the human race.

Males vs Females: Creating Assignments that Boys Enjoy

2006-03-02T09:20:00.000-08:00

“If we leave [the] processes of reading and writing cloaked in mystery, telling ourselves that it all either comes naturally or else it doesn’t, we surrender to voodoo pedagogy. In voodoo, privileged people, objects and rituals are invested with secret magical power, and to some of our students it certainly seems that there must be mysterious, unnamed powers needed to do well in English.”(Pirie, 2002, p. 52)
As a female English teacher, I am discovering more and more the need to identify with my male students so that I don't leave them victim to voodoo pedagogy. One of the main errors, we make as female teachers is that we fail to acknowledge the differences in boy's and girl's learning styles, and therefore, fail to teach the process of discovery within the text. This makes boys feel stupid and eventually turns them off to reading and writing. Still, not only do we fail to teach the process of textual analysis, but as I had stated in an earlier post, we fail to create themes that intrigue boy learners.

I found two pretty interesting ideas for teaching reading and writing to boys. In the first, the teacher focuses on sports and in the second, the teacher focuses on masculity.
Working with a local sports organizationIn the United Kingdom, the Arsenal football club (or soccer club, as we would call it in North America) has set up an outreach literacy program for schools, using specially designed literacy materials. All of the reading and writing in the program is related to football and its star players. The program is delivered by football-loving teenagers. Among other benefits, the program draws on the uncanny ability of some children to absorb sports 'facts'. Being able to apply that knowledge in a learning situation gives them a surge of confidence.(Klein, 2002)
In one all-boys class run by a male teacher in a co-educational school, discussion focused on an up-front-and-personal investigation of masculinity: what meanings are associated with being a boy at school and a man in the wider community?The teacher harnessed boys' personal interests and experiences as a starting point for literacy activities. He brought to their attention for debate and discussion the privileges and limitations associated with living life as a male, and the problematics of gender and power relations that circulated among them. As well, boys were invited to discuss how particular versions of masculinity were produced and disseminated in popular media.(Alloway and Gilbert, 1997, p. 138)

Finally, based on the discussion we had last class about incorporating multi-modal/media writing assignments to attract boys, I found evidence acknowledging that indeed incorporating more kinesthetic learning activities will engage boys. They enjoy any kind of manipulatives. While we may not have ready access to computer-related activities, we can begin to incorporate more drama and small group research projects into our classroom activities.

Ultimately we have got to be steadfast in seeking an answer because the gender gap in literacy is a real problem. According to one source, “Boys’ underachievement is a major concern. Nationally, boys fall behind girls in early literacy skills and this gap in attainment widens with age. The challenge of raising achievement directly addresses the learning needs of our students and the professional growth of our teachers, and enhances the role of the school as an agent of social change. We want to give boys and girls the best opportunity to become powerful learners.”(UK Department for Education and Skills, n.d.)

The Reflective Portfolio: Two Case Studies from the United Arab Emirates

2006-02-27T14:27:00.000-08:00

http://exchanges.state.gov/forum/vols/vol42/no1/p18.pdf

I wanted to continue the theme of using portfolios for ESL students, because I think the skills they teach these students also pertain to students who may use English as a primary language, but may not be proficient. In many urban, poor areas students are not always coming away from school with a full command of the English language; portfolios seem to be an excellent resource for these students to learn. Speaking as a literacy tutor, my current student (who is a 46-year old male) could not sit down and bang out a timed composition no matter how hard he tried. Other students I've tutored have told me when they were called on in class, they would act up so they were sent to the principal's office rather than look foolish in front of their classmates.
I believe portfolios can relieve some of these types of students' anxieties (and I also believe there are a lot more students like these than many people think). Giving them time to go over their work is an excellent way for them to feel more comfortable and thereby learn more.
Anyway, I found an interesting article on a pilot portfolio experiment at Dubai Men's College and the United Arab Emirates University during the Spring 2000 semester.
What I found most helpful with this article is it gave concrete examples of portfolio inclusions. For example, students created five entries: a letter of introduction to their teacher; using the past tense to write about a prior important experience; a formal complaint letter linked to their content area; a student generated topic (in this case, the problems or challenges facing the UAE and possible solutions); and finally, a reflective letter, detailing their feelings about the experiment, areas they felt where they improved and areas that still needed improvement.
The only quoted student wrote that he is more comfortable in writing now, and his problem areas of grammar and spelling have improved. While he agrees portfolios are the best way to improve writing, he did find them boring.
What struck me as most beneficial about the reflective letter is the idea of forcing the student to think in a new way. Most students will likely never have thought much about their work, it is usually just produced automatically. Forcing a student to think critically about themselves and their work is a good exercise in critical analysis that translates to the everyday world (as anyone who has been on a job interview can attest).

"Boys far behind in WASL" [focus: stereotyping]

2006-02-27T08:55:00.000-08:00

http://www.heraldnet.com/stories/05/12/04/100loc_wasl2.cfm

Something Andrea said during our last class session piqued my interest. She talked about boys' interests and the fact that video games are probably more appealing to them vs. reading and/or writing. This lead to Natasha's comment about technology, how working on the computer is usually more appealing to boys. I agree with both observations, however I wonder how large a roll stereotyping plays in this scene. Are boys supposed to like video games more than girls? Are boys supposed to be better on the computer than girls? Did boys have to stake claim on playing video games and working on the computer as boy activities?

Have any of you read or see the public broadcast of "Raising Cain"? http://www.pbs.org/opb/raisingcain/ It provides a wonderful peek into the lives of boys in today's society. Like girls, boys seek acceptance by their peers, however it is crucial for boys to find a niche. Sometimes this comes naturally; sometimes boys force themselves into a group, just to say they have a group. As far as most boys are concerned reading/writing is not an option as a group to join. “That’s something girls do.”

The media, books, teachers, parents, and kids have created a stereotype for how boys should act – defined what groups they should join. How can this stereotype be broken down, so that boys feel safe to explore literature, prose, poetry and writing without feeling like they’re breaking an unspoken boundary?

The article "Boys far behind in WASL (Washington Assessment of Student Learning)" by Eric Stevick and Scott North, hits on, among other things, the issue of stereotyping. In the article, First Lady Laura Bush, “linked boys' classroom woes with societal ills, including gang violence. …Bush in May told The Miami Herald that "in a lot of ways, we have neglected boys over the last several generations, that we bought into the stereotype of boys that all of us know intuitively is not right, that boys don't need nurturing, that they can take care of themselves, that they don't need special mentoring from a father or really from anyone."

Recently Natasha responded to a blogger post: “it is finding valuable works by African-American writers who have come out of the oral tradition, and teaching those texts in a mode that is familiar and appealing to our African-American male students. I believe strongly, that if we could and would do this with an understanding, we could reach more of our African-American boys and men. They are not idiot; we just haven't learned how to reach them.

What Natasha posted is exactly what’s suggested in this article, at least as a place to start. “Writing can be personal and boys are often reluctant to open up, said Peter Hendrickson, an assessment specialist with the Everett district. One option is to connect reading and writing to an activity boys enjoy - especially sports and other physical pursuits. As a soccer coach, Hendrickson used to bring books onto the practice field to share with his players. "The boys need that OK, that permission, from an important male" to develop reading and writing skills, he said.”

Assessing Multimedia: Multimedia on Assessment

2006-02-25T16:19:00.000-08:00

Today I stumbled on Dr. WHithaus's hypertext essay on "Evaluating Student-Created Hypertext" published in Kairos. You can find the article here.

So, he offers therein 4 suggestions, or "thumbnails" as he calls them, for assessing Hypertexts.

distribute the process of assessment among a variety of readers;

situate assessment within the dynamics and goals of an institution or a program;

rely on descriptions of material in student writing and their own perceptions about how their reading experiences could or should have been different - and better - and;

interact with the community of practice created by the students and teachers within a given course by anticipating and questioning the already internalized traditions of assessment students display from prior English classes.

Based on the readings in Lynne that we have been doing, these four suggestions certainly seem to fall within that social-constructionism contextual paradigm like that advocated by Lynne.

So, as a reader, the question occurs to me: How would I assess this Hypertext?
personally, I did not enjoy my reading of this work. As a reader I resented having to hunt for the links scattered amongst and within the visual images. Mostly I ended up reading the essay from left to right and top to bottom as much as possible. The discrete pages and only vaguely implied links between them drove me crazy. As a writing tutor, these kinds of a to c jumps are exactly what I council people to avoid in their papers.

What, I wondered, was he trying to get at here? Knowing him as an instructor and likely credible source, I wondered what was the purpose behind the design of the links. Did the essay not work for me because it wasn't written for me? Lastly, I wondered if part of the problem was my inexperience with reading Hypertext essays.

How can we assess multimedia composition without a context? Especially if we, as the instructor, are not the primary audience for the work. How do art instructors/critics assess art?

Assessing Multimedia: What is an Author?

2006-02-23T13:32:00.000-08:00

Without sounding too dramatic (hopefully), I believe it is possible that the internet/postmodern revolution and the resulting changes in how texts are produced may change the concept of authorship in ways not seen since the printing press/protestant reformation revolution. Technologically produced texts are the product of not just the person turning it in but of a host of contributors (most of them unwittingly contributing). When my brother-in-law was in high school, he did a powerpoint presentation on the 1970s. It was well organized and aesthetically pleasing. TV theme songs played in the background as quotes from the shows highlighted the relationship of the shows to the social issues of the day (like M*A*S*H and "All in the Family").

He did a lot of work on the project and learned a lot, but I think this project illustrates problems of authorship. He did not write the music. The majority of the text was quotes. And even the nice, pretty formatting was simply a matter or cycling through preset layouts designed by Microsoft. Yet, I would say he was the author. Why? Because he played the elements together in a way not done before. However, If his presentation had gone out over the airwaves, lawyers from a host of firms might say something different about who the author was.

Foucalt wrote an essay "What is an Author?" (an excerpt: http://foucault.info/documents/foucault.authorFunction.en.html) in which he argues that the author is more of a liability because it constrains conversations about a text. Knowledge that Shakespeare is thought of a great playwright will color our conversation about "Hamlet." Furthermore, general conceptions about Shakespeare will tend to make us ignore elements of his writing that go against what we already believe. The question of authorship keeps the text "safe" because it constructs texts as the reflection of an individual and not as a simultaneous influence and result of a society's ideology.

In his essay, Foucalt quotes Samuel Beckett as saying "What does it matter who is speaking?" To the general public, it might not. To those of us who are to evaluate what is said, it does matter what part of the student's work is hers (and therefore directly evaluatable) and which part is a "source" (and therefore evaluatable only in terms of whether it was an appropriate selection).

Foucalt doesn't specifically address the question of multiple contributors except in his understanding that an author of a work is an abstract figure that represents creation more than it is a specific person. But teachers grade students, not abstractions. So do we grade based on whether or not the piece "works" (as my brother-in-law's teacher did), ignoring issues of what belongs to whom? Do we try to see the student's piece as both a former and a formation of society? Is that practical for grading? If a text is seen as a result of the society its located in, can we judge it without judging the society? Not exactly practical in terms of helping the student improve as a writer (ack! what is the difference between author and writer in a world where things can be revised and edited AFTER they've been published -- as in a blog?).

If we lose the author in a sea of contributors, can we still evaluate the result in a way that actually helps the student improve? Can we pretend the student-author is the only contributor? Do we judge merely her selection of texts, and thus view any text she actually wrote herself as just one among many choices for her selection? If so, are we grading her writing skills or her compiling/editing skills? Is there a difference?

I think the answer has to return to the issue of usefulness. We must understand that the creator of multimedia texts enters a complicated world and is many things at once. Effective evaluation will probably have to ignore some issues in order to effectively deal with others. The importance of the author might have to be altered from assignment to assignment. That's a messy proposition, but its something we'll probably have to juggle -- at least until the revolution is well over.

Helpful Website for Multimedia

2006-02-23T08:01:00.000-08:00

Georgetown University's American Studies department has created a helpful website for exploring various multimedia curricula, tutorials, exhibitions, and collections. There are many innovative ideas for educators and users of multimedia in general. Of particular interest is a Norton anthology in multimedia, designed by Charles Hannon, an English professor at University of Alabama. His goal was to design an English course that would literally place the texts into visual, historical and cultural context. He ended up working with Norton to design a multimedia assignment bank for their Anthology of African American Literature. There are 20 graphics-based assignments that use photos, lithographs, political cartoons, etc. to go along with a text. Students can share their responses electronically with the professor as well as other students. It is a fascinating idea. His plan is generally laid out and seems to be successful. Noticeably absent from this document, however, is any sort of rubric or grading method. While the collaboration/exploration is of enormous educational value, there remains the ever-present hurdle of assigning individual grades.

Assessment Portfolios Systems for English Language Learners (ELL)

2006-02-23T05:58:00.000-08:00

As I reread my post last week about the Japenese language teachers use of portfolios in teaching English to their students, I realized this is an area I wanted to research again. Upon the beginning of this assignment I hadn't even thought about ELL or ESL as target students for portfolios, but it seems to me they may be the biggest beneficiaries of this system. As anyone who has learned a second language will tell you, it's a very time-consuming process that requires you to actually retrain your thoughts and mind processes into another system. The reflection and revision process (as well as peer reviews, perhaps including native speakers) seem to be ideally situated to the unique position on ELL or ESL students.
On that note, I discovered an article that deals primarily how to implement a portfolio system for ELL in a school district, but it offered other insightful information as well. Unfortunately, it is an older article (12/2000) so hopefully it isn't too out of date. It is on the Center for Applied Linguistics Web site at
http://www.cal.org/resources/Digest/0010assessment.html

Arguably, the primary benefit of implementing a portfolio assessment for ELL students is the inclusion of the students at all. According to the authors of the article, "Many states have policies that restrict English language learners from taking commercially developed, norm-referenced and criterion-referenced tests for up to three years after their arrival in a U.S. School system. In contrast, all students can be included in an assessment portfolio system."

Additionally, the school district benefits because portfolios (hopefully) help to establish greater school accountability for all students. "By providing sytems with a richer source of information about school learning, assessment portfolios can help school systems identify and meet the needs of diverse students, including ELLs."

All right, now come the negatives. Besides the standard worries about cost and reliability, the authors of this article include teacher performance as a possible inclusion in this catagory. The authors quote other researchers and write, "When using portfolio assessments, performance conditions may vary, and teacher bias can affect students' performance. For example, the maount of support teachers provide to students, the amount of time students are allowed to spend on portfolio samples, and the extent to which student work is augmented by support from external sources have raised questions about the validity of inferences about student competence based on portfolio work."

I hadn't really thought out the possible teacher bias angle because I assumed a teacher who would advocate using portfolios (which include extra work for themselves) would naturally be very supportive of students efforts and allow ample time for portfolios, but perhaps this isn't always the case.

Portfolios: A Requirement for High School Graduation

2006-02-22T21:11:00.000-08:00

I found the article, +News+>+Education+--+Students+object+to+requiring+portfolio&expire=&urlID=16030630&fb=Y&url=http://www.signonsandiego.com/news/education/20051026-9999-6m26swgrad.html&partnerID=621">“Students Object to Requiring Portfolios” written by Chris Moran which appeared in The San Diego Union-Tribune on October 26, 2005 to be very interesting. The article explains and presents the opinions about a type of portfolio requirement in California high schools that I had never heard of. It is known as the Senior Portfolio, and the main components are as follows:

- The portfolio is a requirement for graduating high school seniors.
- Portfolios include a reflective essay and a compilation of class room work.
- High school seniors are also given an interview to “make their case for graduation”.
- The Senior Portfolio (and the interview) is required to receive diploma

At the time the article was written, this portfolio requirement had been in use for two years, and it was the district administrators’ of South County answer to standardized tests. The administrators felt that the portfolio requirement could help give them an “in-depth gauge of students’ abilities”. However, students and some parents did not like the requirement, and the students deemed it a race and not a reflection of how much they had learned. In fact, students and parents protested at a school board meeting for the “abolition” of the Senior Portfolio.
I thought the article and the Senior Portfolio were intriguing because I did know that high schools were using portfolios as a graduation determiner like some colleges as George Mason. Furthermore, I thought it was effective that the portfolio requirement spanned across curriculums. Yet, the interview that was tacked on to the portfolio requirement was rather interesting and slightly odd. The fact that students had to, in a way, make a plea to graduate did not set well with me. (Maybe I need more information on that aspect.) I do wonder had these students been taught about formative and writing assessment before being asked to complete this high stakes portfolio requirement. In addition, in the article, a school board trustee was quoted as saying that he questioned the validity of the portfolio requirement as an educational leader.

"Girls and Boys as Readers and Writers" (Barrs, Huot & Lynne)

2006-02-22T18:41:00.000-08:00

For volume thirty-three of Orbit (OISE/UT's Magazine for Schools) Myra Barrs contributed an article called "Girls and Boys as Readers and Writers". http://www.oise.utoronto.ca/orbit/struggling_readers_sample.html This article supports much of what we've discussed thus far about gender differences and reading/writing; it also brings to light a few new thoughts. Barrs considers the fact that because girls are usually more proficient readers, their experience with words allows them to feel more free to write, and that boys and girls gain much of their gender identity from what they read. This little tidbit I thought could be useful in choosing texts for our classrooms. If we could find poems, short stories etc that show boys as writers or boys who are comfortable with expressing themselves, or even boys who struggle with being a student, writing or self expression, perhaps the boys in our classrooms could identify with this.

"To understand the world and make a difference to it we need to attend to reality, not to statistical abstractions, to real individual cases and their peculiarities, not to national averages. This is where teachers have to start" (Barrs). I've read Huot and Lynne, and I hear what they are saying about theory and such, but I'm just not getting the point. To me, teaching is a task that requires individualization. How on earth could one theory encompass the whole lot? Lynne wrote on page 83, "...he (Huot) discusses: writing assessment should be "site-based," "locally controlled," ..." - if so, how can he suggest one or two theories could define THE best way to teaching or THE best way to write (or teach writing)? On page 75 Lynne wrote, "We want to be fair to students from diverse backgrounds that are still sensitive to our course and program objectives." What does she mean, "still sensitive"? Her expectation is that students will just get over being who they are and eventually transform into 'the norm'? Not only are boys and girls different, each of our students are different in their needs, goals and ablities. We should strive to meet each need not strive to make them fit a mold.

Patricia Lynn's Point of View of Portfolios

2006-02-22T09:08:00.000-08:00

In Patricia Lynn's thoughts on portfolios from her text "Coming to Terms; A theory of Wriring Assessment" she states that "for many years portfolios have been used for fine arts." However, in "writing instruction portfolios became popular during the early 1990's. (p37) She states that "ETS developed a proposed Portfolio Assessment Plan in response to educators' demands that testing reflect pedagogical theory" in the early 1980's. The plan for this was to have each student submit a variety of essays, a student selected piece, and an introductory letter addressed to real people such as admissions official or prospective employers. However, due to the "it was abandoned by ETS on the basis of its faliure to meet the angency's required standards of time-and cost effeciency, scoring reliability, and the appearance of scientific objectivity."

Again there seems to be the issue of cost and time that keeps popping up when it comes to assessing. From the research that I have read, students seem very fulfilled with their portfolios. Also, with technology being part of all of our lives, this is where students can learn technology and writing at the same time. I think when students use peer evaluation, they also learn from each other because they might critic or praise the others work and in turn learn something new.

I think the research is lacking when it comes to portfolios in the United States; however, from some of the articles I have read (one from Japan and one from Finland) there seems to be a real interest in this type of learning and assessing. However, because it is a new way of teaching and many people are resistant to change, in my opinion, I think it will take some time before portfolios and electronic portfolio become part of the curriculum.

Portfolios: Graduation Portfolios at GMU

2006-02-21T21:34:00.000-08:00

In class a few weeks ago, I mentioned that I graduated from a program with a graduation portfolio requirement. I thought some of you might be interested in more information about the assignment.

Here is a brief explanation of the graduation portfolio.
Here is a detailed explanation of the portfolio guidelines.

The core content of the portfolio is the following:
I. Title page and table of contents
II. List of courses taken
III. Essay: You as a learner (this is the main reflective essay)
IV. Discuss 4 NCC competencies*
V. Creative dimension: Thinking outside the box
VI. Experiential learning
VII. Bibliography
VIII. Future directions
IX. Postscript

The second link explains each of these requirements in detail.

I cannot begin to explain how much work actually goes into creating one of these portfolios. The collection process takes place throughout the undergraduate experience and draws from all courses. In terms of selection and reflection, it actually begins with multiple portfolios created for individual courses. Students then reflect back through all their portfolios and other coursework for the final portfolio. Seniors have to register for 1 credit devoted solely to the portfolio during their last semester.

One of the most interesting components of this portfolio was the postscript, which was a reflection on the portfolio creation process. Clearly, there are many layers of meta-cognition embedded in this assignment, and I found it overwhelmingly beneficial in my growth as a learner and as a concluding event in my undergraduate education.

*NCC- New Century College; a department within the College of Arts and Sciences at George Mason University.

Males and Females: Encouraging Our Students

2006-02-21T14:27:00.000-08:00

Informal Science Writing Produced by Boys and Girls: Writing Preference and Quality
By Tamar Levine; Zehava Geldman-Caspar
Found on J-Stor
Published in British Educational Research Journal Vol 22, No. 4 (Sep., 1996) 421-439

This informal study about writing among the sexes takes an in depth look into what males and females find interesting. 374 seventh graders were sampled with 172 being males and the rest being female. Students were given five topics to write about and they were asked to choose one. After all of the samples were submitted, 30 samples were selected randomly and examined using very specific criteria. Two raters were used to evaluate the writings and the inter rater reliability was pretty good. There was only a .4 average difference between the two.

Here are some interesting things that the study found:
Task 2 (dealing with science related inventions was popular among boys and girls)
Task 1 (an expressive type of writing about a science related personal experience) was significantly more popular with the females.
Girls tended to write longer pieces than males.
More girls related a personal experience to their writing

From my assessment of the article’s findings, it shows that writing patterns tend to be the same regardless of the writing topic. Males tend to have a more masculine approach to writing and females have a more feminine approach. No matter what the subject matter, a pupil’s personality will come through in their writing. It doe not necessarily mean that one is better than the other and if teacher’s are looking for something specific, they should inform the students and help them achieve that goal. When a student is constantly expected to do badly, they probably will.

Assessing Multimedia:Arguing for the teaching of Multimedia

2006-02-21T07:53:00.000-08:00

Thus far, the search for articles devoted to the assessment of Multimedia has come up effectively empty, for me at least. What I have found are articles making the case for the inclusion of multimedia into classroom pedagogy.

I recently finished reading Charles A. Hill's "reading the Visual in College Writing classes" (Only published in Visual Rhetoric in a Digital World and intertwine: Reading Pedagogy in College Writing Classrooms) and the online article "Critical Visual Literacy: multimedia Communication Across the Curriculum" by Barb Blakely Duffelmeyer and Anthony alerts found here.

What these two articles have in common is the fact that they spend the majority of their time making arguing for the inclusion of the visual in classrooms to begin with. Hill's article, for example spends almost half of the article arguing for the need to teach students about visual images and that they can be used to persuade audiences in compositions. The online article by Duffelmeyer and alerts likewise spends a great deal of time establishing the worthiness of images and multimedia to be included and taught to students. Only at the very end does it present a case study of teaching multimedia composition to students, but it ignores issues of assessment of these same projects in general and focuses on one student in particular. This student's flash composition is "assessed" only in a descriptive fashion of what were the rhetorical goals of the composition and how did the student use the multimedia technology to address them.

It may be that the only way to effectively assess multimedia compositions is in this descriptive fashion of what the composition set out to do and did it accomplish these goals. We may also lack specific theories on the assessment of multimedia compositions until they become a widely accepted part of curriculum of at least undergraduate composition if not in secondary schools.

A check-up on the Kentucky-mandated Porfolios

2006-02-16T07:56:00.000-08:00

In November 2000, researchers Qaisar Sultana and Lisa Key of Eastern Kentucky University presented a paper to the Midsouth Education Research Association. The title was, "Pre-and Post KERA Students' Writing Skills - A Comparative Study." In short, a writing prompt from 1989 was repeated in 1997 to allow for a comparison of student writing. 50 essays from each year were randomly selected and read by 2 graders. The results: "no significant difference was found at any level between the means of the two groups" (3). The researchers interpreted this to mean that while the post-KERA students had produced a lot of writing for their portfolios, and with "...access to the spell check, grammar check, thesauraus, etc. {...} in the classroom or home environment ... [and with] unlimited help from their peers, parents and teachers" (7), they have become dependent on technology and "have not really learned to write (8).

This is helpful to our discussion about portfolios because it seems that we are looking at two different things: students' attitude toward writing vs. learning to write well (or should I say, in a timely, organized fashion). The portfolio system may not be "valid" (ew! that word!) for the persisitent objectivist testing environment.

Portfolios: It’s a Journey for Teachers and Students

2006-02-15T21:14:00.000-08:00

After more searching, I was finally able to find an article which gave some good insight on what teachers and students thought about portfolios as well as about middle school usage. In the 2001 article “Portfolios That Make a Difference, A Four-Year Journey” by former teacher, Judith Ruhana, an inside look is given at Ruhana’s journey with portfolios usage over a four years, the first year being her first encounter with portfolios. During the time of this article, Ruhana was a language arts teacher at Nicholas Middle School in Evanston, IL, a middle school classified as urban and multicultural with many disadvantaged students. Ruhana brought up some valid fears and concerns when she learned that she would be using portfolios. Her fears included worries about students being bilingual and minority difference in learning.

Two key concepts were:

Portfolios opened communication channels between Ruhana and her students. Students were given a voice in the classroom, and Ruhana was able to have a better understanding of her student’s writing.

Communication and relationships across curriculums were fostered. Students learned how to write effectively in all subject areas not just English.

Ruhana also presented her process of writing assessment with the use of portfolios. Her adaptation was taken from the Kentucky Department of Education, but she added some key points of her own through the four-year journey which included:

Establish criteria for reflection with the students at the beginning of class.

Teach students how to reflect. Ruhana actually has her students do two reflective letters- one at the beginning of the year which includes goals and current and writing ability and one at the end of the year.

Ruhana included “mini” assessments every six to eight weeks throughout the year. After mini assessments, one week was given for revisions.

Peer reviews were also done in classes, and students used an assessment rubric. After the student’s evaluation with the rubric, Ruhana then used the same rubric to evaluate.

The article presented a lot of valuable information through the sharing of Ruhana’s journey that I think can benefit teachers and future teachers of all grade levels. Also, please read the article for some real examples of students’ writing and reflection.

Assessing Multimedia: Metaphor and Metonymy

2006-02-15T20:51:00.000-08:00

In the current issue of Kairos , a “refereed online journal exploring the intersections of rhetoric, technology, and pedagogy,” there is a hypertextual essay by Madeleine Sorapure addressing the issue of assessing multimodal compositions.

In the essay “Between Modes: Assessing Student New Media Compositions” Sorapure argues for a necessary “transition from assessment practices based in theories about print literacy to assessment practices based in computer-assisted composition theory” (1).

She suggests “an assessment strategy that focuses on the effectiveness with which modes such as image, text, and sound are brought together or, literally, composed” (2), and argues for an emphasis on the use of metaphor, implying a relationship of substitution, and metonymy, implying a relationship of combination (5). These terms “provide a language with which to talk to our students about how the different modes in their projects come together to make meaning” (14).

The essay includes some excellent examples of multimodal works, and Sorapure demonstrates how metaphor and metonymy can be applied to assessing these works. (If you have some time and feel like exploring these pieces, I highly recommend it.)

I found these two terms, and particularly Sorapure’s explanation of how to put them into multimodal assessment practice, enlightening. I agree that a focus on the rhetorical situation is an integral part of evaluating these texts, and Sorapure’s contribution to this conversation is an important one.

On a related note (referring to a person concern), how do you cite a hypertextual essay? As I considered the quotes I would be including in this post, I had to ask myself, “Is it better to include quotes that I don’t know how to cite or not to quote at all?” A reasonable way to include citations would be basing them on the PDF version of the article, which is what I have done here. I find something troubling about citing in such a way, however, as it seems to deny the fundamental condition of the article as a hypertext.

Males and Females: Finally we're getting somewhere

2006-02-15T19:00:00.000-08:00

Gender differences in writing motivation and achievement of middle school students: a function of gender orientation?

Authors:Pajares, Frank and Valiente, Giovanni, Division of Educational Studies, Emory University
http://www.standards.dfes.gov.uk/research/themes/literacy/TueJun241516322003/

Over the past three weeks when doing research on gender differences in writing, all of the articles I looked at seem to agree that there is indeed a difference in the way males and females write. However this article published in Contemporary Educational Psychology explicitly states that it is motivation, not sex that makes the difference here.
The study examined about 500 11 year olds (roughly 50 percent being male and 50 percent being female). Six areas were examined ranging from the student’s self efficacy to task-goal orientation. Females rated higher in all but 1 area which was performance-goal. The research basically found that because a third party encouraged females to do better in writing, they did. As mentioned before, I have been doing research on this topic for the past three weeks. I have looked at many articles and they all seem to say the same thing. This article was a little different, however. This article reported on a study, reported the results of that study and had a solution for the problem. I can honestly say that no other article has been able to supply that piece. Here are a list of implications that the study suggested:

should teachers be examining practice to discover and remedy unconscious gender stereotyping both in specific in the classroom and outside and to explore its implications for achievement?

are there ways in which to influence students’ views of writing so that it is perceived as valuable and relevant to both male and female perceptions?

how can teachers explain and model gender self-beliefs amongst students to help them to recognise ‘feminine’ expressiveness, and the ‘masculine’ wish to succeed so that they are better able to draw on the potential contribution of both?

what are the professional development implications of these findings for teachers: would a

whole-school approach to gender stereotyping help to reduce differences in achievement amongst boys and girls?

From this list of thought provoking questions, this makes it a little easier to at least begin to get males and females on the same playing field when it comes to ALL subjects, not just writing.

"Supporting Boys' and Girls' Literacy Learning

2006-02-15T11:33:00.000-08:00

http://www.oise.utoronto.ca/orbit/girls boys sample.html

The article, "Supporting Boys' and Girls' Literacy Learning" by Shelley Peterson is a nice follow up to Natasha's posting: male and female: nature or nurture. Like Natasha's post, this article supports the idea that while "the gender gap in Grade 8 writing scores is over six times greater than the gender gap in math", the problem lies not with ability, but stereotyping. Peterson makes three points about stereotyping. (1) Boys associate reading with feminity because more than likely their role models (re: literacy) are their mothers and female teachers (2) Feminity is seen as a weakness by boys (3) What boys enjoy and choose to read differs greatly from required school reading. Peterson suggests three ways to begin coping with these issues. (1) Expand boys' experience with reading to include that of male role models. Ask fathers, male community leaders etc. to read to the class and/or discuss the importance of reading and writing (personally and professionally). (2) Explore new ways to evaluate writing. Include areas where boys succeed, for example a. "demonstrates quick, cinematic pace" b. "emphasizes the exaggerated and absurd" and c. "shows loyalty to popular youth culture". (3) Create a writing environment that includes activities boys enjoy: play writing and reading, reading magazines, writing articles, writing and reading about card collecting. Peterson assures teachers that they do not have to totally revamp their teaching styles and techniques, however they should be more attentive to the needs of the male population in their class.

Learners to Teacher: Portfolios, please!

2006-02-15T07:20:00.000-08:00

Per our class discussion last week, I wanted to explore any negative aspects of using portfolios as a method of assessing student writing. After much Internet research, I discovered a great2004 paper written by researchers at the Japanese Association for Language Teachers (JALT) entitled, "Learners to Teacher: Portfolios, please! Perceptions of Portfolio Assessment in EFL Classrooms."
http://www.jalt.org/pansig/2004/HTML/AppleShimo.htm

(Students in this study collected portfolio assignments dealing with their writing abilities in English, not Japanese.)
While the article previewed the basic positive assumptions about portfolio testing, it outlined some commonly perceived negative aspects, namely the amount of time required in the creation of portfolios, that students will most likely have to be guided through the process since it will likely be foreign to many, some students accustomed to exams may become frustrated when asked to perform so much autonomous work and the overall perception that portfolios are "easy" simply because they're not exams.

After the semester, students were asked specific and open-ended questions about their portfolio experiences. Out of 50 responses to the question, "Write anything concerning portfolios as a means of assessment," 33 students preferred portfolios over traditional exams; 10 students said they are benificial while not specifically indicating they prefer them over exams and seven responded with some form of negative feedback about portfolios. Among those responses were that portfolios were "time-consuming", "too much work", "complicated" and "difficult."
As the researchers note, these perceptions contradict the commonly held idea that portfolios are "easy."
Among the reasons students said they enjoyed portfolios were: joy of creation and ownership; more goal-awareness; individual accountability and continuous and extended learning opportunities.

I thought this article did a great job exploring students' perceptions and reactions to portfolios. While I think portfolio assessment sounds like a great educational tool, it was refreshing to read some critical analyses of portfolios.

Multimedia Assessment:Beliefs About Technology and the Preparation of English Teachers: Beginning the Conversation

2006-02-14T16:59:00.000-08:00

The article at the link below rises out of the May 2005 National Council of Teachers of English (NCTE) Conference on English Education (CEE) Leadership and Policy Summit. http://www.citejournal.org/vol5/iss3/languagearts/article1.cfm

This is a draft version offered up for comments (Deadline Feb 1, 2006) and the final version will be published in the March 2006 issue of the CITE Journal.

The conference was concerned with the preparation of English teachers of k-16 with technology and focused on four areas:

1. Newer technologies v. newer literacies.
2. The influence of newer technologies on theories informing our thinking about text, language, and literacy.
3. Composing with multimodal and multimedia technological tools.
4. The political, economic, and socio-cultural influences operating under the practice of new literacies with new technologies.

While I would encourage everyone to read the entire article in our class, especially those unfamiliar with multimedia terms since the article defines many of them, I was most concerned with the third section of the article, Composing with multimodal and multimedia technological tools.

While most of this article dealt with the issuse of teaching and using multimedia in the classroom, towards the end it did attemtp to address the issue of evaluating multimedia compositions:

The evaluation of the composing process and the subsequent texts produced that
integrate multimodal literacies and multimedia technologies will be complex and
require teachers to work carefully with colleagues and students to create fair,
effective, and valid forms of assessment. Evaluation will need to consider not
only process and product, but also design elements, choice of artifacts, and
critical connections and meaning-making. As such, it is important to recognize
and respect the unique role that teachers of writing play in the assessment of
student writing is a role that cannot be relegated to or replicated by software
programs designed to grade student writing through analysis characterized by
prescribed pattern recognition.

Considering our discussions and readings in class, this statement put forth by the conference holdsfascinatingg implications for the future of multimedia assessment. The first sentence calls for the creation of "fair, effective, and valid forms of assessment"-easier said then done. How then do we go about creating these special forms of assessment? Perhaps we should createrubrics based on the areas of design elements, choice of artifacts, and critical connections and meaning-making set forth in the following sentence. Lastly, there is the attack on software programs designed to assess student writing. Are they giving up the battle against AES software and instead shifting the field to the problems and issues of assessing multimedia compositions?

E-Portfolios: Improviing Instructional Practices

2006-02-14T15:47:00.000-08:00

This article begins with describing how young children bring home a collection of their week's work to share with their parents. These works are called "primitive types of portfolio." This way the parents reflect on the student's work for the week and encourage them to strive harder the next time. This is the same as a portfolio. The article states that "electronic portfolios are not a new concept. Literature research reveals that articles were written as early as 1993 on the benefits and use of portfolios in education."

The three types of portfolios the article describes are:
1.Documentation portfolios: Serves as a diagnostic assessment to indicate where the student is.
2. Process portfolio: documents the phase of the learning process.
3. Showcase portfolio: shows the student's accomplishments.

The article is very positive about portfolios and states they give students ownership and responsibility for their own learning. Along with the portfolio, the students are also learning "web page design, how to incorporate various media into the teaching and learning process, and the value of the use of technology in the assessment process."

I believe that this article shows how the use of portfolios and technology are extremely important because "technology is a real world requirement in today's business world."
As we mature from our elementary days at school, a teacher or student can easily access their portfolio and use it for a future job. The employer gets to know the candidate better before the interview takes place.

The article also states since they have incorporated e-portfolios, the student evaluations improved from 4.7 to 4.95 which surpasses the univerisity mean score (4.35) as well as the School of Education's mean scores (4.51). Therefore, this is a good indication that the students fells they are learning more and effectively by using portfolios.

For more information on this article see :

http://vnweb.hwwilsonweb.com/hww/results/results_single_fulltext.jhtml;jsessionid=Q5Q4NUMWYGEKBQA3DILCFGOADUNGIIV0

http://http://vnweb.hwwilsonweb.com/hww/results/results_single_fulltext.jhtml;jsessionid=Q5Q4NUMWYGEKBQA3DILCFGOADUNGIIV0

Males and Females: The missing black male student

2006-02-14T10:05:00.000-08:00

I personally find the ratio of males to females to be an alarming problem. However, I do not feel that this is a problem everywhere. Instead, I feel that this is a problem within my own race, the African-American race. At Hampton University the ratio of black male students to black female students is 1-21. This is an outrageous and alarming ratio. I say this because despite affirmative action or equal opportunity, we are minorities and are unfortunately feared, unwanted, and oftentimes suspect. Still, while African-American women do not experience prejudice to such an alarming degree, African-American men do, and regularly. Ask anyone of my male students and you would be surprised at the ignorance and injustices they have suffered in their young lives. Therefore, considering the state of our public elementary and secondary schools where most teachers are female, and considering statistical evidence that almost 70 percent of African-American children grow up without the providence of an in-home father, imagine the confusion and ambivalence many black male students may have toward not only education, but infrastructure, in general. Many are angry. Many feel ostracized. And so most conclude that they cannot "do" college because they haven't had a black male role model telling them that this is important, that this will gain them respect or self-actualization, that they have a responsibility to care for their families to the best of their ability. Education for us, therefore, is not simply a way to a fatter paycheck. Education for us is the key to a better life. It is an open-door out of poverty, it is a compass by which one can navigate through the messages of Hip Hop and the antithetical messages of institutions of learning, and it is the place where they gain the ability to use a "voice" that will be respected and ultimately heard by others, a voice that will effect change. With this in mind, I am thankful that there are organizations out there recognizing the severity of this problem, such as those mentioned on http://www.insidehighered.com/news/2005/12/05/blackmale. And it is my hope that we will begin to use these title IV services to reach a long neglected and even hated group: the black male.

male and female: nature or nurture

2006-02-09T16:15:00.000-08:00

According to a study available at http://www.psychologymatters.org/thinkagain.html conducted by Professor Janet Shibley Hyde, a researcher at University of Wisconsin, there is no real difference between male and female students' abilities to perform well in language arts. In an article titled Think Again, Men and Women Share Cognitive Skills, it states:

As for verbal ability, in 1988, Hyde and two colleagues reported that data from 165 studies revealed a female superiority so slight as to be meaningless, despite previous assertions that “girls are better verbally.” What's more, the authors found no evidence of substantial gender differences in any component of verbal processing. There were even no changes with age.
What the Research Means
The research shows not that males and females are – cognitively speaking -- separate but equal, but rather suggests that social and cultural factors influence perceived or actual performance differences. For example, in 1990, Hyde et al. concluded that there is little support for saying boys are better at math, instead revealing complex patterns in math performance that defy easy generalization. The researchers said that to explain why fewer women take college-level math courses and work in math-related occupations, “We must look to other factors, such as internalized belief systems about mathematics, external factors such as sex discrimination in education and in employment, and the mathematics curriculum at the precollege level.”

After reading her study, I am more prone to believe that it is not that there are truly cognitive ability differences in male and female students. Rather there are sterotypes that are formulated about their abilities that carry over into both college and the workforce, resulting in pervasive stereotypes that prevent women from gaining entry to male-dominated majors or male-domintated professions. Likewise, this same prejudice, perhaps, "forces" men into male-dominated areas to avoid a "feminine" label in making their academic and professional choices.

Multimedia Assessment: The difference between analyzing student writing and selling shoes.

2006-02-08T19:07:00.000-08:00

Appearantly, the author of How the Internet Will Help Large-Scale Assessment Reinvent Itself sees a world where everything is connected -- except for communication & academic assessment.

This article equates the business of assessment with the tools of assessing a business. In other words, its guiding thought is that internet technology is developing rapidly in ways that help commercial businesses, and that large-scale assessment projects can benefit from the same advances. Both fields, for instance, benefit from an increased level of interactivity between content provider (test giver) and end user (test taker) and an increased ability to instantly provide "dynamic stimuli" (basically, media such as pictures and video).

The article effectively argues that the internet allows for products to be customed designed (it links to a site that will custom design sneakers) . It also effectively argues that the internet makes long-distance learning a much more viable option. However, its a big jump from the world of customizing shoes to the world of assessing student knowledge. If the article had limited itself to certain kinds of specialty training, then I could begin to see the article's point. A computer program could respond to an electrician (or even nuclear physicist) making virtual decisions in a simulation program. But, without really specififying how, the article says the internet and technologies based on the internet will soon be the only viable option for handling postsecondary school admissions.

It is true that that computers allow for large numbers of students to take customized tests. The computer based GRE, for instance, takes your previous answers into account when choosing the next question. But there is a large issue in postsecondary admissions that the article fails to do more than mention: writing ability. By failing to focus on this, the article really takes a pretty strong stand against writing as an important issue. Because, unless we include essays being sent by e-mail (which does little to move forward the article's claim that technology will not just change delivery systems but reinvent assessment itself), there is little to suggest that technology will ever entirely replace readers. Yes, there are some programs that will assess writing. But these programs are looking for writing that aligns with a norm. By definition, then, it will penalize innovation -- at least until true AI becomes a reality. There will thus always be problems with it -- therefore human readers will continue to exist.

There are other perspectives on the above issue, but the fact that this article can be so confident in technology's ability to assess without dealing with the issue of communicating seems to underscore how out of touch assessments can be with the real world. It spends one sentence and one end note on writing, pointing out that word processed papers seem to give a different measure of writing ability than hand-written papers. That point would have been an excellent place to jump into the place of assessing different forms of media in this new technological world. Because multi-media literacy is no longer an option in the real world: its a necessity. Being able to at least navigate (if not create) webstreams, pdf, html, powerpoint, and other forms of computerized media is essential to most professional fields. The article itself says "businesses are becoming more concerned with what employees know and can do, and less with where they went to school." Can employees communicate? If so, can they communicate with a variety of audiences or just to a computerized prompt? I definitely agree that the internet is a revolution in learning to communicate, but assessing that communication is still going to come down to a human audience. As long as learning assessment and communication continue to be thought of as things that can be discussed separately from one another, I fail to see how technological advancements alone will ever lead to a true "reinvention" of academic assessment.

Portfolios: A Tool Used with 5th Grade Students

2006-02-08T09:37:00.000-08:00

As I searched to find an article about the use of portfolios in middle schools or high schools on the internet, I came across many public school web sites talking about their use of portfolios or their plans to implement portfolios. (The most widely noted state was Kentucky as referred to in a previous post.) I realized that I did not think it was a question of if portfolios are being used, but rather how they are being used and in what formats they are being used. I also found several sites, if not more, on digital and electronic portfolios. So perhaps the article, " Student Bloggers, Podcasters Fall in Love with Writing" by Scott Rees of the Lansing State Journal, is a two-fold one: it illustrates portfolio use with 5th grade students, and it illustrates the use of multimedia portfolio usage.

In the article, fifth grade teacher, Tara Autrey, is spotlighted for her use of portfolios with her students. Autrey has her students create digital portfolios composed of their writing through web logs, and they are published on the internet. Autrey adapted the concept from the Red Cedar Writing Project at Michigan State University, and I believe this is her second year of implementing this project. Autrey encourages her students to read each other's work and post comments as well. In addition, she allows them to incorporate podcasts, recording voice commentary about literary works or topics. Autrey also notes that this type of portfolio enables her to asses her students writing more efficiently, and the students take more consideration about the purpose of their writing and their audience because they know it will be published.

You can view Autrey's students work form last year at http://www.msu.edu/user/autreyta.

Peer Assessment and Portfolios

2006-02-08T08:07:00.000-08:00

The article “Assessing Higher-Order Thinking Using a Networked Portfolio System with Peer Assessment” is a pilot study done by the National Central University in the Republic of China. The study was exploring the “qualitative evidences in order to provide a more clear view about networked portfolio assessment.” The abstract states that the “conventional methods of assessing students’ ability failed to assess higher-order thinking”: therefore, an alternative method is being studied at the University of Republic of China. In order to assess the higher-order thinking this network portfolio system will aid in “helping students train their critical thinking and analytical skills.”

First, let’s discuss the components of a networked portfolio which includes a portfolio and as stated in this article. “A portfolio is a purposeful collection of a student’s work that tells the story of student achievement or growth, and a collection of items that reveal different aspects of an individual child’s growth and development over time.” (p139) The second component is having a “flexible interface for submitting homework” and the teacher’s ability to communicate with the students. Because students are assessing each other there may be some cases where a student may not agree of the peer’s assessment; therefore, the student can challenge his/her peer. In this challenge, the student has a chance to “retort the comments from peers, or explain the misunderstanding.“

The next component should be a web forum whereas a student only browses and assesses three assignments from their peers. All work is anonymous; therefore, the student does not know whose work he/she is assessing.

The next step is self-assessment and in my opinion is the key component in assessing portfolios. After the students assess their peers, they must reflect on their own work. The article states that “when the student assesses the other’s homework, he/she will notice which point he/she needs to pay attention to their next homework.”

In doing peer and self-assessment students take an active role in “controlling and reflecting their learning result” and as a result, take them to a higher–order of thinking.

Article Link:http://vnweb.hwwilsonweb.com.proxy.lib.odu.edu/hww/shared/shared_main.jhtml;jsessionid=XJSPSPY2L14CVQA3DINCFF4ADUNGIIV0?_requestid=27802

Portfolios: Case Study from Kentucky

2006-02-08T08:01:00.000-08:00

Kentucky is one of the few states with mandated portfolios for graduating seniors. This change took place in 1989. In "Portfolios across the Curriculum: Whole School Assessment in Kentucky," (which can be accessed through jstor.org) published in the July 2001 edition of The English Journal , English teacher Lizabeth Berryman chronicles Paul Dunbar High School's progress from the first few disappointing years to measurable improvement. Writing assessment researcher David Russell interviewed 26 of Ms. Berryman's colleagues to: 1) gage their attitude toward the portfolio; and 2) see how they might have changed their teaching methods and/or classroom habits to improve the students' portfolio submissions.

This article goes into great detail about the portfolio requirements, the scoring methods, and the implications for poor performance. Berryman states, "...the stakes were high for schools but low for students" (77). While a portfolio must be submitted for graduation, there was no minimum score to be obtained. However, schools were given 2 years to either show improvement and receive financial rewards, or if no improvement, face the possibility of sanctions.

Four things about this article stand out:

1)the English teachers were charged with the responsibility for helping the students assemble their portfolios including some pieces from other subject areas;

2)the teachers were to score the portfolios, not an outside organization;

3) after initial low scores, the English teachers were able to identify the weakness of their students' work - a lack of audience and purpose;

and 4) the school's scores did not improve until teachers from other disciplines were trained in and performed portfolio scoring, which resulted in many of them making changes to assignments in order to focus student writing, thereby strengthening submissions from the content areas.

When the burden for improvement shifted from the English teachers to every teacher, the scores rose. This is a good example of how education can be strengthened when teachers collaborate to improve a situation in which they are all stakeholders. This is, however, only one example of the impact of the KERA mandate. Additional studies on the performance of other Kentucky schools would be very beneficial.

Assessing Multimedia: Multimedia Literacy

2006-02-07T18:57:00.000-08:00

It occurred to me, while thinking about the topic of assessing multimedia, that inorder to assess, evaluate, or judge a thing we need to understand it. This lead to a search for multimedia literacy and The Institute for Multimedia Literacy website. (Cool introductory movie by the way).

According to their own website the university of courtroom California (USC) Institute for Multimedia Literacy (IML) is trying to expand the traditional idea of textual literacy to one of multimedia literacy and "learning learning how to author and analyze visual, aural, dynamic, and interactive media."

So, there in USC they have entire graduate programs dedicated to the study and assessing of multimedia. They also have a new journal dedicated to this topic as well-"Vectors." The question is: How do they assess their own multimedia productions? No rubric were provided, but perhaps a clue can be found in their own use of ethnographic research.

"Since its founding, the Institute for Multimedia Literacy has used qualitative video ethnography as a primary tool for evaluation of its methodology, pedagogy and educational tools, as well as a source for historical documentation."

Perhaps the best way to assess multimedia is through description?

Sidenote: If your program is multimedia, then how often do you need to redesign your website to keep it looking fresh and so attract students and scholars?

Boys & Girls & Writing

2006-02-07T09:26:00.000-08:00

http://www.tnr.com/doc.mhtml?i=20060123&s=whitmire012306 (If you are not able to pull up the full article and would like a hard copy, please let me know) The article "Boy Trouble", written by Richard Whitmire posted on 1/18/06 in The New Rebublic Online, touched on a top Dr. Whithaus brought up - how the number of boys applying for college has dropped dramatically. "In the early 80s, boys and girls were almost evenly matched in their college amibitions. A decade later, everything had changed. Academic aspirations for girls soared as those of boys pretty much flatlined. And the trend has continued, with girls who say they plan to go to college or graduate school now far outnumber boys. Among female high school seniors, 62.4 percent said they definitely planned to graduate from a four-year college, compared with 51.1 percent of male high school seniors, according to a 2001 survey by the University of Michigan Institute for Social Research." The article represents two theories on this trend - feminist theory and pragmatisits theory. "Why can't boys be more like girls?" The feminists feel that boys should strive to "develope a language of feeling", however the boys who explore this avenue are likely to be teased and bullied. "The pragmatists, mostly male researchers, peer inside the school door and see a feminized world that needs tweaking...Most literature classes demand that students explore their emotions (not a strong point for boys). Other pragmatists point to the simple thing: Basing grades on turing in homework on time guarantees lower grades for boys." What about the fact that "In 1950 about one in three women participated in the labor force. By 1998, nearly three of every five women of working age were in the labor force." ("Women and Work: Then, Now, and Predicting the Future for Women in the Workplace Business Women in the Workplace" http://humanresources.about.com/od/worklifebalance/a/business_women.htm). Moms, the grounding force of our family, are working 9-5. How has this affected our boy students? I can't help but think it's had some impact.

Males and Females:Attitude Towards Writing

2006-02-07T07:08:00.000-08:00

Boys, girls, and writing
Elaine Millard, Sheffield University

http://www.literacytrust.org.uk/Pubs/millard.html
This study, which first appeared in the March 2001 edition of Literacy Today, examines the attitude of males and females when it comes to writing. The attitude, the article suggests, is directly related to the way the students felt about their own writing and the grades they received in turn. These are some of the key points the study found:

Females Males
Enjoyed writing more than males Enjoyed drawing more than writing
Liked to write letters, diary entries Liked to play on the computer
Had problems getting ideas
Enjoyed writing poetry
Wrote shorter pieces of work

The article suggests that one way to fight this problem is to place more emphasis on the actual composition of the work instead of things like neatness, punctuation problems, etc. Males had significantly more problems in these areas, which caused them to receive poorer grades. This, in turn, affected the self-esteem of the males.

Students reaction to portfolios

2006-02-05T16:57:00.000-08:00

http://www.evergreen.edu/washcenter/resources/acl/e2.html

I was interested in exploring student's reaction to a portfolio assignment. While I appreciate White's article and ideas about assessing portfolios, I wanted to research some feedback on his ideas. I discovered an article entitled "Writing Portfolios: What Teachers Learn from Student Self-Assessment" on the University of Washington's Web site, written by Kim Johnson-Bogart who teaches in the Interdisciplinary Writing Program. She succinctly sums up her thesis in a pull quote in the article, "Examples from students' self-reflective essays, created to organize and explain their selections for end-of-term writing portfolios, reveal how much students learn from such reflection. Careful attention to students' reflections invites instructors to change their approach so that it encourages the process of learning that students describe." Johnson-Bogart claims this idea has changed the way she teaches.
But what I found most interesting were the concrete examples from students; two in particular I found to be the most insightful. One student described how she learned from her mistakes, "The next piece is a rough draft and a revision of the opening paragraph of my third paper...What I had failed to do in my second paper was clarify terms and that is what I started to do in this paper as well. To me, the revision shows a clarification process...I had a clear idea of the terms in my head and tried to portray this on paper...These two paragraphs show me that I can read my papers better, although not perfectly, from the readers' perspective now that I could at the beginning of the term."
Another student wrote about the power of peer editing. "Reading other people's papers helped me to develop my analytic reading skills. It was sometimes hard to stay focused in reading other people's papers because I would often find myself saying, 'Why didn't I do that?' or 'Did I also make that mistake?' Reading other people's papers served to break the sort of mind block with which one typically views one's own paper. After reading another person's paper, as unrelated as it may have been to mine, I could look at my own paper with a fresh perspective. I think the reason for this is because in reading somebody else's composition, one must identify with the writer and see the issue as they do, thus breaking out of your own mental cocoon."
Johnson-Bogart concludes that with the portfolio self-assessment project, students realize that merely working for a grade is not always the same thing as truly learning and understanding.
She sums up by saying that before she used portfolio self-assessment, she used to assume that student difficulties in writing stemmed from "a lack of skill or even a lack of rigor." She says she has learned that "such problems often reflect valid and promising intellectual struggles that can be addressed and encouraged by bringing them into the daylight." It is more important to her for the student to understand how they are thinking and learning, rather than for her to receive a polished product at the end of the term.

It was interesting to see White's ideas about portfolio assessment put into real-life practice.

sample: test

2006-02-02T10:03:00.000-08:00

here is the text that I am writing....

NCTE on Assessment and Use of Portfolios

2006-02-02T06:47:00.000-08:00

When addressing assessment and portfolios, it is helpful to consider the policy positions of the National Council of Teachers of English (NCTE). Their website provides several points key to our discussion thus far.

NCTE Standards for Writing Assessment

Of the eleven points stated, the following are the most pertinent to our inquiry:
2. The primary purpose of assessment is to improve teaching and learning.
3. Assessment must reflect and allow for critical inquiry into curriculum and instruction.
6. The consequences of an assessment procedure are the first, and most important, consideration in establishing the validity of the assessment.
7. The teacher is the most important agent of assessment.
9. Assessment must be based in the community.
10. All members of the educational community -- students, parents, teachers, administrators, policymakers, and the public -- must have a voice in the development, interpretation, and reporting of assessment.

Point number ten is particularly interesting since we have noted in class that the results of studies are often not adequately or correctly reported by the media.

NCTE Beliefs about the Teaching of Writing

In this section teachers are encouraged to use portfolios as one of many tools (section 4) and to use them to “assist writers in their development” (section 11). The most extensive, yet surprisingly brief, statement pertaining to portfolios is in section 6, “Conventions of finished and edited texts are important to readers and therefore to writers,” and is as follows:

“Teachers should be familiar with techniques for teaching editing and encouraging reflective knowledge about editing conventions. For example, some find it useful to have students review a collection of their writing over time -- a journal, notebook, folder, or portfolio -- to study empirically the way their writing has changed or needs to change, with respect to conventions. A teacher might say, “let’s look at all the times you used commas,” or “investigate the ways you might have combined sentences.” Such reflective appointments permit students to set goals for their own improvement.”

So while the policy is in place, one wonders what the actual practice is.

Assessing Multimedia: Is PowerPoint Evil?

2006-02-01T18:02:00.000-08:00

Is PowerPoint Evil?
I must confess that I was drawn to this article because of its title and my shared concerns about the “evils” of PowerPoint. Surely I am not alone when I say I have seen some presentations that were just plain bad because of presentation software. Similarly, I have a hard time thinking of presentations which were improved by PowerPoint. That being said, there have been a few occasions in which a PowerPoint was essential to a presentation's effectiveness, and I can see the potential value of PowerPoint for students.

This brief piece explores the benefits/drawbacks of PowerPoint by highlighting a few things to consider when having students create multimedia presentations:

1. PowerPoint doesn't bore people: people bore people.
2. The sins of the overhead user shall be visited upon the computer user.
3. There are more visual learners than meet the eye.

While these issues are important, I wonder if he isn't missing the real problem. I think most of us have experienced, if not created, the sort of generic presentation that multimedia software (like PowerPoint) enables. People now expect every presentation to include a PowerPoint, and it is often a requirement in the classroom. Maybe we are all just sick of the same old presentation. Or maybe, as the author of the article implies, a PowerPoint alone is never going to save an otherwise lousy presentation.

So, should we continue to encourage our students to learn and use PowerPoint? And in terms of assessment, how can we take into consideration the advantages and disadvantages of these technologies when evaluating a student’s presentation? Finally, is their an alternative to requiring presentations be accompanied by PowerPoints?

Gender Differences and Writing

2006-02-01T10:35:00.000-08:00

I don't know if anyone has had time to read the PDF attachment on one of my Blackboard posts. The title of the piece is: Gender Differences in Writing Motivation and Achievement of Middle School Students: A Function of Gender Orientation. It's an interesting read. On page 376 Pajares and Valiante conclude, "...it is indeed likely that many gender differences in writing motivation and achievement of middle school students may be a function of gender orientation rather than of gender." In other words how students feel about themselves, and what they feel is expected from them, maybe be more significant in terms of success in writing than simply their gender. Their life experiences in regard to reading and writing mold their performance. It would be interesting to research students' backgrounds and see if there are any significant events or senerios that impact students, like teacher gender, parent readers/writers, early exeriences with writing and how the child was made to feel about her writing.

Portfolios: From the Old Writing Assessment Model to the New

2006-02-01T08:40:00.000-08:00

http://www.aacu.org/aacu_news/AACUNews04/December04/feature.cfm

It was reported by the National Commission on Writing in September 2004 that many employers were finding that college graduates did not know how to clearly communicate through writing, and many businesses were spending billions of dollars on remedial writing courses. Therefore, the commission recommended that educators revamp the way that they taught and assessed writing. In this article, "Portfolios Transform Writing Assessment at Carleton College", in 2001 Carleton College a private, liberal arts college in Minnesota made the transition from assessing their students writing through one, single course to requiring that students submit writing portfolios with three to five pieces and reflective essays that introduce the portfolio at the end of their sophomore year. The essays are evaluated by volunteer faculty members against "locally developed standards", and are scored "pass", "exemplary" or "needs work". Those that receive a score of "needs work" must resubmit the portfolio. Carol Rutz, the director of the writing program at Carleton, asserts that requiring the students to submit the portfolios mid-way through there college careers allows time for intervention, especially for those students who receive scores of "needs work". At the time of the article (December 2004), Carleton College was noticing positive trends including increased faculty and student relationships in regards to writing assessment as well as 78% of students who were passing the portfolio assessment. To learn more about Carleton College's Sophomore Writing Portfolio or the results of the survey from the National Commission on Writing, please see their web sites.

Writing portfolio study

2006-02-01T05:49:00.000-08:00

http://www.ncrel.org/sdrs/areas/issues/methods/assment/as5naep.htm

In 1990, the U.S. Department of Education conducted a special study using portfolios as a way of assessing writing ability. Approxiamtely 4,000 students (half in 4th grade and half in 8th grade) who took the 1990 NAEP test (National Assessment of Educational Progress) were invited to participate.
Despite some logistical problems (only about half the teachers invited accepted the invitation and some of those who did, recieved their invitation too late to have much leeway in choosing student work), the study is at least an indicator of the interest being shown in portfolio study. Basically, the portfolio writing was compared to the more traditional timed writing assesment. Unfortunately, I couldn't find a newer study and the full report of this study is only available for purchase through the Dept. of Education, but I thought it was worth posting to illustrate the fact of a growing governmental interest in portfolio assessment.

Assessing Multimedia: California Student Media and Multimedia Festival

2006-01-31T11:46:00.000-08:00

http://www.mediafestival.org/

This year will mark the 40th anniversary of the California Student Media and Multimedia Festival. From a writing assessment perspective, what is really interesting is the scoring rubric for the multimedia category. Entries are assessed on three dimensions: Multimedia, Collaboration, and Content on a five point scale (Primary-Trait type). The Multimedia category is looking, naturally, for: "The integration of media objects such as text, graphics, video, animation, and sound to represent and convey information. Videotapes which include sound and images fit this definition."

To paraphrase Sherlock Holmes, what is most interesting is the curious incident of the dog who didn't bark. Nowhere on this rubric are issues of copyright protection addressed. Will students be judged and/or investigated as to the source of their multimedia pictures and sound? Considering the fact that this is a student festival, issues of plagiarism have likely been addressed, at least on a textual level, within the classroom. Can the same be said of multimedia plagiarism/copyright protection?

news: WASL (opinion col) and FCAT

2006-01-31T10:24:00.000-08:00

In the continuing debate about the WASL, Neal Starkman has a guest opinion in the Seattle PI. There's not much in the piece for those of us thinking seriously about assessment issues, but Starkman has some clever moments. Mostly his piece works as a defense of the exam, because he pokes fun of alternatives (e.g., "Plans are also under way to provide alternatives to the WASL, such as the Way Easier Assessment of Student Learning, the WEASL.")

Down in Florida, EDDY RAMIREZ has a story in the St. Petersburg Times about how Citrus Springs Middle School has developed some clever programs for helping students who fail the FCAT.

Males and Females: "Student Performance" Kleinfeld

2006-01-31T09:34:00.000-08:00

According to JUDITH KLEINFELD in her article titled "Student performance: males versus females,"
In a nutshell: On standardized achievement tests of basic school skills, females surpass males in writing ability and reading achievement while males surpass females in science and mathematics. Generally, these gender differences are small. The one exception is the significant female advantage in writing skills. Indeed, the female advantage on standardized tests of reading and writing achievement substantially outstrips the male advantage on standardized tests of science and mathematics.
I guess the question now is why do girls outperform boys in writing skills. Is it the overwhelming female teacher representation in schools in which female teachers assign reading and writing assignments that appeal to girls more than boys? Is it the psychology of mind in which boys and girls are only products of a self-fulfilling prophesy? In other words, we tell them that boys are better in math and science and that girls are better in the humanities and the arts, and so they believe us and thus play out their mandated roles? Is it the fact that substantial school funding has been dedicated to leveling the playing field for girls, who have been cited as unfairly shortchanged by our school systems? Or is it something else altogether?

Males and Females: Boy Trouble

2006-01-31T08:40:00.000-08:00

http://www.tnr.com/doc.mhtml?i=20060123&s=whitmire012306
"Boy Trouble" The New Republic Online

This article starts with recanting the comments that Harvard University President made last year. He basically stated that males are superior academically in comparison to females and this quality was innate. This comment was one man’s opinion about why women do not hold distinguished fellowships and professorships in the same high quantity as men. But as the article states, the U.S. Department of Education found that males are much more likely to be identified with a learning disability, repeat grades and/or drop out of high school than females. The article also goes on to state, “Between 1992 and 2002, the gap by which high school girls outperformed boys on tests in both reading and writing--especially writing--widened significantly”

So what explains this drastic difference in the way males and females perform in the classroom? There are many suggestions, most being by male researchers. Some suggest that the reading material in class is too feminine. Activities that require a male to explore his “inner emotions are not exactly interesting for males. Bad memory has also been to blame. Some say that males have worse memories than females and not remembering to turn in an assignment affects male’s grades.

These low grades can be used to explain why there are so few men in college. But some also argue that if one were to look more closely at grades, that number should be even lower. Colleges also admit males because they need males, not because those males have the grades.

males and females: differences on the Washington state exam

2006-01-30T17:29:00.000-08:00

Back in Dec., I was tracing press coverage of the scoring differences on the WASL. There is also a couple of comments from folks in NYC and in Washington state on gender differences in terms of scoring on standarized (writing) tests and admission to college. All of these concerns pre-date the Newsweek cover story that I'm sure K. will link us up with.

Assessing Multimedia: Multimedia Mania

2006-01-30T13:07:00.000-08:00

http://ced.ncsu.edu/mmania/

Apparently, ISTE (International Society for Technology in Education) has been running a contest for a few years now called “Multimedia Mania.” The winners sites from 2004 are not all still up, but those that are up are very impressive. Though none of them intensively feature anything we might call “creative” writing, the formats for the sites look like they could easily be adapted to a wide variety of purposes. And if we really are to include video, power-point, and other technologies as acceptable forms of expression, then these sites could be considered approaching web-based portfolios (the things they’re missing are a sense of progression and a reflective statement).

The thing that really impressed me was the contest’s cross-software multimedia rubric (http://www.ncsu.edu/midlink/rub.multi.htm). It tackles a question I had: how does one grade technology? If the teacher allows any software to be used, can there be a baseline for comparison? If the teacher allows only one kind of software, is that an unfair advantage to students already familiar with that particular software?

The ISTE’s answer is to allow and even encourage a broad range of software applications. A baseline for comparison is acquired through a 16 part of rubric that basically comes down to “is it user friendly?” As teachers of writing, we would want to include elements to the rubric that reflected our concerns about self-expression and written articulation – but, if you think about it, “is it user friendly” is really just another way of saying “does it connect with its intended audience,” which is a major interest in good writing. So I was very impressed when I came upon this site and think that, while not cut&paste applicable to the writing class, it has components that we can use.

issues in writing assessment

2006-01-29T15:52:00.000-08:00

During Feb. we will be talking about three issues related to writing assessment: (1) assessing multimedia and multimodal compositions, (2) portfolios, and (3) scoring differences between males and females. Look for the opening set of posts shortly.

To make the tracking of threads easier, posts will be labeled:
(1) assessing multimedia: XXXXwhatever the post titleXXXX
(2) portfolios: XXXXwhatever the post titleXXXX
(3) males and females: XXXXwhatever the post titleXXXX

Enjoy!

Writing Assessment

2006-01-23T06:47:00.000-08:00

The Literacy of America's College Students

The Pew Charitable Trusts just released a report titled "The Literacy of America's College Students." The entire report, as well as a more modest fact sheet can be found at their website through this link:
http://www.pewtrusts.com/pubs/pubs_item.cfm?content_item_id=3193&content_type_id=8&page=p3

The study examined the "English-language abilities of 1, 827 students in their final year at 80 randomly selected 2-year and 4-year public and private colleges and universities across the United States." The language abilities were meshed in three areas: Prose literacy( ability to read and understand newspaper articles, brochures, and other material of that type; Document literacy measured the ability to comprehend such documents as job applications, maps, tables, or food and drug labels; Quantitative literacy was the ability to perform computations such as balancing a checkbook or calculating a tip. (Report Fact Sheet)

More importantly for our discussion of writing assessment is the fact that the study is introduced on the Pew Charitable Trusts website with one-page defense of the importance of assessment.

"With the recent attention on accountability measures for elementary and
secondary schools, accountability in institutions of higher education has been
all but overlooked. The National Survey of America's College Students (NSACS) is
a study that examines the literacy of U.S. college students, providing
information on how prepared these students are to continue to learn and use the
skills that they will need in the years to come. Such an examination provides a
valuable set of indicators of performance in higher education, informing such
issues as the relationship among educational experience, literacy, and
preparedness for the job market."

Since writing and reading are so closely entwined, I would argue that the assessment of literacy skills holds implications for the assessing of writing as well. If nothing else, a Pew Charitable Trust report on writing skills of college students may also be on the way. Of course, when this was reported on public radio this morning, the low scores in quantitative literacy were emphasized above the rest of the items in the fact sheet which were either neutral or tending towards the positive. Should a wide scale assessment of college students show up, I suspect that the negative findings will likewise be promoted.

Writing Assessment in Georgia

2006-01-20T08:27:00.000-08:00

There was a story in the Atlantic Journal Constitution on Gwinnett County's high school juniors outperforming their peers statewide. I'm not posting the link, because AJC is requiring one to create an account to see their pages.

Pearson Knowledge Technologies releases WriteToLearn

2006-01-19T17:37:00.000-08:00

In our tracking of IT and writing assessment, there is a note on the business side of things today--a product release by PKT.

Note the claims made in the Business Wire Press Release: "The KAT engine gives students immediate feedback on the content of their essays - specific enough so students know where they need to improve, but open-ended enough for students to teach themselves to be better writers."

With claims like these being made, English teachers need to understand how the different software programs designed for automatic essay scoring and feedback work. How does one evaluate whether WriteToLearn or another piece of software "works?"

Hopedale, MA. tests out computer-based MCAS tests

2006-01-19T17:27:00.000-08:00

Here are the opening three sentences from Patrick Lally's article in _The Milford Daily News_:

Jessica Lovely sat in the Junior-Senior High School’s wireless computer lab yesterday, reading her assignment off of a laptop computer.
"Your teacher would like you to write about how you have changed since the beginning of the year," it read.
The answer was staring Lovely in the face. For the first time in her school career, she was taking a Massachusetts Comprehensive Assessment System exam without pencil or paper.

The issues addressed in Lally's article resonate with points brought up in the colloquium discussion today. If anyone wants to summarize points from the colloquium as they relate to moving writing tests online or into computer-mediated environments that would be useful.

Other important notes from Lally's article: the test were administered by Measured Progress, a Dover, N.H., testing company. Michael Russell, a professor at Boston College is quoted as seeing many advantages to computer-based testing, and the English teacher, Ann Marie DeVito, Lally quotes took the GREs online, uses technology and therefore feels that moving student exams online is a logical step.

Major Happenings Last Week

2006-01-16T15:52:00.000-08:00

Last week (Tuesday) was the 4-year anniversary of No Child Left Behind.

Last week (Weds.-Fri.) was also the first meetings of the Steering and Planning Committees for the NAEP 2011 and 2019 Writing Standards.

The first class meeting of the colloquium on testing and technology in English happened last week (Thurs). Major issues were: illegal literacy, usability, validity and reliability and situated cognition. Students have posted within Blackboard. So far, the posts have been on the differences between typing and writing with a pen and paper. I'd classify these posts as focusing on situated cognition, although they also address the issue of usability. We'll have something to share on the blog later this week or more likely early next week.

We're also leading towards a discussion of Brian Huot's Chapter 3 "Assessing, Grading, Testing and Teaching Writing." Defining assessing, grading and testing should be a useful activity that may be worth sharing on the blog.

400,000 students to be tested in NYC

2006-01-10T18:45:00.000-08:00

New York Newsday reports "Elementary Students Wake Up to Statewide Tests" In the article WIL CRUZ and JOHN HILDEBRAND do an excellent job of locating standards-based testing in relationship to political influence on educational policy. They point out that the expansion of testing from grades 4 and 8 to 3 *through* 8 is a direct result of George W. Bush's "No Child Left Behind Policy." They write,

President George W. Bush, who pushed for the law's passage four years ago, has promoted it as a means of boosting academic achievement nationwide, while also closing the performance gap between students of various racial and ethnic backgrounds.

They also do a good job tracing the opposition to this testing by pointing to major concerns raised by groups that study testing, school administrators, and the teachers' union.

They quote Jane Hirschmann from Time Out From Testing who says "standardized testing is supposed to be the measurement of reform -- not the reform itself."

"That is not the way you improve or raise standards for children ... by putting in testing," Hirschmann said. "And parents have come to see that ... children are not being educated."

International News: Battling Illiteracy in Oman

2006-01-07T17:06:00.000-08:00

There's an interesting article in the Times of Oman on the efforts there to battle illiteracy. The measurement and assessment of writing is connected to the measurement of the success of the educational programs. It's not exactly Paulo Freire here, but the work is necessary and it highlights an aspect of writing assessment that gets less attention than high stakes testing of individual students.

WASL editorial letter in the Bothell Hearld

2006-01-07T16:52:00.000-08:00

Jackie Minor has written a response to an editorial in the Herald that talks about alternatives to the WASL. Minor's letter has a slightly skeptical/critical tone toward the "former governor" who joined the WASL foes (see Herald article from Dec. 10,"Former governor joins WASL foes at meeting")and toward Rep. John McCoy argument that the state should provide "multiple ways [for students] to prove their knowledge." Minor ends her letter with this sentence: "Lets hope that those in charge will not lower the standards but will inspire our students to meet these needed goals." It's funny to me that the suggestion of alternative assessments means lowering standards. Alternative assessments such as portfolios or course grades do not mean lower standards, just alternative methods for demonstrating student knowledge.

The rhetorical move to equate other forms of assessment besides a one-size fits all, standarized test (often heavily weighted toward multiple choice questions or a single one-shot writing exercise) with "lower standards" was very common in the 1990s debates. It was a major tool in the lead up to our current age of high-stakes testing in many states.

An analysis of the way this rhetorical move has been used in public debates about assessment, particularly writing assessment, would be a valuable contribution to the research literature.

Agenda for the next four months

2006-01-06T13:47:00.000-08:00

Next week the semester will start at Old Dominion University in Norfolk, VA. I'm going to invite the 11 members of my graduate seminar on Testing and Technology to write and manage the blog. We will keep up with the latest news on writing assessment, but we will also be running in parallel a series of discussions on themes and issues from the course (such as holistic essay scoring, Automatic Essay Scoring (AES) software and blogs in the high school and college writing classroom). We hope you enjoy these "themed" blogged entries as well as our coverage/commentary on the latest writing assessment issues in the news.

Re-cap of Issues from Dec.

2006-01-04T13:02:00.000-08:00

After a week + away from the blog for a break and time with the family, I'm ready to go again. It seems that the way back into the writing assessment blog and into the semester is to think abit about some of the issues from Dec.

Hot spots were: (1) WASL. the WASL results, particularly the gender gap (boys scoring lower than girls) and the potential that some Seattle area schools would seek the option of using alternative forms of assessment (e.g., portfolios and course grades).

(2) Troubling blogs. Although I didn't spend much time on it here, the trouble of three Chicago students and their blogs earned a good amount of attention on blogs around the web and on the tech-rhet list serv.

(3) TOEIC. Internationally, the TOEIC and ETS CEO Kurt Landgraf earned top billing in Asian media coverage of writing assessment and ESL issues.

Representations of Student Work and Assessment Scores (databases)

2005-12-22T08:54:00.000-08:00

As we move toward the end of 2005, here is a nice little article from Noblesville, Indiana about using FileMaker Pro to make assessment data accessible to teachers. According to William Fouts,

Data Dashboard can instantly pull ISTEP assessment information for individual students or groups of students and graphically display how those scores compare to previous year's scores, how they compare to the average scores in each school as well as state averages.

The issue of how student performances or abilities are displayed/represented is also at issue in Kathi Yancey's "Postmodernism, Palimpsest, and Portfolios: Theoretical Issues in the Representation of Student Work" (in CCC 55.4 June 2004).

The juxtaposition of an article about using a database to more effectively represent student scores on ISTEP and Yancey's article is a bit strange. But, I think there is an important issue in both pieces--how teachers see not only student work but also the results of student assessments relates to the usefulness of the assessment. If assessment data are buried and impossible to read, then the data does not provide formative feedback. It's a dead end. However, if only scores are represented for writing proficiency--that is, if the complex and multiple performance that constitute a student's writing abilities are reduced to a single number or letter--then the teacher is still getting a limited picture of the student's abilities as a writer.

The potential to use database programs on the local level, as Dan Chapin and Tim Sturgeon have done at Noblesville Middle School, to represent student writing abilities is well worth discussing. (Eportfolios running through OSPI are one way of tackling this question). What I find intriguing about Chapin and Sturgeon's work is that it is a great example of innovation within the schools. Assessment data must be useful, and teachers and principals are an--if not the--essential audience for this data. When they start shaping the IT for their own needs, we are moving in the right direction.

Google Top 10 on Writing Assessment: Why these links are hot

2005-12-20T17:36:00.000-08:00

Although I will begin a review of the top writing assessment stories in Dec. 05 shortly, I thought it would be fun to look at and analyze the top ten links in Google when you enter "writing assessment." Here's the screen shot; click on it to see the image in a larger--readable--format. I went ahead and included the sponsored links, because they also tell a story.

Top prize today 12_20_05 goes to the College Board. It's a sponsored link, so although the College Board is important, take its placement at the top of the Google list with a grain of salt.

The next spot--and the true number 1--is held by a "6 Traits" page. Although not the official NWREL site, 6 Traits is kind enough to point toward the NWREL site.

Number 2 on the list is "Writing Assessment Services," a tutor who specializes in working with home schoolers.

Number 3 on the list is NWREL (which is now 6+1 Traits) and includes a nice PDF file you can download.

Number 4 is from Tennessee and discusses the TCAP, a performance-based writing assessment.

Number 5 is NCTE's "Writing Assessment: A Position Statement."

Number 7 is a GRE-related site.

Number 8 is NAEP, the Nation's Report Card.

Number 9 is a cool, but dated bibliography from 1997 by Janet Pariza from Northern Illinois University.

The ranking of these top five is curious. Surely NCTE deserves higher billing than a tutor for home schoolers, who is hanging out his shingle on an aol.members website. But "deserves" is a funny word in the world of Google ranks. Links and hits count. The 6+1 Traits sites remind me of their importance, and remind me that writing assessment has a longer reach than secondary and post-secondary students. The TCAP ranking is fascinating. Why is the Tennessee test in the top five Google links today? Why not the WASL or the FCAT?

The title of this post promised answers. I don't have them, but let's note the current standings and see how things change over time.

cross-cultural issues and writing assessment

2005-12-20T16:58:00.000-08:00

I found this post over at CAC.ophony.org. Some interesting comments on assessment, plagarism and cultural issues.

Coverage of the new writing portion on the TOEIC in Asia

2005-12-16T09:11:00.000-08:00

A couple of days ago, I included a post on the TOEIC. The Japan Times has just run on story called "TOEIC test upgraded to include writing, speaking components" by SEANA K. MAGEE.

As in the previous piece on the TOEIC from the chosun Kurt Landgraf, ETS president and chief executive officer is quoted. His comments come from a news conference in Princeton, NJ from Monday (12/12/05). The Japan Times gives a fuller version of Landgraf's comments:"The current version of the TOEIC test has served its constituents faithfully since its introduction in 1979. However, a decade of ETS research on the English language and how people communicate in practical, everyday circumstances has confirmed the need for assessments that reflect more authentic tasks such as those encountered in the global business environment."

These two articles underscore the business of testing--particularly testing individual's proficiency of writing in English--in Asia. This area of writing assessment is not discussed enough by L1 compositionists within the U.S.

WASL and spending

2005-12-16T09:05:00.000-08:00

Washington governor, Gregoire has proposed spending $38.5 million to help students (high school sophmores) who fail the reading, writing or math portions of the WASL. Gregoire' plan would give funds to schools where students do not pass. This approach is distinct from states such as Florida where schools are punished in terms of funding when a substantial number of students do not pass stante-mandated tests (e.g., FCAT).

NAEP Writing Assessment

2005-12-16T07:33:00.000-08:00

The NAEP writing assessment presents a broad view of how well America's students are writing. The 1998 and 2002 studies provide a snap shot of writing in American schools. The NAEP criteria and framework for writing include narrative, persuasive, and informative types of writing. They also ask for students to compose in a variety of genres and formats, such as "writing a letter to the editor of a newspaper, offering advice to younger students, reporting to a school committee, and writing a story based on a poem."

Given this effective framework for evaluating writing in multiple genres through a variety of tasks as a way of assessing writing in America's schools, why do we assess individual students through decontextualized, single-shot exams such as the SAT w-component? The contradiction seems to be between an effective *method* for measuring how the entire school system is working (i.e., The Nation's Report Card) compared with a method for assessing individual students based on a single sample of their writing.

Understanding the NAEP method may be an effective way for developing alternative forms of writing assessment that measure individual student's writing abilities.

The challenge facing NAEP is how to measure the changing forms of writing. How does one account for the impact of information and communication technology (ICT) on student writing? Where the heck would blogs fit in? Can NAEP's flexible and multiple methods of assessment be adapted to reflect the impact of ICT on students' composing processes? Yes. Of course, but the interesting question is how....

Writing Assessment in Georgia

2005-12-14T17:14:00.000-08:00

The Atlanta Journal-Consitution notes that 98% of students passed the Georgia writing examine. Here's a brief description of the Georgia exam:

"The assessment, taken by juniors in October, is a composition on an assigned topic. Students have to demonstrate persuasive writing in no more than two pages."

Here's a link to the full story.

It's an impressive pass rate. There is also a fascinating question about why the writing is limited to "no more than two pages."

TOEIC and ETS

2005-12-14T09:34:00.000-08:00

Here is a fascinating bit from ETS CEO Landgraf as reported at chosun.com:

“A decade of ETS research on the English language and how people communicate in practical, everyday circumstances has confirmed the need for assessments that reflect more authentic tasks such as those encountered in the global business environment,” ETS CEO Kurt Landgraf said.

Few would want to argue with Landgraf's statement, but the practices of ETS in the past have not suggested much concern with authentic tasks. As editor of Assessing Writing, Liz Hamp-Lyons has often been critical of ETS's practices. Her editor's column, "What is writing? What is “scholastic aptitude”? What are the consequences? SAT I Writing — a trip down memory lane," in 10.3 outlines ETS's "un"-authentic assessments of students' writing abilities on the new writing component of the SAT.

I'll try to link to Hamp-Lyons' column through Elsevier's web version. You--or your library--must have a subscription to Assessing Writing to follow this link.

PLATO Learning

2005-12-14T09:23:00.000-08:00

PLATO Learning is a company that has its roots in the early computer-assisted instruction (CAI) work of the 1960s. It's still going strong (traded on the NASDAQ as TUTR). This story in Business Wire reports that PLATO Learning "announced revenues for its fourth quarter ended October 31, 2005, totaling $33.7 million. This is an $8.7 million or a 21% decrease from the $42.4 million reported for the comparable period of fiscal 2004. The revenue decline was in line with expectations announced on September 1, 2005, and was due to low sales productivity, caused by changes in sales processes, procedures and organization during the year, and by attrition of sales personnel."

Understanding the origins of PLATO and TICCIT and other early CAI programs in terms of their social and business contexts is important for understanding how educational software has developed and how it has been marketed. My article, "The Development of Early Computer-Assisted Writing Instruction (1960–1978): The Double Logic of Media and Tools," at http://www.springerlink.com/openurl.asp?genre=article&eissn=1572-8412&volume=38&issue=2&spage=149 explores this issue.

outsourcing the grading of writing (back story from Sept. 05)

2005-12-08T12:04:00.000-08:00

Back in Sept. Inside Higher Education posted this story about "Outsourcing Grading." Matt Barton and Clancy have both blogged about SmarthThinking on Kairosnews.

edublogs and trouble with student bloggers in Chicago

2005-12-08T09:05:00.000-08:00

Two quick links today:

One to Ewan McIntosh's edublogs where Jose suggests that blogs can actually help assessment. No longer will we have to assess 34 versions of the same essay, but rather individualized writing activities will need individualized (or I would argue *situated*) assessments.

The other to a link that Kathy Fitch sent into the tech-rhet list-serv: http://www.suntimes.com/output/news/cst-nws-blog08.html AP students in Chicago write that they will "slit" their teacher's throat "like a chicken's" on Xanga.com and they are expelled. Not exactly writing assessment, but man it shows writing being used and having consequences. There is a whole question of private/public divisions here, and that issue resounds within writing assessment theory....

I'm off to lunch at the Corner Stone Cafe and a critique of the introduction of _Writing Across Distances and Disciplines_ with Julia, Kevin, and Joyce....

WASL and alternative assessments

2005-12-07T08:34:00.000-08:00

In DEBORAH BACH and JESSICA BLANCHARD's "Schools may seek WASL alternatives" (http://seattlepi.nwsource.com/local/250949_school06.html), there is some great coverage of how the Seattle school district is proposing alternative assessment measures to the WASL's standarized format of writing assessment.

Currently, the Seattle district has legislative proposals that "would allow students to skip the WASL and complete an alternative assessment instead." For instance, students could create "a portfolio of their work to demonstrate mastery in the required subjects." But it's not just about one assessment tool for the Seattle district, they want high school graduation requirements to be more like the college admissions process and take into account a student's entire academic history.

This move seems smart. We know that multiple measures are a better means of assessing student learning and their potential for future success than any single measure. We also know if we are going to focus on one criteria high school GPA is a better marker for success than an outside, standarized exam.

It will be interesting to see how the Seattle district's lobbying efforts develop. Will they be able to convince the state legislature to allow alternative assessments? Will it be possible to convince lawmakers that alternative assessment measures such as portfolios or success in demanding high school courses are more accurate assessments of students' learning and abilities than the WASL? These are fascinating points of contacting between educational assessment and policy. We'll see what happens....

Uses and Misuses of WASL

2005-12-07T06:50:00.000-08:00

DEBORAH BACH AND JESSICA BLANCHARD of the SEATTLE POST-INTELLIGENCER have just written an article that addresses the misuse of the WASL as an assessment tool. http://seattlepi.nwsource.com/local/250949_school06.html They report that Seattle School Board President Brita Butler-Wall has said, "The WASL was designed to tell us whether we're doing a good job of educating the child. But what it is explicitly not designed to do is make a decision about an individual child. That's a pretty big difference."

In (Re)Articulating Writing Assessment, Brian Huot has written about this type of misuse of an assessment as invalid testing. That is, if the test is designed to measure one thing and the results are used to judge (and have an impact on students) in a way not included in the exam, then the whole testing operation becomes invalid.

I'll look back at his work and try to articulate this point a little more articulately when I have a moment. (Must go teach in 3 minutes).

Butler-Wall's comment reminds me of the way the WAT exam at CUNY was (mis)used as a rising junior exam rather than a placement exam.

More on WASL gender gap

2005-12-06T07:48:00.000-08:00

Following up on Sunday's report in the Seattle Post-intelligencer Eric Stevick and Scott North of The Daily Herald discuss how and why boys are scoring lower than girls on the reading and writing portions of the WASL. http://www.msnbc.msn.com/id/10318670/ They quote Gov. Christine Gregoire, who says, "I think there is a lot more pressure on boys that it's cool not to be smart. That is something that we need to turn around."

I'm a bit puzzled by Gregoire's comment because the boys' scores do not seem to have legged behind girls in the math sections.

Stevick and North also say, "The results of the computer-assisted analysis of WASL scores run counter to widely held perceptions that girls are the students who need extra nurturing to reach their potential." They draw on Michael Gurian, author of The Minds of Boys: Saving Our Sons From Falling Behind in School and Life, and the New York-based Academy for Educational Development to show that there is a "growing crisis in boys' education" (Academy for Educational Development).

It's fascinating to me to see the way that the gender gap in the reading and writing scores is being contextualized within a larger "crisis" in the education of boys. Is this part of a trend that could later shape educational policy and funding? What do the writing scores really tell us, particularly about how public high schools in Washington state are preparing boys as writers/communicators.

On WASL boys score lower than girls

2005-12-05T06:51:00.000-08:00

The Seattle Post-intelligencer reports that there is a significant gender gap in scores on the writing portion of the Washington Assessment of Student Learning (WASL). The paper notes, "Last year, nearly 75 percent of 10th-grade girls passed the WASL in writing. Boys had a 57 percent pass rate in writing."

Ric Williams, assessment director for the Everett School District, makes the interesting observation that a national campaign similar to efforts made in the 1990s to help girls close the gender gap in science and math is needed--but this time to help boys close the gender gap in writing ability.

Here's the article:
http://seattlepi.nwsource.com/local/6420AP_WA_WASL_Gender_Gap.html

Here's a searchable database comparing results on the WASL:
http://www.heraldnet.com/wasl

back

2005-12-04T17:19:00.000-08:00

After 8 months of being away from writingassessment.blogspot.com, I've decided to return to more regular postings. Lots has happened in the world of writing assessment since March 05. Books have come out; articles have appeared; the SAT exam with its new writing component has been given. In short, the world of assessing student writing has continued to evolve, to shift and to transform itself.

We will look back at some of the events of the last 8 months as well as report on current happenings.

I'm preparing for a new semester in which I will be teaching a seminar on writing assessment and IT. That's part of my motivation for getting back in the blogsphere.

I'm also hoping on tracking more online news about writing assessment.

Finally, issues of ICT--information and communication technology--will take front and center stage as we track developments in writing assessment.

Writing on the STAR exam in California

2005-03-03T11:59:00.000-08:00

from The Mercury News, March 1 via NCTE's Inbox

California's STAR testing system includes a writing test designed to measure students' ability to take information given to them and turn it into something logical, intelligible, and readable. Many educators wonder how much value the test has. "There's very little relation between the scores students receive from the state and their performance" measured by an objective set of criteria, said Leif Fearn, a professor in the School of Teacher Education at San Diego State University.

http://www.mercurynews.com/mld/mercurynews/living/education/11020012.htm

The clause "a writing test designed to measure students' ability to take information given to them and turn it into something logical, intelligible, and readable" in the above sentence is key. It's a certain link between reading and writing that this exam is attempting to measure.

Student fights standarized exams

2005-03-03T11:03:00.000-08:00

from The New York Times, February 24 via NCTE's Inbox

A School Exam's Conscientious Objector

Macario Guajardo was one child left behind when his classmates took the all-important Texas statewide reading test for promotion to the sixth grade. He stayed out of school in protest against what he called "the big deal" of the testing program, which he said "keeps kids from expressing their imagination."
http://www.nytimes.com/2005/02/24/national/24refuse.html

More coverage of the SAT w-component

2005-03-02T16:05:00.000-08:00

Here's a link to an essay in _The Buffalo Times_.

Funny. This piece makes the W-component sound much more controversial than other news stories have. I wonder if the influence is in talking with the Princeton Review folks. It's funny, but the Princeton Review has been critical of the new w-component. It would seem that they should see it as a positive, but their criticism seems to be along the lines of this essay is a very narrow form of writing.

I don't know if this criticism comes from concern about their revenue stream (will fewer students go to the Princeton Review if they are tested on writing rather than analogies?). I don't know.

My guess is that the comments you hear from Princeton Review folks are their views of the new exam. And, at this point, it's really all we have to go on.

_The Buffalo Times_ piece is worth noting because it is more critical of the w-component than any other mass media piece that I have seen.

It will be worth tracking how the public receives the new SAT, and in particular the w-component. This will be a story over the next few months.

According to admissions officers at B.U. "SAT's new writing section levels admissions playing field"

2005-02-25T10:27:00.000-08:00

Kelly Walter, Boston University's executive director of the Office of Admissions is quoted as saying, "Applicants’ true writing skills soon will be more readily apparent, however, as a result of a new essay section in the College Board’s SAT" in the B.U. Bridge.

Walter is clearly buying the line from the College Board that a 25-minute essay will reflect "true writing skills." The problem is that the 25-minute essay will *better* reflect students' skill at writing than the word analogies on the old verbal section of the SAT; however, the 25-minute essay will not reflect "true writing skills." It will only show how well a student performs in response to a rather bland essay prompt in 25-minutes. Skills involving revision and higher order thinking will not be tested. I'd argue that these are really "true writing skills."

Walter's claims show the dangers of the w-component of the new SAT--although the test designers are careful in their scholarly works to limit claims about the exam's validity, admissions officers and other administators and faculty will expand the reach of the exam. They will make claims for the exam's validity that make the exam invalid (i.e., when you claim that an exam tests more than it does and then you make admissions decisions based on these claims, you take a valid testing instrument and put it to invalid uses. Brian Huot's work is very clear on this issue).

Although test designers are careful about their claims for the exam's validity, the larger organization of the College Board is almost certainly extending claims about the exam that lead to statements like Walter's. Given the information that she has Walter's claim makes sense, but it shows the dangerous path of expanding the meaning of test scores.

More on the TAKS

2005-02-24T10:03:00.000-08:00

Writing for The Gainesville Daily Register ANDY HOGUE says,

The TAKS attempts to measures [sic] how much students have retained and can apply what is learned from state-mandated TEKS (Texas Essential Knowledge and Skills) curriculum: in reading at grades 3-9; in writing at grades 4 and 7; in English and language arts at grades 10 and 11; in mathematics at grades 3-11; in science at grades 5,10, and 11; and social studies at grades 8, 10, and 11.

A great quote from the piece is:

White said he noticed an improvement from the old TAAS test to the TAKS test

"The TAKS test is tied closer to curriculum, which is the TEKS," White said. "In that respect, yes, it's a better test."

The idea of tying high-stakes tests to the curriculum can be both good and bad. On one level, these tests become meaningful because they ask about what is being taught. On another level, the curriculum is no longer responsive to what local teachers see as the issue their students are facing or the problems their students are having. Rather the curriculum is standarized, centralized in such a way that it works against local control.

These issues connect to the SAT and ACT's writing components because the dynamics of these 25-minute timed writing components will influence the *type* of writing that secondary student prepare for. If they work on this narrow form too much, it takes away time from developing more sophisticated writing techniques when they are exposed to more difficult and challenging writing tasks. In addition, there is no thought toward multimodal or multimedia composing. Writing is very limited, very narrow.

More news from KY on assessment

2005-02-22T21:13:00.000-08:00

Writing for the Lexington Hearld-Leader, BRUCE SCHREINER reports:

The Kentucky Senate narrowly voted Tuesday to allow select school districts to develop alternatives to the statewide test that gauges school performance.

Supporters said it would allow successful districts, through pilot projects, to seek improvements over the testing program known as CATS - Commonwealth Accountability Testing System.

On the surface, the move would seem to give schools more local control over testing and the implementation of standards. What surprises me is that "The bill passed on a party-line vote. Twenty Republicans voted for the measure, while 15 Democrats and Independent Sen. Bob Leeper of Paducah opposed it."

Why the surprise? Because, as Sen. Walter Blevins, D-West Liberty pointed out that the bill wouldn't comply with the federal law coined as "No Child Left Behind." So, here is the tension, Republicans are giving more local control and Democrats are supporting a state-wide assessment regime. Funny. The translation of educational assessment into educational policy takes some strange turns.

Harcourt adds new writing component to pharmacy admissions test (PCAT)

2005-02-22T20:58:00.000-08:00

A piece in the San Antonio Business Journal documents that Harcourt is adding a new writing component to pharmacy admissions test (PCAT).

Gene Bowles, Harcourt Assessment's vice president of operations and technology, says, "The role of the pharmacist has evolved, so it is important that our test evolve, too. In the past, pharmacists focused on drug distribution, but today's pharmacists are drug therapy managers and health advisers who need excellent communication skills. The new PCAT writing component is designed to measure those skills."

This move by Harcourt and the American Association of Colleges of Pharmacy is fascinating because not only does it stress the importance of writing but it moves writing assessment into a very narrow discipline-specific type of writing. This exam is a much narrower target than the GMAT writing exam. It will be fascinating to see how this writing assessment tool plays out.

here's what is happening in high schools to prep for the new SAT

2005-02-22T11:19:00.000-08:00

Candice Reed's piece, "Students get a head start on SATs," from the NCTimes points out that students in Vista, CA are giving up their free period to study for the new SAT. They are focusing on the writing section and studying with English teacher, Doreen Robinson.

TAKS tests

2005-02-20T13:42:00.000-08:00

The Texas Assessment of Knowledge and Skills will start Tuesday (2/22/05) and run through Friday (2/25/05).

slow news day in writing assessment, ACT book release

2005-02-20T13:37:00.000-08:00

The Dover Community News gives us the ACT Press Release about The Real ACT Prep Guide. The big news? The book, according to Jon Erickson, ACT's vice president for educational services, includes "samples and advice from the people who designed [the new writing test]."

Test prep.... ACT.... SAT.... standardized exams to measure writing ability.... and the test prep books who love them.

eportfolioshttp://www.blogger.com/img/gl.link.gif

2005-02-18T15:57:00.000-08:00

So you think eportfolios have gone away.... that they are no longer the *hot* topic at conferences.... well, maybe.... but it also seems like they are continuing to heat up....

we may have just turned the corner on the early adopter stage, and now we are entering something new George Siemens has this write up at elearnspace:

http://www.elearnspace.org/Articles/eportfolios.htm

It's a very nice, concise overview of eportfolios as of Dec. 2004.

rolling back writing portfolio assessment in Kentucky

2005-02-17T20:38:00.000-08:00

The KEA is supporting legislation to remove the writing portfolio from Kentucky's standarized testing system, according to Nancy C. Rodriguez's piece in The Courier-Journal. (Feb. 16, 2005)

http://www.courier-journal.com/apps/pbcs.dll/article?AID=/20050216/NEWS01/502160445

Rodriguez writes, "Kentucky's largest education association has come out in support of legislation that would remove student writing portfolios from the state's testing system."

Why?

Portfolios "take up too much class time." They "have led to schools using inappropriate practices such as having students rewrite pieces too many times, or teacher setting aside specific times, in some cases days or weeks, to work on portfolios."

It is terrible when students actually have time in class to write, and then that writing has to be assessed.

And, of course, "scoring inaccuracies ... have raised concerns."

But, the bill keeps on-demand writing. (As if that has no scoring inaccuracies."

Once, again reliability--and even fake, manufactured reliability--is winning out over validity in writing assessment.

Liz Hamp-Lyons's Review of Mark D. Shermis and Jill C. Burstein, Editors, Automated Essay Scoring

2005-02-15T10:41:00.000-08:00

Assessing Writing 9.3 contains Liz Hamp-Lyons's Review of Mark D. Shermis and Jill C. Burstein, Editors, Automated Essay Scoring (Erlbaum 2003).

Her review is critical but fair. (Tt does tend toward the traditional English and writing studies rejection of IT for scoring purposes). There are two excellent paragraphs:

Page refers in passing to the work of such persons as Rebecca Zwick and Janice Lauer, appearing to dismiss this as “incorporate(ing) an applied linguistics approach or attempt(ing) to develop theoretical frameworks for study of writing assessment” (p. 45) while “in the meantime” (ibid.) his work focused on large data sets such as the NAEP (National Assessment of Educational Progress, USA) using his Project Essay Grader (PEG). The implication here appears to be that it is more fruitful to efficiently score very large numbers of essays than to get to the heart of questions about what writing really is, and how its qualities can be identified for all individual writers. That, no doubt, is an issue readers of Assessing Writing will have much to say about!

It is, unfortunately, true that much ‘direct assessment of writing’ occurs in very sterile conditions: large exam halls, a one-size-fits-all prompt, 30 min to “write an essay”, followed by mass scoring sessions at which any single student's text may be looked at by two individuals who each spend less than 60 seconds to reach a “judgement.” AES may do this job better than humans. But that's because the job itself is not intrinsically worth doing. As long as such pseudo-writing acts are required, perhaps we are better off giving them over to the machines.

I don't have time to say more about these now, but Hamp-Lyons is--once again--bringing up the curcial issues in writing assessment from the persepective of those of us concerned about assessment, teaching and learning.

New test gauges ICT literacy

2005-02-15T10:17:00.000-08:00

Robert Brumfield of eSchool news has written a piece that documents ETS's development of an ICT literacy http://www.eschoolnews.com/news/showStoryts.cfm?ArticleID=5504

According to ETS, the web-based exam is "a testing program that measures postsecondary students' ability to define, access, manage, integrate, evaluate, create, and communicate information in a technological environment."

My thoughts are that this exam--and I'm sure it comes from the ICT group whose white paper I discuss in my book--is trying to do what the new W-component on the SAT should do. That is gauge students' abilities to *communicate* via ICT as well as via traditional print.

FCAT's new grammar section embedded within the writing portion of the exam

2005-02-10T16:40:00.000-08:00

Steven Harrison's "FCAT writing raises the bar" in the Miami Herald Feb. 8, 2005
http://www.miami.com/mld/miamiherald/10842657.htm?1c talks about the higher standards for Florida's 4th, 8h and 10th graders on the FCAT.

Harrison notes one issue that will interest those of you concerned about the increasing reliance on multiple choice grammar questions on "writing" portions of standardized exams will want to be aware of:
"This year, for the first time, the writing test will include a multiple-choice grammar section. Those scores will eventually count toward a school's letter grade, but this year the section is a trial run."

Along with Virginia's decision to include sentence diagramming in the SOL curriculum for 6th-8th grade (but the decision not to put diagrams on the SOL exams, yet), this note about the FCAT suggests that standarized "writing" exams may become less about writing and more about abstract grammar knowledge. (Note also the SAT's reliance on multiple choice grammar as part of the w-component of the exam).

regular blog posting

2005-02-10T16:25:00.000-08:00

I've been mulling over the difficulty of regular blog posting on issues around writing assessment.

It's tough, because there is the issue of the history of writing assessment and then there is the issue of current news. On one level, it hardly seems worth while having a blog that is all about history. I mean come on now if it's history why do you need to blog it?

Yet, one of the agendas of this blog is to talk about writing assessment theory and practice and why we are where we are today.

Still, it feels that the blog has to be about the current, about what is just now emerging..... the latest gasps in terms of technology.... the latest on the w-component of the new SAT.... how assessing composition is becoming a multimodal, multimedia affair.... there is so much that is out there on these subjects that to discuss it feels vital.

I guess I will eat my Wheaties and try to keep this blog better than I have in the last six weeks.

speech recognition software and assessment

2005-02-08T10:05:00.000-08:00

Harcourt Assessment acquires Ordinate (from eschools)

http://www.eschoolnews.com/resources/partners/showrelease.cfm?ReleaseID=602

vane

2004-12-06T21:43:00.000-08:00

ok, ok, so a streak of vaniety runs in us all....

here are the top 5 sources that i listed (back when i started this blog):

Top Ten List (Overall)
Yancey, Kathleen. (1999). Looking back as we look forward: Historicizing writing assessment. CCC 50.3, 483-503.

The Neglected "R": The Need for a Writing Revolution. (2003). National Commission on Writing in America’s Schools and Colleges: College Board.

Huot, Brian. (2002). (Re)articulating writing assessment for teaching and learning. Logan, UT: Utah State UP.

Elbow, Peter. (1996). Writing assessment in the 21st century: A utopian view. Composition in the twenty-first century: Crisis and change. Eds. Lynn Z. Bloom, Donald A. Daiker, and Edward M. White.

White, Edward M. (1985). Teaching and assessing writing. San Francisco: Jossey-Bass. {Note: 2nd. Edition 1994}.

Three of my top five appear in scholar.google's top 5.... how cool is that? (Of course, I am counting Huot's 2002 book, which is basically an update of "Toward a New Theory of Writing Assessment" (1996) AND I'm counting White's 1985 _Teaching and Assessing_ although google cites the 1994 edition. But, still we're in the same ball park.

It does not surprise me that "The Neglected 'R'" has not yet made it up there in the research literature. It may never make it, because it is a polciy document as much as a research piece. Also, Elbow's "Utopian" view won't get much play in strictly writing assessment circles---although it should.

top 5 writing assessment works according to google's scholar

2004-12-06T21:29:00.000-08:00

So I had to do it.

Enter "writing assessment" into scholar.google.com's beta version and here are the top five results:

[BOOK] Teaching and Assessing Writing: Recent Advances in Understanding, Evaluating, and Improving Student … - Library Search - Web Search
EM White - Cited by 33
San Francisco: Jossey-Bass Publishing, 1994

[CITATION] The literature of direct writing assessment: Major concerns and prevailing trends - Web Search
B Huot - Cited by 23
Review of Educational Research, 1990

[CITATION] Toward a new theory of writing assessment - Web Search
B Huot - Cited by 13
College Composition and Communication, 1996

[CITATION] Looking back as we look forward: Historicizing writing assessment - Web Search
KB Yancey - Cited by 9
College Composition and Communication, 1999

[CITATION] Effects of Computer Versus Paper Administration of a State-Mandated Writing Assessment - Web Search
M Russell, T Plati - Cited by 9
Teachers College Record, 2001

White, Huot, Huot, Yancey and Russell and Plati. Not too far off my own lists. The Russell and Plati work's presence in the top 5 is a bit surprising, but hey.... sometimes you never know. I'll put writing a review of that piece on my to-do list. Although you can already tell that my blog to-do list often does not get gotten to....

Bob Broad What We Really Value

2004-11-22T11:29:00.000-08:00

Bob Broad's What We Really Value: Beyond Rubrics in Teaching and Assessing Writing (2003) makes some interesting moves. Although it was savaged by Lisa Johnson in her Assessing Writing review (9 2004 181-185), I find Broad's work fascinating.

His move to reach "beyond rubrics" is based on the work of Huot (1996), Moss (1992), and Guba and Lincoln's 4th Generation Evaluation (1989).

Broad writes,

These authors have pointed to a shift in assessment theory and practice toward an interpretive or qualitative paradigm. To date, however, the effort has remained mainly theoretical, and the limited application of such methods to writing assessment has been conducted by researchers, not practitioners. (14)

Fair enough, although I wonder why he does not cite the work of Darren Cambridge and Peg Syverson with the Online Learning Record at the University of Texas. The OLR explicitly calls itself "beyond portfolios," and has been in existence since the mid 1990s. There are moments, when I wish researchers in writing studies would build on previous work rather than reinventing the wheel....

That's grumpy and not entirely fair though.... Broad's work provides not only a complex ethnographic map of how writing assessment in a non-rubric portfolio system occurs, but he begins to chart out the implications for writing assessment beyond the limits of that particular program. Through "Dynamic Criteria Mapping" he is trying to develop a method (what I would argue should be a technique) for writing assessment that is richly contextual.

There are interesting potential links with Broad's work and the techniques I describe as situation, distribution, description, and interaction in Teaching and Evaluating Writing in the Age of Computers and High-Stakes Testing (forthcoming 2005).

Rebecca Zwick Fair Game?

2004-11-15T09:45:00.000-08:00

Reading Zwick's Fair Game? this weekend, I realized that in the opening of she sets up the same dichotomy between pro-standarized testers and anti-standarized testers that I was going for with my discussion of Peter Sacks's _Standardized Minds_ (1999) and Kim Swygert's blog.

Fair Game? is still at home, so I'll have to post some quotes from it this evening of maybe tomorrow.

Peter Sacks standardized Minds (1999)

2004-11-10T09:54:00.000-08:00

During an MA exam this morning, a student brought up Peter Sacks's _Standardized Minds_ (1999). He makes arguments counter to Kim Swygert's claim "that standardized test scores tell you much more about a potential applicant than class rank."

Sacks claims that

Standarized tests generally have questionable ability to predicts (sic). one's academic success.

Standarized tests scores tends to be highly correlated with socioeconomic class.

Standarized tests reward passive, superficial learning, drive instruction in undesirable directions, and thwart meaningful educational reform. (7-9)

Sacks clearly has a polemic agenda--he is anti-test in the way that Swygert is pro-test. The question is how to sort out the conflicting claims. Find the data and determine whose claims are accurate.

SAT III? by Cecilia Capuzzi Simon (NYTimes 1_18_04)

2004-11-10T09:44:00.000-08:00

The next area that I'd like to tease out is the work being done by psychologists to move the SAT and other standarized college entrance exams beyond aptitude or achievement.

Simon points to the work being done by Neal Schmitt at Michigan State, Robert Sternberg at Yale, and William E. Sedlacek at Maryland.

Dr. Sternberg says the goal of his research, which he calls the Rainbow Project, is twofold: to increase a college's chance of identifying students who will profit from its environment and to counteract the pronounced gap on SAT scores of white and non-Asian minority applicants. His test challenges traditional ideas about intelligence. In his view, there are three separate intelligences: analytic, creative and practical. Analytic intelligence -- what the SAT measures -- lends itself to memorization and analyzing information; people express creative intelligence by applying knowledge in a novel way; practical intelligence reveals itself in everyday situations. Dr. Sternberg says that analytic intelligence may not be enough to excel in college or in life. A person who is strong in creative or practical skills may thrive in an academic setting even if he doesn't excel in the analytic skills tested by the SAT. That student, however, is derailed by the test. (Simon)

Schmitt's, Sternberg's, and Sedlacek's work and emphases on dimensions of learning reminds me of my work with *situation* in terms of writing assessment in _Teaching and Evaluating Writing in the Age of Computers and High-Stakes Testing_ (forthcoming from Erlbaum).

more on Swygert's reading of Harper's article

2004-11-09T06:44:00.000-08:00

I promise I'm just about done beating this particular horse, but I was wondering about this statement on Kim Swygert's blog:

* Harper found a college admissions officer who was willing to be quoted by name in saying what all psychometricians and admissions officers know, which is that standardized test scores tell you much more about a potential applicant than class rank.

It flies in the face of everything that I have ever read and heard about the SATs. Throughout NCTE and CCCC--and really the research publications of composition studies--the received wisdom is that high school GPA is a better predictor of college success than SAT scores. So here is my question--and I'm really asking it as a honest, not rhetorical, question: Is Swygert lying or do psychometricians and admissions officers 'really know' "that standardized test scores tell you much more about a potential applicant than class rank."

Ahhh.... it just hit me... maybe the difference is "class rank" and "GPA," although I sort of doubt it. I think the crux of the difference is that psychometricians and composition researchers have fundamentally different discourses. We talk about learning and knowledge in such different ways that we cannot even begin to define--let alone talk in productive manners about--the meaning, usefulness and signficance of standarized testing or assessment.

psychometricians and blogs

2004-11-09T06:37:00.000-08:00

Given Kim Swygert's cheering about John Harper's 8/26/02 article on the new SAT, I had to answer Foucault's good 'old query--"First question, who's speaking?"

Here's what Education News had to say about her blog.

In Defense of Testing Series
Number 2 Pencil: The BLOG of the Century
Tuesday, October 26, 2004

It seems hard to believe that Kimberly Swygert's BLOG, Number 2 Pencil, approaches its third anniversary. It still seems so new. For those of you unfamiliar, Kimberly is a card-carrying psychometrician who expresses in her BLOG her open, honest, and informed opinions on education policy in general and standardized testing in particular. Before she started, no psychometrician had been willing to do this, in the interest of protecting their careers. We all owe her a debt of gratitude for her courage.