A Look at Tools for Assessment and Accountability

• PRINT
• EMAIL

Years Ago, Superintendent Raymond Yeagley met a group of Russians in an exchange program. When asked which aspect of the United States had made the biggest impression on them, the visitors pointed to the local supermarket — the amount of available choices simply overwhelmed them.

Yeagley is the superintendent of the Rochester, New Hampshire, School District, and an early adopter of assessment technology. He says the buzz about assessment and data collection, in the wake of the re-authorized Elementary and Secondary Education Act (ESEA), is a bit like the Russians in the supermarket: too many options. "School districts are suddenly confronted with an expectation that they will use data," Yeagley notes. "Many people are overwhelmed. But the key is to start with the question 'What does our school or district need to know?'"

Clearly, technology is becoming more and more important to the efforts of school districts to measure and account for student progress. Using computers to conduct assessments and manage the data has so many advantages in terms of speed, effectiveness, and flexibility that the days of the No. 2 pencil are certainly numbered. In fact, current efforts to deliver customized assessments that are integrated with instruction would simply not be possible without technology.

The new ESEA mandates more accountability, annual testing tied to state standards, and an emphasis on proven teaching methods. It has also loosened restrictions on some of the ways in which schools can pay for tests. Consequently, there are more rules about collecting and disseminating data and more freedom to pay for it — but very little direction on what works and what doesn't. No wonder so many administrators are feeling as overwhelmed as the visiting Russians.

MAKING SENSE OF THE OPTIONS

One of the favorite new catch-phrases in education is "data-driven decision making," which simply means using data analysis to inform school policy and procedure. Before you choose from the many assessment-related products, take a look at the different categories. Some products generate assessment data, some manage it, and others are simply tools for instruction. Generally, products break down as follows:

• Computerized assessment tools provide standards-based and norm-referenced tests. Some models allow for customization of tests, others require use of pre-loaded questions, and still others are templates that users fill in.
  
• Curriculum management tools allow teachers to access standards-based lesson plans, assess student progress and link to prescriptive assignments, and maintain class lists and attendance records. Personal digital assistants (PDAs) with downloadable tools are now entering the market too.
  
• Administrative tools include lesson planners, attendance databases, and assessment products linked to standards.
  
• Instructional management tools are high-end products that manage the sequencing of curriculum for all grades and subjects. These deliver standards-linked instructional materials right to a student's or teacher's desktop, perform online assessments, and manage information on individual and group performance.
  
• Data warehousing systems allow educators to collect data from multiple sources and create a longitudinal study of an individual child or program. Data warehouses require many elements that work together — including a storage component; data extraction, transformation, and loading tools; and a report generator.
  
• Data analysis tools enable users to import data from a wide variety of sources, disaggregate the information, and display the information through text and graphical reports.

"There's no one-size-fits-all solution," says Geannie Wells, director of the Center for Accountability Solutions at the American Association of School Administrators (AASA). Wells suggests that administrators think about how their data is going to be accessed and stored, who will need it, and how frequently it will need to be updated.

Another issue is whether or not to outsource, Wells says. Administrators should consider whether the district's infrastructure can support a Web-based product delivered by an Application Service Provider (ASP), and whether placing sensitive student data on non-district servers is acceptable.

Charlie Garten, executive director of technology for the Poway Unified School District in California and board member for the Consortium for School Networking (CoSN), says that administrators should think in terms of learning environment plans, rather than technology plans. Over the past years, Garten's technology crew has created a customized data warehousing system for the 60,000 students in his charge, overlaid with a curriculum management tool, SAS inSchool. Garten says his mantra has been: "Build as you need it, change as you have to, and keep it flexible."

BETTER DATA, BETTER DECISIONS

One major benefit of having access to student assessment data is the ability to produce answers quickly and easily. "One night at a board meeting, a board member wanted to know how our kids were doing in math," Yeagley says. "I had my laptop, and within minutes I had a graph that I could show him." To get these quick results, Yeagley used Quality School Portfolio (QSP), a data management tool.

The Rochester schools serve as a beta test site for the product, which is being developed at UCLA's Center for Research in Evaluation, Standards, and Student Testing (CRESST) under a contract with the U.S. Department of Education. Currently, QSP is offered free to schools around the country, and AASA offers training sessions on its use. A Web-enabled version that extends the data access to teachers is currently in development. The district lacks a dedicated technician devoted to data support, so Yeagley appreciates that the system is straightforward and accessible.

Derek Mitchell, the principal architect of QSP, says he designed the product to be both easy to use and robust. QSP includes a data manager that allows schools to import data from many sources, to set goals, and to measure them. The data manager creates a longitudinal data set and also maintains records for students who are no longer active, allowing administrators to track metrics such as dropout rates.

Other products link data to an action plan. NCS Learn's new High Stakes Management suite uses data analysis to forecast how students will do on upcoming state tests, says Steve Gardner, the company's vice president for instructional solutions. "Customers using our software wanted to know how to translate what they were being held accountable for," Gardner says. Schools using NCS Learn's SuccessMaker software can perform statistical analysis on students' achievement scores, correlate them to state assessment scores, and create an instructional plan.

ON-THE-FLY ASSESSMENT

An important innovation in assessment is the trend toward computerized adaptive tests, which tailor questions to students' individual ability levels. This means assessment can be customized for each student, making the test more effective. The Northwest Evaluation Association (NWEA) offers Measures of Academic Progress (MAP) — a bank of 20,000 computerized assessment questions that can change on the fly as the student takes a test. When the student answers correctly, the questions get harder. When a wrong answer is given, the questions get easier.

The strength of adaptive tests lies in their ability to provide a clear picture of a student's strengths and weaknesses in a given subject. Since computerized test scores are immediate, teachers and students can discover academic weaknesses — and begin addressing them — right away. MAP is currently used in 650 school districts, and versions that align to state standards have been built for 47 states, says NWEA Executive Director Allan Olson.

Another company, EdVISION, provides a slightly different approach to adaptive testing. Its Performance Series product offers adaptive testing tied to state standards and delivered via the Web. Because the product is hosted remotely, school districts don't need to install software in order to use it, says William Tudor, EdVISION's CEO. Launched last year, the product is used by 400 districts nationwide, as well as by the state of South Dakota.

Computerized assessments should not be adopted without careful review, however. Given the high stakes associated with test scores, administrators should choose assessment products that have been audited by an outside agency, Tudor says. "There is no such thing as a low-stakes test anymore," he says. "Schools better make sure the tests they use can stand up in court. Not all software companies ensure that their tests are legally defensible."

KEYS TO SUCCESS

If assessment data isn't kept accurate, its usefulness as a decision-making tool vanishes. "Bad data flowing into the best data warehouse is useless," says Bob Olsen, project director for the state of Oregon's Technology Enhanced Student Assessment (TESA) system.

Olsen, who has worked in educational assessment for the past 30 years, says data handling is one of the biggest pitfalls for districts today. Schools often want to customize their questions, which can lead to unclear or "unclean" data. TESA has completed 30,000 online tests this year, and to help keep the data pure, the state requires a difficulty index for each item so questions can be easily tracked and compared.

One vendor's response to this issue is to offer some flexibility while at the same time maintaining control. Riverdeep's Learning Management System allows users to submit questions which the company then aligns to district and state scope and sequences.

Another way to balance flexibility with clean data is to allow teachers to create customized tests based on a bank of pre-written questions. Richard Hyland, director of Title I programs in the Granite School District in Salt Lake City, Utah, says he chose Lightspan's eduTest product because of this flexibility. EduTest will also develop tests for a customer based on specific standards or requirements.

In the end, user acceptance is key to any school district initiative to gather and analyze student assessment data. In the Forsyth County Schools in Cumming, Georgia, teacher input and a dedicated technology support person helped ensure the success of TestTrax, the district's data warehouse system. Paula Gault, the district's superintendent says, it's important that teachers feel their needs are being met — without overwhelming them with too much information. The fact that every teacher in the district's 22 schools has a laptop was also a big help.

Finding a technology-based assessment strategy that fits your needs may seem like a daunting task. But it comes down to following a few key steps: Decide what you want to know. Decide how the data will be used. Find a flexible system that will work with the data and systems you've already got, and that will grow and adapt as you do. Make sure your staff has input, support, and understands the system's limitations as well as its benefits. And last but not least, take a lesson from the Russians in the supermarket — don't bite off more than you can chew.

---

Addressing Concerns

While the benefits of electronic assessment are clear, there are still many concerns.

In a recent issue brief prepared for the National Governors Association, Dane Linn, the group's director of education policy studies, noted that technical variables may affect the dependability of test results. For example, a student taking a Web-based assessment using an older PC with slow Internet access may be at a disadvantage.

Another issue is that of computer familiarity. Some question whether computer-based assessments are biased against students who are unfamiliar with operating a computer.

In the Spring of 2001, TESA tested students in math and reading to determine how well students performed in paper and pencil tests versus technology-based exams. According to the results, the performance of students were almost identical. They found that the technology is not a significant factor.

---

Software Tools

Achievement Management System
TurnLeaf Solutions, Inc.
(866) 829-5900

Chancery Student Management Solutions
Chancery Software
(800) 999-9931

Classwell
Classwell
(866)210-3033

LeapTrack
LeapFrog SchoolHouse
(510) 425-5100

Learner Profile
Sunburst Technology
(800) 321-7511

Lightspan Assessment Builder and eduTest Assessment
Lightspan, (858) 824-8074

Measures of Academic Progress (MAP)
Northwest Evaluation Association (NWEA)
(503) 624-1951

NetSchools Orion
NetSchools
(770) 226-5000

Pinnacle
Excelsior Software
(800) 473-4572

Quality School Portfolio
Center for Research in Evaluation, Standards,
and Student Testing, UCLA

Riverdeep Learning Management System (LMS)
Riverdeep Interactive Learning
(888)242-6747

Scholastic Reading Inventory
Scholastic Inc.
(877) 268-6871

TestTrax
EduTrax Inc.

Wireless Generation
Wireless Generation
(646) 336-5451

Pamela Wheaton Shorr is editor of The Heller Reports' Educational Sales and Marketing Insider, and is a frequent contributor to Scholastic Administr@tor.

Key Components of Report Cards Explained
Although report cards vary widely from one district to the next, you'll almost always find a descriptor section and a narrative section of some sort. Here's a way to familiarize yourself with the both elements.

November Survival Guide
November brings many more "firsts" for new teachers — your first observation, your first round of report-card writing, and your first full slate of parent-teacher conferences. The November New Teacher Survival Guide offers insight on how to get through it all with flying colors.

Clip & Save Checklist: Perfecting People Skills
Want to improve your knack for getting along with students, parents, and colleagues? These questions will help you assess your people skills and pinpoint areas where there's room to grow.

LESSON PLAN

What is Good Writing? Introduces a lesson plan that new teachers can use to show students how to assess and improve their writing. These strategies for the writing...