Papers and Publications

The Good and Bad News of Software Use
for Mathematics Proficiency Test Preparation

Patricia Deubel, Ph.D.

In March 2000, 126,252 public school eighth graders took the Ohio Ninth Grade Proficiency Test (ONGPT). Approximately 46% passed all tests taken, a decrease of 3% from last year’s performance. Pass rates for this year’s eighth graders were lower on all five tests, with the biggest drops in writing (7%) and reading (5%). Mathematics results showed an overall decline of 1% from last year's passing rate (ODE, 2000b). Typically, mathematics has been the most difficult part of the ONGPT for students to pass. The problem can potentially get worse. Ohio's Senate Bill 55, the academic accountability package passed by the legislature in 1997, requires that beginning in 2003 the state's proficiency tests shall measure student knowledge of core academic areas through a tenth-grade level. The first class to be impacted by this new law is the Class of 2005 (ODE, 1998).

In the Ohio Department of Education (ODE, 2000a) report, The State of Education in Ohio, State Superintendent Zelman said that proficiency test results suggest that Ohio needs better approaches in mathematics instruction. In terms of the use of software to prepare students for the proficiency tests, however, the ODE's assessment center does not recommend products to school districts for preparation for the state's proficiency tests. School districts are told: Let the buyer beware (Gillespie, 1999).

Standards adopted by The National Council of Teachers of Mathematics (NCTM, 1989) and goals of the Ohio Schools Technology Implementation Task Force (1999) include that teachers should use electronic products to improve student learning. Students should have access to technology that will help them meet high standards of achievement. Has software use, however, made any difference on students passing the mathematics test of the ONGPT?

To address that question, Grade 8 mathematics, special education, and proficiency intervention teachers (N =113, which was an 88% response rate) in 35 middle schools across 13 mid-sized districts in the Urban Schools Initiative were surveyed prior to the March 2000 tests. The survey addressed teacher beliefs on individual and organizational factors related to the use of software during class time, what software was used, how it was used, and software's instructional and technical merit for proficiency test preparation. Mathematics achievement was measured for over 6000 Grade 8 students. The study received support from the ODE Urban Schools Initiative, which is now the Office of Comprehensive School Improvement. This article presents major findings of the study. The news is both good and bad.

The good news is that software use during math class time appears to have increased among all 13 participating school districts from 1994-2000. Reported use from teachers showed a rise from 3% in 1994-95 to 52% in 1999-2000. The bad news is that software use during class time was not an integral part of the mathematics curriculum in most settings for those districts. Forty percent of participants had never used software in their instruction, which does not necessarily mean that teachers were resistant, incompetent, lacked expertise, or were technophobes, according to Cuban (O'Neil, 2000). Typical among the 52 reasons that teachers gave for not using software for instruction were statements such as:

• There is one computer in my room with the Optimum program loaded. Students use the program during study hall for practice. I don't use it in class because the program doesn't provide enough feedback to students to make good use of class time. I would be thrilled to use more computer software and even previewed some this fall, but the district doesn't provide funds for purchase.
• I don't have the knowledge or training to use computers or software effectively. I also don't have clear goals, whether to teach the 8^th grade curriculum or teach the test.
• Proficiency software was moved out of the classroom and into the labs. I have not taken students as of yet to the labs.
• I have no software on my computer…I have spoken with our computer teacher to see if I can get programs from the computer lab sent up to my classroom. Apparently there are issues with licenses and/or wiring.
• I do not have one computer per student. Also, computers do not teach. They are weak instruments that can be used to help clarify concepts and provide drill and practice. (But if a kid can do something, how much D & P is needed?) In this system, the 8^th graders are so far behind, one does not have time to develop skills before the prof. test is given. Hence, we cram on everything. It is like telling a farmer to plant his corn June 20 and expect it to yield as well as corn planted May 15^th. Get serious! This is not a complaint or excuse. It is simple reality.
• Given all the time most of my 8^th graders spend in front of a computer screen both at home and in our computer labs, most have no intuitive math comprehension or grasp of basic skills. Our learning problems in [school name omitted] are not regarding what tools we use to teach, but rather how we show a need to be educated to people who are involved only in daily basic survival techniques.
• The software we have consists of practice problems and scoring. The value of this software is strictly as a change of pace to our normal proficiency review cycle.
• We do not have access to them in our classroom. The lab has one program, but the computers do not have enough memory to get the newer programs.
• Everyone is on different levels, so we do individual assignments.
• I do not feel that the software we have available is of much benefit and certainly pales in comparison to that which takes place in my classroom.
• Having only one computer is the major reason and the lack of materials. This is also my first year of teaching math and I feel that I was not ready for that step.
• I have asked for such software, but not received it.
• Need more information and willing to use; would be helpful.

Top test improvement strategies that all teachers used for proficiency test preparation included small group tutoring (81%), whole class instruction (75%), individual tutoring (71%), communicating high expectations (56%), followed by using computer software (50%). Even among the 52% of teachers who used software during 1999-2000, most only used it occasionally all year (43%), which meant at most a few times a month (50%).

The occasional use of software during class time actually had a significant negative impact on students passing the test. Software may not have been used in a focused manner to help individual students to master specific skills in which they were weak. More students who had not used software during class time passed the test than those who did use software during class time, a significant difference (Table 1).

Software use made a significant difference on passing the test for students who did not use software during class time, but had used software in proficiency intervention classes that met in addition to regular math classes. Unfortunately, most schools did not have the luxury of offering such classes. The implication is that the personalized, regular and focused effort on using software over time to combat individual weaknesses and to build skill fluency led to achievement gains (Table 2).

So why haven't teachers used software more often in their instruction? Results indicated that administrative support, teacher instructional style, teachers' perceived priority of learning about computers and software, computer availability and access, availability of technical assistance when needed, and software quality were significant factors in teachers' decisions to use software in their instruction.

Surprisingly, source of computer learning, whether by staff development or learning on one's own, and lack of time to learn about computers and software were not factors that distinguished software users from non-users. Only 40% of teachers felt staff development met their needs to learn to use technology in the classroom. Likewise, only 19% of each group believed they had adequate time to learn about computers and new software. Anxiety was not an issue of concern. Only 12% of all teachers expressed anxiety when using computers and software.

Other good news is that teachers rated overall software quality as good for preparing students for the ONGPT math test. Ten of the top 12 software packages they used have correlation to NCTM, national and state standards, or to the mathematics learning objectives of the ONGPT (Table 3). The most frequently used program, Math Blaster (1997), was not among those, however. In fact, Soloway and Norris (1998) criticized Math Blaster for its gaming format. Students get to play a shoot-'em-up game having nothing to do with what they just learned as a reward for success. They proposed alternative software called Tenth Planet Explores Math (1996-2001). The fraction module, for example, provides multiple linked representations, and allows students to manipulate objects and to keep computer journals of their investigations. Problems are framed in real-world applications. The Tenth Planet fractions module makes much better use of the computational media than does Math Blaster, but the cost is four times as much for this thoughtful learning curriculum. Most software was used for remediation and drill and practice, but teachers clearly pointed out the need for drill and practice software for students who were failing the ONGPT math test because they lacked the skills tested on the exam.

The bad news is that schools may not be getting the total return on investment of technology dollars. As one teacher in this study stated, software use was a pleasant break for kids. Ten of 36 software titles on the survey were titles that districts said were available, but were not used during math class time, although they may have been used during other times (e.g., tutoring sessions during or after school or during intervention classes). In fairness to teachers, some abandoned using available software because of its weaknesses. Some software dated back to 1991, or could not run on available computers. For some, building wiring and software licensing issues prevented access to software they wanted to use.

Valuable educational software has stated learning objectives that are adhered to, is motivating to students, has well designed navigation icons, and contains multimedia features. It is also available above Version 1.0, which often has not been debugged and should be avoided (Abramson, 1998). Although overall software quality was rated good, findings revealed that software did not always have those valuable qualities, or the ability to foster higher order thinking skills.

Weaknesses in software's instructional and technical merit revealed that teachers want a management system that allows them to modify software to individualize instruction. The management system should keep track of student progress, identify areas of weakness and strength, suggest paths to improve, and have the ability to automatically adjust for student needs. The software should have more than one entry level, more than one level of difficulty, and provide some repetition to assist in retention. Software should allow students to save data, so that if they are not finished with a lesson, they do not have to begin again. Students should be able to change answers before software grades assessments.

Valuable software has an extensive database of problems, so that upon repeated use of software, students encounter a different set of problems. Problems should reference real-life applications and software should accommodate more than one solution method. Feedback should be tutorial in nature, not just indicate if responses were right or wrong. Help and audio features should be under user control. Teachers want software security against student errors or their intentional attempts to disrupt software operation. Software should meet NCTM, national and state standards, and learning objectives for the ONGPT. All of these criteria provide a set of guidelines to help districts in selecting software.

Clearly, much can be said about improving mathematics instruction using technology. Money continues to be an issue for many urban schools, particularly those that do not qualify for a lot of federal aid. One middle school principal stated their district needed computers, but fell in the middle between the haves and have-nots and must purchase their own technology. Although district and school budgets paid for the majority of software, teachers in 8 of 13 districts indicated some use of outside funding sources, including grants, parent associations, or Ohio School Net.

Schools cannot keep up with purchases of the latest software to run on older computers. Teachers cannot be expected to have contingency lesson plans for technology that fails when they need it. Therefore, they continue to use textbooks, overhead projectors, and chalk because they are reliable and flexible, as this study demonstrated. Let the buyer beware does not appear to be a sufficient answer when purchasing software, if the goal is to raise achievement of students with technology in Ohio's classrooms.

Abramson, G. W. (1998). How to evaluate educational software. Principal, 78(1), 60-61.

Desktop Tutor (Version 1.0) [Computer software]. (1996). Hollandays Publishing Corp., PO Box 161, Clayton, OH 45315.
http://www.hollandays.net/web/default.asp  [Note URL update July 23, 2007.]

Gillespie, C. (1999, September 24). Firm offers a trial Ohio proficiency test on Net. Cleveland, OH: The Plain Dealer, p. 5B.

Go Figure [Video and Computer software]. (1997). WVIZ Channel 25 and Ohio University Public TV, 4300 Brookpark Road, Cleveland, OH 44134.  [January 25, 2008 Update: The original Go Figure project has been expanded and is available from WOUB Center for Public Media, 9 South College Street, Athens, Ohio 45701. http://www.woub.org/gofigure/index.htm]. 

Hot Dog Stand: The Works [Computer software]. (1996). Sunburst Technology Corporation, Customer Service: 1550 Executive Drive, Elgin, IL 60123. http://store.sunburst.com/   [Note URL update July 23, 2007.]

Math Blaster [Computer software]. (1997). Sunburst Technology Corporation, Customer Service: 1550 Executive Drive, Elgin, IL 60123. http://store.sunburst.com/  [Note URL update July 23, 2007.]

Math Concepts and Skills SuccessMaker (Version 5.2) [Computer software]. (1999). Mesa, AZ: Pearson Education Technologies. http://www.pearsondigital.com/  [Note URL and product availability updated July 23, 2007.]

Math for the Real World [Computer software]. (1996). Knowledge Adventure, 2377 Crenshaw Blvd., Suite 302, Torrance CA 90501. http://knowledgeadventureschool.com/default.aspx [Note URL and product availability updated July 23, 2007.]

Math Munchers Deluxe [Computer software]. (1997). Riverdeep, Inc., 500 Redwood Blvd., Novato, CA. 94947. http://www.riverdeep.net/   [Note URL and product availability updated July 23, 2007.]

Mystery Math Island [Computer software]. (1995). Lawrence Productions, Inc., 1800 South 35^th Street, PO Box 458, Galesburg, MI 49053-0458. 

National Council of Teachers of Mathematics Curriculum and Evaluation Standards. (1989). Reston, VA: National Council of Teachers of Mathematics. Available: http://standards.nctm.org/  [URL updated on July 23, 2007.]

Ohio Department of Education. (1998). News release: Ninth-graders break overall, individual records on ninth-grade tests. http://www.ode.ohio.gov/www/ims/prof9_news_oct97.html
Updated January 26, 1998. Accessed July 22, 1999.  [Note at update July 23, 2007, URL no longer available.]

Ohio Department of Education. (2000a, February). The State of Education in Ohio.

Ohio Department of Education. (2000b, June). News Release: Class of 2000 Maintains Steady Performance on Ninth-Grade Proficiency Test. http://www.ode.state.oh.us/comm/news/06-05-00.htm Updated June 2000. Accessed June 6, 2000. [Note at update July 23, 2007, URL no longer available.]

Ohio Schools Technology Implementation Task Force report: Senator Robert R. Cupp, Chairman. (1999, March 3). pp. 1-35. http://www.ohioschoolnet.k12.oh.us/extlink/frameset.asp?exturl=/resources/document/documents/doclib_490.pdf&returl=/ Accessed July 8, 2001. Last accessed July 23, 2007.

O'Neil, J. (2000). Fads and fireflies: The difficulties of sustaining change. Educational Leadership, 57(7), 6-9.

OPT for Success (Version 2.0) [Computer software]. (1998).  Kamico Instructional Media, 4413 Spicewood Springs Road Austin, TX 78759-8592 . http://www.kamico.com Email: info@kamico.com [Note:  This product is now known as Path to Mastery.  Address and URL for availability was updated on July 23, 2007.]

Optimum (Version 1.0) [Computer software]. (1997). Photonics Graphics, 2244 Park Avenue, Cincinnati, OH 45206. http://www.optimumtest.com Email: photonics@photonicsgraphics.com

Passing the Ohio Ninth Grade Proficiency Test (Version 2.0) [Computer software]. (1997). Englefield & Arnold, Inc., 2121 Bethel Road, Suite D, Columbus, OH 43220. http://www.eapublishing.com

SkillsBank (Version 4) [Computer software]. (1998). Achievement Technologies, Inc. School Division: 10400 Little Patuxent Parkway, Suite 310, Columbia, MD 21044. http://www.achievementtech.com/    [Note URL and product availability update on July 23, 2007.]

Soloway, E., & Norris, C. (1998). Using technology to address old problems in new ways. Association for Computing Machinery, Communications of the ACM, 41(8), 11-18.

Tenth Planet Explores Math [Computer software]. (1996-2001). Sunburst Technology Corporation, Customer Service: 1550 Executive Drive, Elgin, IL 60123. http://store.sunburst.com/  Last accessed July 23, 2007.

Patricia Deubel holds a Ph.D. in Computing Technology in Education from Nova Southeastern University in Florida, a Masters degree in Curriculum and Instruction from Ashland University in Ohio, and a B.S. in mathematics from Ohio University. She has over 25 years of experience in mathematics and computer education teaching, teacher training and staff development, and curriculum development. She has presented computer workshops at the state and local levels and currently teaches mathematics at The Ohio State University in Mansfield. Her recent articles have been published in the Journal of Instruction Delivery Systems and HyperNexus: Journal of Hypermedia and Multimedia Studies. For further consultation regarding the outcomes of this study email: deubelp@neo.rr.com

back to top

*Follow-up note regarding Hot Dog Stand:  While in December 1999, Sunburst Technology Corporation (their new name) did not provide any correlations, correlations to national standards are now available at their web site, http://store.sunburst.com/,  according to B. Daly at Sunburst (email communication April 20, 2001).

Note (July 23, 2007): Some URLs since publication of the article have changed, and are reflected in the updates.  Likewise, some products are no longer available, or developers have created newer versions.   See our section on math software for product availability.