The 2001 National Assessment of Educational Progress in Geography
By Joseph P. Stoltman
September 2002

The National Assessment of Educational Progress (NAEP) in geography is a periodic survey of geographic knowledge and skill of students at grades four, eight, and twelve.  NAEP is administered by the National Center for Educational Statistics (NCES) in the United States Department of Education's Office of Educational Research and Improvement (OERI). The
first NAEP in geography was conducted in 1994 and the second in 2001. Like the first assessment, the second assessment probed students' ability to recall, understand, analyze, and interpret geographic information. Students were also required to apply geography content to the completion of various practical tasks. 

Educators and researchers can evaluate progress in geography education by comparing the 2001 data with that from the 1994 NAEP in geography.  Such information is important for several reasons, two of which are paramount.  First, by comparing the 1994 and 2001 results, educators and researchers can evaluate the changes in geography education between the first and second administration of the assessment.  This longitudinal information is important for both teachers and policy makers
as they evaluate the effects of school reform on student progress in geography.  Second, the 2001 administration is a snapshot of performance in geography at a single point in time by a national sample of students in grades four, eight, and twelve.  This information will enable educators and researchers to evaluate whether or not children in the United States are developing the geographic skills and knowledge essential for effective participation in the economic and political activities of the nation. 

This Digest discusses (1) the framework of the 2001 NAEP in geography, (2) the findings, (3) comparisons of the 1994 and 2001 national assessments in geography, and (4) conclusions about the significance and usefulness of the national assessment for the teaching of geography.

THE FRAMEWORK

A single comprehensive framework guided the structure of the geography assessment in both 1994 and 2001.  Three sub-content areas of geography constituted the framework.  First, space and place: knowledge of geography as it relates to particular places on Earth, to spatial patterns on Earth's surface, and to physical and human patterns that shape such
spatial patterns.  Second, environment and society.  Third, spatial dynamics and connections: knowledge of geography as it relates to spatial connections among people, places, and regions. 
The nature of the framework necessitated that the assessment include both multiple-choice questions and constructed-response questions for which the students wrote their own responses.  Assessment items measured three cognitive categories: (1) knowing; (2) understanding; and (3) applying.  The first category of questions asked students to observe and recall information.  The second category asked students to attribute meaning to an observation and to explain events.  The third category asked students to hypothesize, use reasons, and solve problems. 

Fourth grade students were asked more knowing types of questions; the assessment included more applying questions in the eighth grade version, and even more in the twelfth grade version.  In most respects the assessment required students to apply knowledge to higher-level cognitive operations rather than to recall information.  The constructed response questions
challenged students to write answers ranging in length from a few words or sentences to several paragraphs. 

REPORT OF STUDENT PERFORMANCE

Results for each grade -- fourth, eighth, and twelfth -- are reported according to three achievement levels: Basic, Proficient, and Advanced. These achievement levels were determined by expert judgments about what students should know and be able to do in geography at each of the grade levels.  The Basic level indicates partial mastery of knowledge and skills fundamental for proficient work at each grade.  A score of Basic means additional knowledge and skills are necessary for competent performance in geography.  The Proficient level represents solid academic performance and competencies in working with challenging subject matter.  The Advanced level signifies superior performance, demonstrating excellence in knowing
and using geography.  Two percent of fourth graders, 4 percent of eighth graders, and 1 percent of twelfth graders achieved the Advanced level. Nineteen percent of fourth graders, 26 percent of eighth graders, and 23 percent of twelfth graders attained the Proficient level.  Fifty-three percent of fourth graders, 44 percent of eighth graders, and 47 percent of
twelfth graders reached the Basic level.  Twenty-six percent of fourth graders, 26 percent of eighth graders, and 29 percent of twelfth-graders scored below the Basic level. 

The assessment revealed descriptive as well as statistically significant differences on performance between major subgroups of the population.  For example, males at each grade scored higher on average than female students.  At grades four and eight, white and Asian/Pacific Islander students scored higher on average than black, Hispanic, and Native American students.  White, Asian/Pacific Islander, and Native American students scored higher in the twelfth grade than black and
Hispanic students.  In grades eight and twelve, the more education that students reported their parents had attained, the higher the student performance on the assessment.  At all three grades, students attending non-public schools performed at a higher level than did students attending public schools.  Students attending central city schools scored lower than students in urban fringe areas, large towns, rural areas, or small towns. Students eligible for free or reduced price lunch (meeting poverty
guidelines) scored lower on average than ineligible students. 

COMPARISON OF THE 1994 AND 2001 GEOGRAPHY ASSESSMENTS

The overall geography scores for students at the fourth and eighth grades were higher in 2001 than in 1994. The difference in performances of twelfth grade students in 2001 and 1994 was statistically insignificant. At both fourth and eighth grades, the improvements in performance occurred among the lowest performing students.  The proportion of students who performed at or above the Basic level increased within the fourth and eighth grades between 1994 and 2001.  There were no statistically
significant changes from 1994 to 2001 in the percentage of students at any grade at or above the Proficient level. 

In addition to collecting data about student performance, NAEP also collected information about the classroom context of teaching and learning.  A higher percentage of fourth grade teachers indicated that they were very prepared to teach geography than in 1994.  Forty-four percent of eighth grade teachers reported that they were very prepared to teach geography.  

Instructional time was greater in 2001 than in 1994 for the following geography topics:  map and globe studies at the eighth grade; the study of natural resources at the eighth and twelfth grades; and countries and cultures at the eighth grade.   

The amount of classroom instruction as suggested by geography courses taken revealed the following patterns: a higher percentage of eighth grade students reported studying geography in grades six, seven, and eight in 2001 than in 1994, and eighth grade students who took more geography had higher average scores than those who took it for fewer years.  The
percentage of twelfth grade students reporting geography courses in each of the high school years was greater in 2001 than in 1994, but students who took one year or less of geography scored higher on average than those who took three or four years of geography.

Students at fourth, eighth, and twelfth grades who used Internet or CD materials to a small or moderate extent scored higher than students who did not use those tools. 

The 2001 geography test also included special needs students and reported no significant differences at the fourth and twelfth grade levels when special accommodations were provided for administering the test.

CONCLUSIONS

While student performance in geography since 1994 has generally improved, a large proportion of students in 2001 did not reach either the Basic or Proficient levels and did not demonstrate achievement in the essential content and skills in geography judged necessary for responsible citizenship.  Although more teachers believe they are very prepared to teach geography, and students at the eighth and twelfth grades are taking more geography courses, performance remains low. 

The review of the released items suggests that many students in the early grades do not know basic information such as the name and location of the state where they live.  Similarly, students at grades eight and twelve do somewhat better with definitional information, but a large proportion were unable to analyze the information related to an environmental issue, provide reasons for or consequences emanating from the issue, or suggest a possible solution to the geographic issue or
problem.  Items that require those steps are rigorous and challenging.  To answer them successfully, students must regularly have opportunities to apply content and skills to issues of geographic significance. 

Also, the alignment between the NAEP Geography Framework and the National Content Standards in Geography is vague.  Geography experts can conceptualize the linkages, but they are not readily apparent to many people.  The national content standards influence instruction, student materials, and teacher preparation and professional development.  The
alignment between the two frameworks should be improved to enhance the usefulness of both for teachers, students, and parents.  

SOURCES OF INFORMATION ABOUT THE NAEP IN GEOGRAPHY

Information in this Digest is from THE NATION'S REPORT CARD: GEOGRAPHY 2001.  To order this publication or any other NAEP-related products, contact Education Publications Center (ED Pubs), U.S. Department of Education, P.O. Box 1398, Jessup, MD 20794-1398; toll-free 877-433-7827; FAX 301-470-1244.  This publication is also available from
the ERIC Document Reproduction Service (EDRS), 7420 Fullerton Road, Suite 110, Springfield, VA 22153-2852; toll-free 800-443-3742.     

The NAEP Web site contains information about the NAEP in geography and general information about assessment, publications, and analysis tools used by various NAEP projects <http://nces.ed.gov/nationsreportcard>.

Joseph P. Stoltman is Professor of Geography at Western Michigan University.