The School of Education is creating a measurement system that will track student experience and outcomes to provide insight into the value of an undergraduate education. Soon, other universities will be able to replicate and implement our approach on their own campus.
|
Is college worth it?
It’s a now-ubiquitous question that is becoming more complex as the global economy is changing, student debt is rising, and an increasingly diverse student body is matriculating to campus every fall. Ask a dozen people the question, and you’ll get a dozen answers, each fraught with anecdotes and anxieties, opinions and caveats. “Existing research is inadequate to truly address the question in a meaningful way,” said Richard Arum, dean and professor at the School of Education. “If we want to really identify the value of college, we need to design new instruments and collect better data.” |
A student studies in Langson Library on the campus of UCI.
|
UCI is serving as a pilot site to accomplish just that. This fall, work began on the Next Generation Undergraduate Success Measurement Project. Arum, the study’s principal investigator, and a team are in the process of developing and implementing ways to improve our understanding of the value of undergraduate educational experiences, and promote evidence-based models of undergraduate student success.
The study, supported by the Andrew W. Mellon Foundation, tracks for two years a random sample of 500 UCI freshmen, 250 junior transfer students, 250 continuing juniors, and 50 freshmen honor students. Everything from transcripts to online classroom behavior, living situations to student moods will be considered.
“There’s been a huge change in technology that makes it a lot easier to capture data, regardless of industry,” Arum said. “Companies are hungry for data, and use data to improve consumer experiences; we’ve seen this trend in healthcare, entertainment, and it’s time for it to come to higher education. It’s possible to figure out what students are doing and make use of that data.
“Frankly, it’s irresponsible not to.”
The study, supported by the Andrew W. Mellon Foundation, tracks for two years a random sample of 500 UCI freshmen, 250 junior transfer students, 250 continuing juniors, and 50 freshmen honor students. Everything from transcripts to online classroom behavior, living situations to student moods will be considered.
“There’s been a huge change in technology that makes it a lot easier to capture data, regardless of industry,” Arum said. “Companies are hungry for data, and use data to improve consumer experiences; we’ve seen this trend in healthcare, entertainment, and it’s time for it to come to higher education. It’s possible to figure out what students are doing and make use of that data.
“Frankly, it’s irresponsible not to.”
|
Capturing the Data
Data on the 1,050 UCI students will be collected in three different areas, or strands. Each strand is overseen by a School of Education faculty member. The data collected from each strand will work in concert to paint a complete portrait of undergraduate student experience. The first strand is collecting data on social background, secondary academic preparation, and collegiate course performance. This strand is led by Assistant Professor Rachel Baker and Michael Dennin, dean of the Division of Undergraduate Education and vice provost for Teaching and Learning. “A lot of this data has existed for a very long time, but we haven’t been using it to improve student experiences or outcomes,” Baker said. |
Richard Arum, dean and professor, UCI School of Education and principal investigator, welcomes students at the project's official launch on September 25, 2019.
|
Data in this strand will help the research team better understand how a student moves through the university – what classes they take, in what sequence, and the diversity of their peers in the classes they take.
The second strand examines student clickstream activity in the Canvas Learning Management System, the most widely used platform for UCI courses. The strand is led by Professor Mark Warschauer, who is also director of UCI’s Digital Learning Lab, and has previously studied online learning systems.
“A lot of clickstream research looks at particular classes, and we know that frequency of posts, or how spread out interactions are can be predictors of student performance,” Warschauer said. “What’s been done much less is looking at clickstream behavior across courses, together with other kinds of administrative and survey data to broadly understand student experience.”
The third and final strand utilizes innovative survey and experiential sampling methods to capture student experiences. Surveys that measure psychological functioning and assessments that measure cognitive performance will be administered at the beginning and end of the two years.
A subgroup of the population will also receive random text messages over the course of two years. The messages will ask questions to capture students’ emotional state at any given moment. Questions can include where the students are, what they’re doing, how they’re feeling, and their interest level in the activity. Students will also receive a “daily diary” prompt where they’ll be asked to share their experiences over the past day or week.
The second strand examines student clickstream activity in the Canvas Learning Management System, the most widely used platform for UCI courses. The strand is led by Professor Mark Warschauer, who is also director of UCI’s Digital Learning Lab, and has previously studied online learning systems.
“A lot of clickstream research looks at particular classes, and we know that frequency of posts, or how spread out interactions are can be predictors of student performance,” Warschauer said. “What’s been done much less is looking at clickstream behavior across courses, together with other kinds of administrative and survey data to broadly understand student experience.”
The third and final strand utilizes innovative survey and experiential sampling methods to capture student experiences. Surveys that measure psychological functioning and assessments that measure cognitive performance will be administered at the beginning and end of the two years.
A subgroup of the population will also receive random text messages over the course of two years. The messages will ask questions to capture students’ emotional state at any given moment. Questions can include where the students are, what they’re doing, how they’re feeling, and their interest level in the activity. Students will also receive a “daily diary” prompt where they’ll be asked to share their experiences over the past day or week.
|
The third strand is being overseen by Arum and Distinguished Professor Jacquelynne Eccles.
“We will be able to be descriptive as to what college looks like – what percentage of kids are doing a certain activity, and what percentage of students are feeling a certain emotion,” Eccles said. “We can then look at it from an individual level, and begin to see pathways through college. For example - we can analyze who ends up developing mental problems, who chooses certain majors, who changes majors, who drops out.” It’s no secret that any one source of data can have its limitations. Combining three different data types, however, can reveal new patterns and findings in areas that have been both examined and unexplored. “It’s a mistake to think that it’s all about academics in higher education, and that everything that’s important happens in a classroom,” Arum said. “Students spend most of their time socializing, in extracurricular activities, or at work, and an increasing amount of classwork is being done online. This study considers all those factors.” |
A student moves into her dorm room. The project is tracking students in and out of the classroom.
|
Spreading the Word
A main goal of the study is to not only understand the student experience at UCI, but also create tools that other universities can replicate to assess the undergraduate experience on their own campus.
The findings from the UCI study will inform the development of a large-scale longitudinal study of college and universities coordinated by the Interuniversity Consortium for Political and Social Research (ICPSR) at the University of Michigan. The ICPSR stores, curates, and provides access to scientific data for others to use and validate research. Approximately 776 universities, government agencies, and other institutions are members of ICPSR.
“Our goal is to establish a set of tools and practices that others can then apply in their own settings to improve practice,” Arum said. “These efforts are a way to make visible student experiences, perspectives and realities, which are not always obvious to faculty or administrators.”
The study emerged out of the Mellon Foundation’s Value of the Liberal Arts initiative, which identified the need for deeper, more holistic and authentic measurements of student experience and outcomes, and for the findings to be utilized nationally.
“Colleges and universities face growing pressure to prove their value to their students and society at large,” said Mariët Westermann, Mellon Foundation executive vice president for programs and research. “Developing thoughtful and robust models and measures of the economic, social, and personal outcomes of a liberal arts education will greatly help all of us understand better what the worth of such an education is, and communicate that value to academic decision makers and the public.”
A main goal of the study is to not only understand the student experience at UCI, but also create tools that other universities can replicate to assess the undergraduate experience on their own campus.
The findings from the UCI study will inform the development of a large-scale longitudinal study of college and universities coordinated by the Interuniversity Consortium for Political and Social Research (ICPSR) at the University of Michigan. The ICPSR stores, curates, and provides access to scientific data for others to use and validate research. Approximately 776 universities, government agencies, and other institutions are members of ICPSR.
“Our goal is to establish a set of tools and practices that others can then apply in their own settings to improve practice,” Arum said. “These efforts are a way to make visible student experiences, perspectives and realities, which are not always obvious to faculty or administrators.”
The study emerged out of the Mellon Foundation’s Value of the Liberal Arts initiative, which identified the need for deeper, more holistic and authentic measurements of student experience and outcomes, and for the findings to be utilized nationally.
“Colleges and universities face growing pressure to prove their value to their students and society at large,” said Mariët Westermann, Mellon Foundation executive vice president for programs and research. “Developing thoughtful and robust models and measures of the economic, social, and personal outcomes of a liberal arts education will greatly help all of us understand better what the worth of such an education is, and communicate that value to academic decision makers and the public.”
|
Leading the Conversation
Arum called UCI the perfect setting for the study, as the university believes in a data-driven approach and in the power of interdisciplinary research, science and measurement. Additionally, he said, the university is committed to serving its students. “Those conditions don’t exist in higher education as broadly as one might hope." Some of the study’s research will be conducted in UCI’s new Anteater Learning Pavilion, a 65,000 square foot facility that brings active learning and 21st century educational opportunities to students across the campus. The facility is home to two lecture halls, 10 classrooms, meeting rooms, computer labs and other collaborative spaces. In those collaborative spaces, Arum explained, the School of Education can work alongside the Division of Undergraduate Education, and bring in national and international visitors who are interested in the study and in seeing how UCI supports and enriches the academic experience of undergraduates. |
The newly established Anteater Learning Pavilion on the campus of UCI, where some of the study's research will be conducted.
|
“We’re leading conversations on how to better serve undergraduate students, and how leadership can think of institutional improvements that are driven by data and measurement,” Arum said. “We’re uniquely positioned to do this work.”
For more information on the Next Generation Undergraduate Success Measurement Project and its research team, please click here.
For more information on the Next Generation Undergraduate Success Measurement Project and its research team, please click here.
