NSF project detecting whether students are grasping concepts in the classroom

[image:11922 class:fright]One of the challenges for any instructor is telling whether their students are getting the concepts being presented in class. In many cases, by the time it is clear that they collectively or individually don’t understand the material; it is too late to make a difference.

To address the issue, University of Illinois engineering professors Geoffrey Herman and Joshua Peschel are developing techniques to assess students' representational fluency. They are exploring the similarities and distinctions between experts (those who firmly understand) and novices (those who are in the infant stages of understanding) by studying their use and production of sketches during problem solving for engineering courses. Through a $250,000 National Science Foundation (NSF) Research in Engineering Education (REE) grant, they will build on existing techniques for digital sketch recognition, focusing on two domains – digital logic and truss design.

“As students are developing their solutions to in-class homework problems, whether through the creation of a sketch or deriving equations, we strive to assess at what point they begin to develop expertise,” Peschel explains. “In other words, we are looking to identify at what point we can determine if they’ve mastered a particular concept or understand some part of the engineering process, whether analysis or design.”

The study is intertwined with one on Sketch-Based Learning, in which Peschel is a co-principal investigator with Civil and Environmental Engineering professors Megan Konar and Cassandra Rutherford and which is partially funded through the College of Engineering’s Strategic Instructional Initiatives Program (SIIP). Peschel and Herman are also PIs with Emma Mercier from the College of Education on a related NSF Cyberlearning Future Learning Technologies grant, which is looking at the use of these types of tools to support collaborative problem solving.

“I think it’s absolutely necessary to approach this through an educational psychology and curriculum design perspective,” Peschel said. “We couldn’t do it, however, without the computation. They go hand-in-hand.”

In differentiating this project from the two related ones, Peschel said, “This one is targeted at the individual student level, trying to get inside their mind and see through their drawings in order to make some inferences. Its results will ultimately allow us to build better teaching tools and testing methods.”

 “We also are interested in how early there are indications that students are mastering the concept,” he adds. “If we can tease that out of a student simply by noticing a certain kind of mistake, then we intervene so that the student can eventually master it or have a deeper understanding.”

At the onset, Peschel and Herman are exploring expert and novice graphical communication through digital sketching. They’re going to data mine student sketches and compare those to interviews in which students reveal whether they understand a concept and then try to figure out how the drawing that a student creates when they grasp the concept differs from the creation process when the student had not yet fully understood the concept. This will help in establishing a foundation for developing a theory about how novices and experts use sketching during engineering problem solving.

The team wants to address fundamental questions. What are the differences between domain concept experts and novices and their use of sketches in problem solving?  What are the differences in the process of using sketches during problem solving? Can it be used as a predictive tool to begin with and if so, how does that translate to the classroom?

Beginning with ECE 120 (Introduction to Computer Engineering) and TAM 210 (Statics), they collecting the sketch and problem-solving data, analyzing it and trying to develop metrics.

“TAM is a gateway to whatever students choose to do in engineering,” Peschel said. “These courses are critical points in determining whether they continue in engineering as well as providing them with the foundation in knowledge, skill and abilities to do well later on.”

Through the team's findings, learning could be more individualized and tailored to a struggling student, while someone who is already mastering a concept might do even better. 

If Peschel and Herman are successful in determining whether this kind of evaluation is possible, they plan a future proposal for broader scale implantation, which would include higher level courses within in engineering, but also relevant instruction in other disciplines.

“If we can show that we can tease out these patterns and anomalies through sketching, that may also be translatable to other domains, learning to write in a foreign language for example, or any subject where there is sketching,"Peschel said.

Peschel’s motivation is both personal and professional. While he hopes to be at the forefront in advancing sketch understanding, he understands first-hand the frustration students face early in their engineering education.

“I was one those students who struggled early on,” he admitted. “Sometimes there is big disconnect between the math part and the picture part.  This is a way that we can start with the pictures as a better basis of understanding. Fortunately I was able to make it and succeed where I am today, but others aren’t as fortunate and transfer out of engineering. So this is personal for me; our goal is to make sure fewer students fall through the cracks.”