Open-ended survey responses contain a rich source of insights about students' needs and sentiments. However, without the right tools or methodology, reading through hundreds, if not thousands of student comments can quickly become a daunting task for even the largest research and planning offices. But it does not have to be this way.
By applying natural language processing (NLP) techniques to response analysis, institutions can save significant amounts of time while getting valuable insights from students' comments. Read on to learn about the four key benefits of NLP and how you can apply them to your institution.
Benefit #1. NLP is much more than just word clouds
Data visualization helps us tell a meaningful story from our data. The word cloud like the one above, is a common way to quickly visualize keywords from open-ended comments. The word cloud generates a visual representation of keywords by scaling the size of each word relative to the frequency of words. Using over 1,200 online course reviews from Coursera, we can see that “professor”, “course”, and “often” appear to be the most frequent, but how are they related to each other? It is difficult to interpret this without more context.
One better approach is to look at the frequently appearing words and their relationships to other words. In the graph below, we use the same open-ended comments to create a bidirectional network graph (word pairings). Now we can see more clearly how the words come together to reveal a much clearer story -- overwhelmingly positive reviews from students!
Benefit #2. Minimize human bias and get campus buy-in
Manually reading and categorizing student feedback can create inherent biases based on the person assigned to the task. Particularly for surveys relating to strategic planning or campus climate, confirmation bias may cause the individuals sorting to miss or misunderstand important information.
Some institutions have adopted a collaborative approach to mitigate this bias by assigning two or more staff to review the same comments. However, this approach can take twice as long, because any differences must be manually reconciled. It also takes additional time to train staff to recognize the institutional specific themes.
Text analysis can help reduce human bias when grouping comments into themes. Modern text models are built on over 3 billion English words.
Benefit #3. Operational Efficiency
Text analysis helps increase operational efficiency, especially when your team is short-staffed or faced with a tight turnaround. According to our recent survey of institutional research and assessment professionals across Canada and the U.S., we found that 77 percent of respondents manually read and tag their survey comments. They have described this process as “time-consuming”, “tedious”, and “labor-intensive.
In follow-up interviews with the respondents, many also mentioned that it took them about two weeks to analyze students’ comments with a team of three to four. Furthermore, the tools they were using for qualitative research took more time than the manual approach, because they require a lot of time to setup and code the data. With NLP, analysts would be able to spend more time interpreting the results and developing action plans.
Benefit 4. Understanding unique student characteristics
Recognizing distinct student personas can also help institutions appreciate the diverse needs of students and be more inclusive in its planning, which is another powerful benefit of NLP. For instance, when we analyzed a recent climate survey for a U.S. higher education institution, we found that students of colour placed the most emphasis on policies that promote open discussions while protecting privacy.
The graph below is one way to visualize the themes identified from text analysis. It shows that a BIPOC student (y-axis) is more likely to raise concerns around racial issues (35% vs. 22%) and college access and affordability (8% vs. 5%). Meanwhile, a non-BIPOC student is more likely to raise concerns around political perspectives (35% vs.40%) and accessibility (7% vs. 12%).
Another example where NLP was used to identify distinct student groups is when we examined the concerns from over 400 undergraduate students in the U.S. during the start of COVID-19. From this research, we identified six personas with unique needs and demands. The large orange circle in the graph below, for example, represents a cluster of students who are mainly seeking academic services, and blue dots inside represent other services the same group of students are looking for (e.g. better communication, academic counselling, tutoring, and e-learning resources). The large blue circle below represents another group of students who are more concerned about emergency services such as financial aid, housing, and food. Our research showed that this group is more likely to seek refunds than other groups.
Case Study: Increasing Student Retention
For many institutions, increasing student retention is ultimately the reason why we want to better understand student concerns. Bastyr University, one of the world’s leading naturopathic medical schools, made it their strategic goal to maintain a high level of student satisfaction above 90 percent. To help the university achieve this goal, Kai Analytics analyzed their students’ responses to the following question,
"Bastyr University strives to continually improve your student experience.
Please provide any ideas or suggestions you may have to improve the University."
Amidst the hundreds of great suggestions, text analysis was used to narrow down the comments to the main areas for improvement. One example was discovering that students were having a lot of issues with printers in one of the labs. Upon sharing the results with the leadership team, the printers were promptly replaced. The analysis eventually led to an email from the president to all students on campus informing them that their feedback was heard, and improvements were being made. In addition to the printers being replaced, Bastyr University committed to investing in more study spaces as well as upgrading their eLearning platform.
The results presented themselves the following year when the survey was re-administered. Survey response rates increased from 43 percent to 59 percent, approval rates stayed above 90 percent and retention improved by 4 percent. It is estimated that the 4 percent improvement in retention leads to an estimated $259,000 in tuition revenue saved.
Applying your knowledge
Institutions that can successfully leverage the benefits of NLP will be able to enhance students’ virtual learning experience and better serve the needs of diverse groups of students to support their academic success. Furthermore, through quick and efficient analysis of students’ comments, institutions will be able to quickly respond to their students and maintain high levels of satisfaction throughout the academic year.
Need a bit more help? Fill out the form below or reach out to us at info@kaianalytics.com.