Harnessing AI for Identifying Instructional Interventions

By Sydney Brown, PhD., Assistant Director, Center for Transformative Teaching

This article shows one way to collaborate with AI to address instructional challenges. It includes the components of an effective AI prompt along with an example of an exchange with an AI (Claude 3 Sonnet). Additionally, the same prompts are used with four different AIs to compare and contrast responses. AI prompts are denoted by the label “Prompt:” along with indented, bold-faced italicized type. AI responses are in italics are labeled with "Response:" and indented, italicized type.

The five components of a prompt

There are several aspects of getting the most out of AI, but the first step to getting useful output is learning to craft a good prompt. In my experience so far, there are five basic components.

1. Perspective or role
2. Task description or objective
3. Relevant context
4. Specific instructions or considerations
5. Output specifications.

Addressing each of these five items forces me to analyze what I’m trying to accomplish, and I often discover missing aspects in my thought process. I like to think we train each other as we collaborate. Were I using AI in the classroom, I might consider having students discuss or even create rationales for the components of their prompts.

A step-by-step example

I usually begin by giving the AI a perspective or role to help it reference knowledge domains. Of course, this represents what expertise I think is needed to answer the question. For someone who is a novice in a particular domain, giving a role or perspective might be more challenging. A class discussion about what perspectives are useful and how perspectives might shape or bias a response might be of interest if students are learning to use AI.

Anatomy of the Initial Prompt

Prompt: "You are an instructional designer who specializes in practical applications of learning analytics in higher education."

I then describe the task and give some context,

Prompt: "I am going to give you scenarios about what I might see in a learning analytics dashboard, and I would like you to act as a consultant, giving me some possible reasons for what I’m seeing, then follow that with at least three and at most five instructional interventions or teaching strategies to improve learning. Also, explain why they might be effective. Ask me follow-up questions if you need more information to present actionable suggestions."

and specify output.

Prompt: "Please format your reply as a narrative with appropriate headings."

I’m now ready to collaborate. Additionally, by letting the AI know I am going to give one or more scenarios, I won’t have to repeat my initial prompt – at least for a while. If given many scenarios, or if the follow-up conversation is extensive, it’s possible that it may drift and may need to be reminded of what I was working on and how I wanted it to interact with me.

Prompt: "Here is the first scenario: There is a gender gap in a large enrollment introductory math class with men significantly outscoring women on exams despite their relatively similar scores on homework."

The AI will probably give a summary of each of the interventions that is too broad to be immediately useful. Follow-up prompts can help focus the AI on your goal. You may also find yourself finetuning your request and goal. For example, I felt my initial scenario was fairly specific, but after reading the response, I realized there were a few ways to go. Am I looking to do something to address the issue right away? Or would I rather wait and dig deeper into the research around different suggestions? Probably both, at least eventually, but to get the most out of my exchange with the AI, I’ll want to tailor my follow-up prompts to focus on a more specific goal. Two examples follow.

Follow-up Prompts

See Appendix C for AI output for these two prompts.

Follow-up prompt 1: Immediate and long-term remedies

Prompt: From these broad strategies, please give me a specific technique I could easily implement within the next week, something a little more complex that I could add yet this semester, and finally a larger change that I could incorporate in my course design and teaching for the next time I teach this class. Do you need any more information to provide me with useful strategies to implement in the near, medium, and longer time frames?

Another might be along the lines of effectiveness.

Follow-up prompt 2: Focus on evidence of impact

Prompt: From these broad strategies, please give me more detail about the two of them with the most empirical research support. Include references if available.

Comparing AI Responses

AIs will differ in their responses. I used the following prompt with four widely accessible generative AI’s: Google’s Gemini, Microsoft’s CoPilot, Open AI’s ChatGPT, or Anthropic’s Claude.

Prompt:

You are an instructional designer who specializes in practical applications of learning analytics in higher education.

I am going to give you scenarios about what I might see in a learning analytics dashboard, and I would like you to act as a consultant, giving me some possible reasons for what I’m seeing, then follow that with at least three and at most five instructional interventions or teaching strategies to improve learning. Also, explain why they might be effective.

Ask me follow-up questions if you need more information to present actionable suggestions.

Please format your reply as a narrative with headings. Begin with a succinct summary of the problem, follow that with relevant theory, and conclude with 1 to 5 interventions, or teaching strategies, instructors could consider.

Here is the first scenario: There is a gender gap in a large enrollment introductory math class with men significantly outscoring women on exams despite their relatively similar scores on homework.

The following chart summarizes the different reasons and interventions the four AIs identified. I have included the full text of their responses at the end of this article. Three of the four specifically named stereotype threat, and the fourth, Gemini, should have because it named ‘test anxiety’ instead. However, there is no reason for test anxiety is to afflict women more than men unless something like stereotype threat, lack of representation, or instructor bias is present – none of which Gemini identified.

This may be a subtle indicator of AI training to avoid saying anything that could be construed as ‘woke’ bias. In February 2024, several widely circulated instances where an attempt by Google to correct its AI for bias resulted in apparent over-correction as evidenced by the generation of absurd images such as an Asian woman and black man depicted as WWII German soldiers or stating “[there’s] no right or wrong answer” as to whether Elon Musk’s memes on X were worse than Hitler’s murder of millions (Kleinman, 2024).

In any case, Gemini’s omission of stereotype threat, a well-researched construct (Nguyen & Ryan, 2008), as well as avoidance of “lack of representation” seems to evidence that attempts have been made to avoid terms that could be construed as liberal or progressive bias. This is further underscored by Gemini’s interventions of “growth mindset” and “female role models.”

Most articles about using AI underscore the need to be aware of bias, but few highlight how subtle that might be or what disciplinary expertise may be required to discern it. This offers an opportunity for student instruction.

Table 1: A comparison of four different AI responses to a single prompt  

By looking at the table, we can see considerable overlap on a few items and in this specific example, the omission or inclusion of a few reasons for the disparity or interventions to address it does not undermine my attempt to gain pedagogical insights because more than one AI was queried.

How AI can and cannot help with pedagogical challenges

AI can be helpful in gaining a better understanding of possible reasons for student behaviors and performance. It can familiarize us with the language of a domain or that surrounds a construct or phenomenon so that we can ask better questions. With the use of personas, it can give us insight into the perspectives of different kinds of instructors and students.

In my own use as an instructional designer, AI often provides a wider variety of possible interventions than I might identify in any particular moment. In this way, I feel it helps keep me from falling into reflexive responses and nudges me away from my own biases. However, AI does not engage in reflection, does not have “institutional memory” and specific knowledge of the many factors particular to the situation that may impact the implementation and effectiveness of a solution. Consequently, I see AI as a great way to start exploring a phenomenon and to equip oneself with introductory knowledge and language that can then be shared with a domain expert to jumpstart conversations. In this particular example, sharing what was learned using the AI would help the instructional designer to more efficiently ask clarifying questions to reveal the unique aspects of an instructor’s context and to identify which interventions might be the most productive for them and their students.

References

Kleinman, Z. (2024, February 28). Why Google’s “woke” AI problem won’t be an easy fix. BBC. https://www.bbc.com/news/technology-68412620

Nguyen, H.-H. D., & Ryan, A. M. (2008). Does stereotype threat affect test performance of minorities and women? A meta-analysis of experimental evidence. Journal of Applied Psychology, 93(6), 1314–1334. https://doi.org/10.1037/a0012702

Appendices

Appendix A: How AI was used in the writing process of this article

Besides developing the example prompts, AI was used to give feedback on the first two drafts. Prompts such as the following helped identify questions the target audience might have.

Prompt: "You are an instructor of undergraduate students at a public university. What questions do you have after reading the following article: [article text]"

Upon completion of the article, three AI personas representing more specific types of instructors, were used to zero in on ways it could be more useful for that type of audience member. For example, “John, the Academic Traditionalist” recommended more use of research support and all three personas mentioned case studies.

Overall, the AI feedback was quite helpful in improving the article before sending it to colleagues for their feedback.

Appendix B: AI Responses to the Initial Prompt

In this section, both the prompt and the responses of each of the four AI’s used are included. 

Prompt: You are an instructional designer who specializes in practical applications of learning analytics in higher education. 

I am going to give you scenarios about what I might see in a learning analytics dashboard, and I would like you to act as a consultant, giving me some possible reasons for what I’m seeing, then follow that with at least three and at most five instructional interventions or teaching strategies to improve learning. Also, explain why they might be effective. 

Ask me follow-up questions if you need more information to present actionable suggestions. 

Please format your reply as a narrative with headings. 

Here is the first scenario: There is a gender gap in a large enrollment introductory math class with men significantly outscoring women on exams despite their relatively similar scores on homework.

CLAUDE 3 Sonnet 

Gender Gap in Introductory Math Class

Analysis of Data Patterns

The gender gap in exam scores for the large enrollment introductory math class is a concerning pattern that warrants further investigation. Despite both men and women performing similarly on homework assignments, the discrepancy in exam scores suggests potential underlying factors that may be hindering the performance of female students during high-stakes assessments.

Possible reasons for this gap could include:

1. Stereotype threat: Female students may experience heightened anxiety or negative self-perception during exams due to societal stereotypes about their math abilities, adversely affecting their performance.
2. Learning preferences: The instructional methods employed in the class may align better with the learning preferences of male students, potentially creating a disadvantage for female learners.
3. Lack of representation: A lack of female role models or mentors in the math field could contribute to a sense of alienation or reduced self-efficacy among female students, impacting their exam performance.

Instructional Interventions and Teaching Strategies

To address the gender gap and improve learning outcomes for all students, the following interventions and teaching strategies could be implemented:

1. Growth Mindset Interventions: Incorporate activities and messaging that promote a growth mindset, emphasizing that intelligence and math abilities are not fixed but can be developed through effort and practice. This can help combat negative stereotypes and boost self-confidence, particularly for female students.
2. Inclusive Instructional Design: Evaluate and modify the course materials, teaching methods, and assessments to ensure they are inclusive and cater to diverse learning styles and preferences. This could involve incorporating more collaborative learning activities, varied assessment formats, and real-world applications that resonate with a broader range of students.
3. Peer Mentoring and Role Models: Implement a peer mentoring program that pairs female students with successful upperclassmen or graduate students in math-related fields. Exposure to relatable role models can increase a sense of belonging and self-efficacy among female learners.
4. Formative Assessments and Feedback Loops: Incorporate frequent low-stakes formative assessments and feedback loops throughout the course. This can help identify and address learning gaps early on, providing opportunities for targeted interventions and support before high-stakes exams.
5. Inclusive Learning Communities: Foster inclusive learning communities within the class by encouraging collaboration, group discussions, and study groups. Creating a supportive and inclusive environment can reduce feelings of isolation and promote a sense of belonging for all students, including those from underrepresented groups.

By implementing these interventions and teaching strategies, the institution can work towards closing the gender gap in exam performance and create a more equitable and supportive learning environment for all students in the introductory math class.

Google’s Gemini

Gender Gap in Introductory Math: Analysis and Interventions

Possible Reasons for the Gap

The observed gender gap in your introductory math class, with men outscoring women on exams despite similar homework scores, points to a potential performance difference under pressure. Here are some reasons to consider:

• Test Anxiety: Women might experience higher test anxiety compared to men, impacting their performance on exams.
• Self-Efficacy Beliefs: Stereotypes about math ability can lead women to doubt their skills, hindering performance during exams.
• Study Strategies: Men might be utilizing different study strategies that are more effective for retaining knowledge needed for exams (e.g., practice problems, group study focused on exam prep).

Instructional Interventions

To address this gap, we can implement several strategies:

1. Incorporate Low-Stakes Assessments:
   • Integrate frequent, low-stakes quizzes or in-class activities that mirror exam questions.
   • This allows students to practice applying knowledge in a less stressful environment, building confidence and identifying areas needing improvement.
2. Targeted Test-Anxiety Workshops:
   • Offer workshops or online modules focused on managing test anxiety.
   • These can equip students with relaxation techniques, time management skills, and strategies for approaching exams calmly.
3. Promote Active Learning and Collaboration:
   • Encourage group study sessions focused on explaining concepts to each other and practicing problems together.
   • This fosters deeper understanding, reinforces learning, and creates a support system that can benefit all students.
4. Highlight Female Role Models:
   • Invite guest speakers or showcase stories of successful women in STEM fields.
   • Seeing successful women in math can challenge stereotypes and inspire female students.
5. Targeted Feedback on Study Habits:
   • Analyze homework completion patterns to identify students who might benefit from guidance on effective study strategies.
   • Offer workshops or individual consultations on exam preparation techniques tailored to the course content.

Why These Interventions Might Be Effective

These interventions aim to address the potential causes of the gender gap. Low-stakes assessments and workshops can help manage test anxiety and improve study skills. Collaborative learning fosters deeper understanding and builds confidence. Highlighting female role models combats stereotypes and motivates students. Finally, targeted feedback equips students with the tools to succeed on exams.

Follow-up Questions

• Does the learning management system allow for anonymous feedback surveys?
• Are there existing resources or programs at your institution to address test anxiety or academic support?

By gathering this additional information, we can tailor these suggestions to leverage existing resources and student feedback to maximize the effectiveness of interventions.

Microsoft Copilot

Gender Gap in Introductory Math Class

Analysis of Data Patterns

The gender gap in exam scores for the large enrollment introductory math class is a concerning pattern that warrants further investigation. Despite both men and women performing similarly on homework assignments, the discrepancy in exam scores suggests potential underlying factors that may be hindering the performance of female students during high-stakes assessments.

Possible reasons for this gap could include:

1. Stereotype threat: Female students may experience heightened anxiety or negative self-perception during exams due to societal stereotypes about their math abilities, adversely affecting their performance.
2. Learning preferences: The instructional methods employed in the class may align better with the learning preferences of male students, potentially creating a disadvantage for female learners.
3. Lack of representation: A lack of female role models or mentors in the math field could contribute to a sense of alienation or reduced self-efficacy among female students, impacting their exam performance.

Instructional Interventions and Teaching Strategies

To address the gender gap and improve learning outcomes for all students, the following interventions and teaching strategies could be implemented:

1. Growth Mindset Interventions: Incorporate activities and messaging that promote a growth mindset, emphasizing that intelligence and math abilities are not fixed but can be developed through effort and practice. This can help combat negative stereotypes and boost self-confidence, particularly for female students.
2. Inclusive Instructional Design: Evaluate and modify the course materials, teaching methods, and assessments to ensure they are inclusive and cater to diverse learning styles and preferences. This could involve incorporating more collaborative learning activities, varied assessment formats, and real-world applications that resonate with a broader range of students.
3. Peer Mentoring and Role Models: Implement a peer mentoring program that pairs female students with successful upperclassmen or graduate students in math-related fields. Exposure to relatable role models can increase a sense of belonging and self-efficacy among female learners.
4. Formative Assessments and Feedback Loops: Incorporate frequent low-stakes formative assessments and feedback loops throughout the course. This can help identify and address learning gaps early on, providing opportunities for targeted interventions and support before high-stakes exams.
5. Inclusive Learning Communities: Foster inclusive learning communities within the class by encouraging collaboration, group discussions, and study groups. Creating a supportive and inclusive environment can reduce feelings of isolation and promote a sense of belonging for all students, including those from underrepresented groups.

By implementing these interventions and teaching strategies, the institution can work towards closing the gender gap in exam performance and create a more equitable and supportive learning environment for all students in the introductory math class.

ChatGPT 3.5

Scenario Analysis: Addressing Gender Gap in Introductory Math Class

Understanding the Discrepancy:

Upon reviewing the learning analytics dashboard for the large enrollment introductory math class, you notice a significant gender gap in exam scores, with men consistently outscoring women despite their similar performance on homework assignments. To address this issue effectively, it’s essential to delve deeper into potential reasons behind this gap.

Possible Reasons for the Gap:

1. Stereotype Threat: Women may be experiencing stereotype threat, feeling pressure or anxiety about conforming to negative stereotypes about their math abilities, which can affect their exam performance.
2. Differences in Study Habits: It’s possible that men and women approach studying and exam preparation differently. Men might engage in more individualized or competitive study strategies, while women may benefit more from collaborative or supportive study groups.
3. Instructor Bias: There could be unconscious bias from the instructor, whether in grading exams or in the way course materials are presented, which inadvertently favors male students.

Instructional Interventions:

1. Implement Growth Mindset Interventions: Encourage all students, regardless of gender, to adopt a growth mindset approach to learning. This intervention emphasizes the belief that intelligence and abilities can be developed through dedication and hard work. Activities such as reflective journaling, peer discussions on overcoming challenges, and highlighting stories of successful individuals who faced setbacks can help foster a growth mindset.
2. Promote Collaborative Learning: Design learning activities that foster collaboration and peer support. Assign group projects or study groups where students can work together to solve math problems, discuss concepts, and provide mutual support. This approach can help mitigate the effects of stereotype threat by creating a supportive learning environment where all students feel valued and capable.
3. Implement Blind Grading Practices: To address potential instructor bias, consider implementing blind grading practices for exams and assignments. This involves removing identifying information (such as names or gender markers) from student submissions before grading. By focusing solely on the content of the work, instructors can minimize the influence of unconscious biases and ensure fair evaluation for all students.
4. Provide Additional Resources and Support: Offer additional resources and support specifically tailored to address the needs of female students in the class. This could include hosting extra review sessions led by female teaching assistants or providing access to online tutorials and resources aimed at building confidence in math skills.

Rationale for Effectiveness:

• Growth Mindset Interventions: By fostering a growth mindset, students are more likely to persevere through challenges and setbacks, leading to improved learning outcomes.
• Collaborative Learning: Collaborative learning promotes active engagement, peer support, and diverse perspectives, which can enhance understanding and retention of course material.
• Blind Grading Practices: Blind grading helps ensure fairness and impartiality in evaluation, reducing the impact of unconscious biases and creating a more equitable learning environment.
• Additional Resources and Support: Providing targeted support acknowledges and addresses potential barriers faced by female students, empowering them to succeed academically.

Appendix C: Claude 3 Sonnet’s Responses to Follow-Up Prompts

Time-phased interventions

Prompt: From these broad strategies, please give me a specific technique I could easily implement within the next week, something a little more complex that I could add yet this semester, and finally a larger change that I could incorporate in my course design and teaching for the next time I teach this class. Do you need any more information to provide me with useful strategies to implement in the near, medium, and longer time frames?

Response: 

Near-Term Implementation (Within the Next Week)

Technique: Feedback Intervention

Provide all students with written feedback emphasizing that their current performance does not reflect their true potential, and that their abilities can be improved through effort and effective learning strategies. Frame the feedback in a way that promotes a growth mindset and encourages perseverance.

Example: “Your exam score does not reflect your true capabilities in this course. With dedication and the right strategies, you can improve your understanding of the material and perform better on future assessments.”

Medium-Term Implementation (This Semester)

Technique: Flipped Classroom with Value Affirmation Activity

Adopt a flipped classroom model where students engage with instructional content (e.g., videos, readings) before class, allowing more class time for active learning exercises. Before each class session, incorporate a brief value affirmation activity where students reflect on their personal values and write about why those values are important to them.

Value affirmation activities have been shown to mitigate the effects of stereotype threat and improve academic performance, particularly for students from negatively stereotyped groups.

Longer-Term Implementation (Next Course Offering)

Design Change: Inclusive Curriculum and Authentic Assessments

Review and revise the course curriculum and assessments to ensure they are culturally relevant, incorporate diverse perspectives, and feature authentic, real-world applications of the mathematical concepts.

Additionally, consider alternative forms of assessment beyond traditional exams, such as project-based learning or portfolio assessments, which may better capture students’ understanding and skills while reducing the impact of stereotype threat in high-stakes testing situations.

To provide more specific recommendations, it would be helpful to have additional information about:

1. The current course structure (e.g., lecture format, flipped classroom, hybrid)
2. The types of assessments currently used (e.g., exams, quizzes, projects)
3. The demographics of the student population (e.g., proportions of different gender and racial/ethnic groups)
4. Any existing initiatives or resources focused on promoting equity and inclusion at the institution or department level

With this additional context, I can tailor the suggested strategies more effectively to your specific instructional context and student population.

Interventions with most empirical research support

Prompt: From these broad strategies, please give me more detail about the two of them with the most empirical research support. Include references if available.

Response From Claude 3 Sonnet: 

Based on the table, the two broad strategies with the most empirical research support are Growth Mindset Interventions and Active Learning & Collaboration. Here are more details on these strategies along with references:

 

1. Growth Mindset Interventions:

Growth mindset interventions aim to shift students' beliefs about the malleability of intelligence and abilities. These interventions typically involve teaching students that intelligence is not fixed but can be developed through effort, effective strategies, and help from others.

 

Research has shown that growth mindset interventions can have a positive impact on academic achievement, particularly for students from underrepresented groups or those facing stereotype threat.

 

Example Studies:

- Blackwell, L. S., Trzesniewski, K. H., & Dweck, C. S. (2007). Implicit theories of intelligence predict achievement across an adolescent transition: A longitudinal study and an intervention. Child Development, 78(1), 246-263.

- Good, C., Aronson, J., & Inzlicht, M. (2003). Improving adolescents' standardized test performance: An intervention to reduce the effects of stereotype threat. Journal of Applied Developmental Psychology, 24(6), 645-662.

 

2. Active Learning & Collaboration:

Active learning strategies, such as collaborative group work, peer instruction, and problem-based learning, have been shown to improve student engagement, conceptual understanding, and academic performance, particularly in STEM fields.

 

Collaborative learning activities encourage students to work together, discuss concepts, and learn from one another, which can help create a supportive learning environment and mitigate the effects of stereotype threat.

 

Example Studies:

- Freeman, S., Eddy, S. L., McDonough, M., Smith, M. K., Okoroafor, N., Jordt, H., & Wenderoth, M. P. (2014). Active learning increases student performance in science, engineering, and mathematics. Proceedings of the National Academy of Sciences, 111(23), 8410-8415.

- Lorenzo, M., Crouch, C. H., & Mazur, E. (2006). Reducing the gender gap in the physics classroom. American Journal of Physics, 74(2), 118-122.

 

References:

- Dweck, C. S. (2008). Mindset: The new psychology of success. Random House Digital, Inc.

- Prince, M. (2004). Does active learning work? A review of the research. Journal of Engineering Education, 93(3), 223-231.

These strategies have been extensively researched and have demonstrated effectiveness in improving academic performance and closing achievement gaps, particularly in math and science fields where gender disparities are often observed.