Identify a good research question
As with other types of research, a SoTL project should begin with a good question. It should be meaningful, in the sense that its answer should stand a good chance of being generalizable. If the inquiry is so idiosyncratic that you are the only person to whom the answer will be relevant, then even if it is worth it for you to pursue the question, the project would not be a good fit for the SoTL community – as making your findings public is a critical aspect of SoTL.
A question that makes a good starting point for SoTL typically has a dependent variable within it somewhere. In other words, there is a specific outcome that you can observe and try to explain. For example, “What caused student performance in one section of my course to be stronger than in the other section?” In this question, student performance is the dependent variable: it is an outcome that you can explore, because it was different in one case (the M/W section, perhaps) than in another case (the T/Th section, perhaps). What caused this? Especially if you have a hunch that something you did is responsible for the difference, this question is a good starting point for a SoTL project.
A research question might be backward-looking, in the sense that you know the outcome and are looking for the cause. For example: “Why did fewer students earn failing grades on my exams this semester compared to last semester?” Alternatively, it could be forward-looking, in the sense that you have an intervention and want to know what effect it will have. For example: “How will peer instruction affect student performance in my course?” (Although the latter type of question is a perfectly reasonable starting point for a SoTL project, do remember that we cannot predict the future; rather, you will need to wait until after the course has concluded to answer this question. Consequently, the question will end up becoming “How did peer instruction affect student performance in my course?”)
Find a suitable approach
With a question in hand, you will need to find a suitable approach for answering it. While we cannot address all of the ins and outs of research design here, there are a few common types of study that frequently appear in the SoTL literature. Below, we briefly describe what these look like. Begin by seeing if one of these general approaches makes sense as a way to answer the question you have in mind.
Single course before-and-after
Many studies measure their dependent variable both before and after a given intervention. A common way of doing this is to administer a pre-test, provide some manner of intervention, and then administer a post-test. The change from pre- to post- reflects the effect of the intervention. Often what is measured in the pre- and post-tests is content knowledge; but not always. It could be student perceptions of or feelings toward something. Here is an example of what this type of design might look like: I want to know whether an activity I have designed will make students more confident in their ability to carry out a particular task. I begin by asking them a small battery of questions to assess their confidence in carrying out the particular task; we proceed with the activity; then I ask the same battery of questions to assess confidence. The increase (hopefully!) in their confidence from before to after is a measure of the effect of the activity.
Although this approach is relatively common, it suffers from a major shortcoming: of course our intervention will have some effect. How do we know what to make of the effect? How do we determine whether it is large, small, or somewhere in between? And, most important, on what basis could we decide this? There is no obvious metric for this, so it is better if we have a natural basis for comparison. The pre-test/post-test approach makes much more sense if we perform these tests more than one – for example, before and after one intervention and then before and after a different intervention (likely in a different section or a subsequent term). This offers a clear basis for comparing the effects of these two interventions. Because of the desire to have built-in points of comparison, you should seriously consider, if possible, one of the following three types of study design instead of this “single course before-and-after” design.
Single course one semester vs. another
Another common approach is to compare a single course to the same course from a different semester. This makes sense when you institute some kind of change that you expect to be beneficial. Generally speaking, many of the things you could say about your course one semester would be true about the same course in a different semester: the instructor is the same, the subject matter is the same, the course level is the same, the class size is (in most cases) similar, etc. Consequently, if you made no pedagogical changes to the course, then you should expect similar (though not exactly the same) outcomes from one term to the next. There may be slight changes due to the different term (Spring vs. Fall, for example), a different time of day, or simply the different group of students. However, you would not expect major changes. Thus, when you do make a significant adjustment to the course (for example, a different assessment structure), you can look for changes from one term to the next in learning outcomes, as evidence that the change had an effect – provided that the changes you witness are logically related to the pedagogical adjustments you made. Any evidence needs to be interpreted carefully, because there are many other things that could have been the actual cause of the different outcomes from one semester to another. But developing precise expectations (i.e., “hypotheses”) about what you would find if the adjustment you made did or did not have an effect, and then observing the actual outcomes (i.e., testing the hypotheses), is a common method for shedding some light on the question you are trying to answer.
Single course one section vs. another
This approach is quite similar to the above – comparing a single course one semester to the same course in a different semester – except that the comparison is done with two different sections of your course, within a single semester. As above, the logic is that these two sections share many of the same fundamental facts: same instructor, same subject matter, same course level, same or similar class size (in most cases), etc. Consequently, there are relatively few variables available to explain any changes you might observe in learning outcomes between the two sections. This helps you isolate the possible effect of pedagogical adjustments that you make. The same caution as above – that evidence needs to be interpreted carefully, because there may be many confounding factors – applies here, as well.
A complication of this approach is that you need to consider whether it is appropriate to make pedagogical adjustments to one section of your course and not the other. Whether or not this is a problem depends in large part on the nature of your project. Many SoTL projects are based around a pedagogical innovation that the instructor expects will have positive effects. If you are instituting an innovation that you expect to have positive effects (even if you still need to study that expectation empirically), then is it appropriate to make that adjustment in one section but not another? If you could apply the innovation to both of the sections, instead of just one, but you choose not to so that you can complete a study, then you are arguably disadvantaging the students in one of the sections. In addition to being a questionable decision, this is something the IRB would likely flag as they review your project for human subjects approval.
Multiple courses/instructors
If you have collaborators, then you might consider doing a larger project, in which you explore the question that interests you by gathering information from multiple courses. This could involve looking at sections of the same course taught by a variety of instructors, or it could involve more variation than that – potentially including multiple courses, multiple terms, multiple disciplines, and/or multiple institutions. In all of these cases, the logic of the above two approaches breaks down: since you have multiple instructors and courses, you are no longer holding these important factors (and everything that goes along with them) constant, so it is much more difficult to isolate a particular pedagogical intervention as the likely reason for the learning outcomes you observe. Instead, this approach requires you to gather enough information that you can control for important variables statistically. A fairly large number of courses will need to be included in order to perform the type of analysis (typically, but not always, regression analysis) that allows you to control for the many variables that are not of interest in your study. There is no easy rule about how many courses you will need to gather in your sample, because it depends on many things – including how confident you want to be in any conclusions that you draw. But, at a minimum, think dozens – and, ideally, hundreds – if you want to generate persuasive findings from this type of design.
Locate partners on campus
You will not be able to complete your project entirely on your own. At a minimum, you will need human subjects approval before beginning; so following the guidance of the Institutional Review Board, and corresponding with them as needed, will be strictly essential. In addition, you will want helpful feedback as you design and complete your project. Colleagues in your own department may be a great asset to you, but you will likely find that colleagues from other departments and colleges have much to offer, as well. Finally, as noted above, you might find it useful to speak with someone at the CTT about your ideas. There are several members of the team with SoTL expertise, but the point of contact for SoTL support inquiries is Abby Mitchell (amitchell22@unl.edu). They will be happy to discuss your project with you no matter where you are in the process of completing it.
Topics