Susan Pass, Department of Teacher Education, Clemson University, USA, spass@clemson.edu

Abstract

Recent research indicates that there appears to be a statistically significant negative impact upon student evaluations of instructors (SEIs) if consistent research is done on those students by the instructor (i.e. one’s students are the subject of one’s research), F (1/15)=12.035, p= .004, N=367.  This finding impacts all professors whose SEIs are included in their review for promotion and tenure.

Introduction

Few SEI rating scales include an item by which students can rate their reaction/concern when their instructor does research on them.  Is it possible, therefore, that students use other items in the scale to report such dissatisfaction?  A brief review of the literature reveals that there are few studies available on the impact that doing research on one's college students has with one's student ratings of professor/course effectiveness.  Yet, an unofficial survey of current and past students of the assistant professor studied in this work indicates that the data in this article is valid and is useful information for those professors going up for tenure and promotion.

The Problem

Data from this case study reveals that, if a professor does research on his/her college students, the SEI ratings of that professor will be low.  Yet, many such professors are urged to do such research not only to improve their programs but to also increase their publications in peer-refereed journals.  Most institutions of higher education include student evaluations of instructor's scores when considering faculty promotion.  While doing research on one's students might result in program improvement and more academic publications, it can also result in a professor not receiving tenure/promotion, because the professor's student evaluation of instructor (SEI) ratings will be too low.  Thus, professors turn to other research that is done outside of their classes.  This might result in college programs not being adequately improved because doing research on current students is the best vehicle towards assessing program needs. The scope of this research prompted two research questions.

Two Research Questions

• ·What is the effect of doing research on one's students upon SEI ratings?

• ·Even if no research is being done, does the perception that research is being done on them (i.e., the instructor's students) have a negative impact upon SEI ratings?

Review of the Literature in College Education

Because the assistant professor in this case study used constructivist teaching techniques when doing research ("because it builds the higher-thinking skills, which I was concentrating on in my research") but traditional teaching techniques ("because it is easier and more efficient") when not doing research, this review of the literature is divided into three sections in order to better explain the research questions being investigated.

Student Evaluations of Professors

Some college seniors believe that SEIs are not worth filling out because, instead of reading them and getting rid of bad professors, administrators ignore the data (Mcilwain, 2006).  However, hard-working professors say that SEIs "worry them because they matter in my [department chair's] evaluations and in my raises" (Maines, 2006, State and Regional News).

Since the early 1970s, SEIs have been added to faculty folios for evaluation and promotion (Nilson, 2003).  Since the late 1980s, researchers have recommended that other data (besides SEIs) be included in tenure-and-promotion assessments (Cashin, 1998; Aleamoni, 1999; Arreola, 2000). Included in this would be alumni surveys and candidate exit surveys.  Both Cohen (1981) and Feldman (1998) found in independent studies that student achievement on an external exam correlates "decently with student ratings of teaching effectiveness” (Nilson, 2003, p. 228).  While some universities use SEIs to improve programs (as well as rate professors), Paul Grayson found that students' satisfaction with their professors' teaching had little to do with students' sense of program satisfaction, which was determined mainly by their satisfaction with their program in the previous year (Grayson, 2004).  Finally, Robert Thorson wrote that the best evaluation of college instruction comes long after the student has graduated.  "At this point, they will be mature, objective and grateful enough to decide which professors really mattered, and which did not" (Thorson, p. A11).

Literature on Problems with SEIs

SEIs might be too subjective in nature.  In 1994, Peter Brady found that positive professor attitudes towards their students result in statistically significantly higher student perceptions of professor effectiveness (Brady, 1994).  In 1997, Susan Lang found that student evaluations are statistically significantly higher when a professor has enthusiasm (Lang, 1997).  Researchers found that instructor friendliness and personality relate only weakly to achievement (Nilson, 2003).

However, other researchers have discovered that there are biases that exist and SEIs need to be read with these biases in mind (Nilson, 2003).   Both Cashin (1988) and Aleamoni (1999) state that some of the biases that do exist include instructor status (regular faculty are usually rated higher than adjuncts) and expressiveness.  In addition, they found that, if students' prior interest in the subject was high and the course was an elective, the SEI ratings were higher (Cashin, 1988; Aleamoni, 1999).  It also was discovered that the higher the level of the course, the higher the SEI ratings.  In addition, humanities courses receive higher ratings than social science courses do (Nilson, 2003).  Students also tend to rate more generously if they believe that their ratings will be used for personnel decisions and if they do not have to sign their SEI (Nilson, 2003).

Literature on Professor's Actions and SEIs

Contrary to popular opinion, the more demanding the course, the higher the SEI ratings (Nilson, 2003) but Greenwald and Gilmore found this was only the case if the professor of the demanding course graded leniently (1997). 

In 2006, Thomas Benton wrote that "The steady erosion of tenure and the use of SEIs as a faculty-culling device are turning college teaches into spineless crowd pleasers"(p.1) and ill prepares graduates for the world outside of college.  Michael Gordon (2006) wrote, "Grade inflation has many well-publicized roots, including the reluctance of faculty members to confronting students who have received low grades, or to endanger their careers by getting low ratings from aggrieved students on end-of-semester evaluations (p. 10).  Because of this tendency towards grade inflation plus the documented biases in SEI ratings, Nilson (2003) believes, "relying too much on student ratings to assess teaching effectiveness is …being unsystematic and capricious" and suggests that universities adopt a more comprehensive system for evaluating faculty performance -- especially for purposes of tenure and promotion (p. 228-229).

Data Collection

Because the SEIs and students of only one assistant professor were studied, this study used only education courses in its three data collections.  All 367 students were college upperclassmen. Data was retrieved on an assistant secondary social studies professor's SEI ratings over a period of eleven semesters and four courses.  .  The data was then sorted into two groups; namely, Group A (when research was being done on the college students) and Group B (when research was not being done on the college students). The table documenting all the data sets collected on the professors' SEI ratings is located at the end of this article in Figure One. Use of this data was done in three different collections:

• First, with all the classes that the assistant professor taught -- see Group A and Group B (page 5)

• Second, with only those two courses that had exactly the same students in them but in two different semesters -- see Group One and Group Two (pages 6-7).

• Third, with two groups enrolled in the same course taught in the same semester but in two different sections; namely Group Alpha (who thought research was being done on them when it was not) and Group Beta (who knew that research was not being done on them) to discover if just the perception of research being done on them can impact SEIs (page 7).

Data Findings

After the data was collected, but before the statistics were computed, some interesting facts emerged and the data tables in Figure One at the end of this article will verify the following facts:

• ·No research was done in the spring of 2001 and the spring of 2004 with Course A. The SEI ratings of those two semesters were significantly higher than the other semesters in which research was done on the students of Course A.

• Interestingly, this tendency also works for exactly the same students in different courses.  In the spring of 2004, no research was done on the Course A students. However, the following semester, in Course B, research was done. Notice that (with the same students) the SEI ratings are lower in the semester that research was done on those students. Conversely, those same students gave a much higher rating to the professor in the prior semester (when no research was done on them). 

• Although the SEI ratings were lowest during the first year that the assistant professor taught, the overall SEI ratings of all the assistant professor’s courses seem to have a significant pattern. Notice how much higher SEI ratings are of the same professor in courses C, D, and E (when no research was done on the students) than courses A and B (when research was done on the students).

The Instrument

A factor analysis was done on the SEI instrument (N=307) and there appeared to be two factors; namely, student assessment of instructor effectiveness and student assessment of course (as set up by the instructor) effectiveness.  A Cronbach's alpha of .80 revealed mild reliability (perhaps not high enough for an instrument upon which tenure and promotion is based). 

First Data Analysis and Results

The Impact of Doing Research on One's Students on SEI Ratings: Courses A, B, C and D

First, data was collected on all SEIs for all courses that the assistant professor taught over a period of 5 ½ years (N= 367).  Two groups were considered.  Group A were those classes that no research was done with the students and Group B were those classes where research was done on the students.  Group A gave significantly lower SEI ratings:
_____________________________________________________

Groups                         N                     Mean                  Std. Deviation  

A                                    3                      4.1800             .6286              

B                                  13                     3.1938             .4049                           .

Total                            16                     3.3788             .5847
_____________________________________________________

A one-way T-test between the mean items 1-14 (on the SEI instrument) between the classes that had no research done with them (Group A) and the classes that did (Group B) revealed that the difference in mean scores was statistically significant with 163 students enrolled in 15 classes, t (14) =3.46, p=.004,  (2-tailed), d=.98615:

_______________________________________________________________

Scores                                                  t           df         Sig. (2-tailed)   Mean Difference

Equal variances assumed                       3.469   14        .004                 .98615

Equal variances not assumed                2.596   2.398   .101                .98615

_______________________________________________________________
 

Since these education courses consisted of college upperclassmen specializing in different education programs (e.g., elementary, secondary English, etc.), a one-way ANOVA was computed to take into account those differences.  This statistic revealed that the differences between the two groups were still statistically significant, with lower SEI ratings given in the classes that the professor did research with, F (1/15) =12.035, p= .004:

_______________________________________________________________

Scores                          Sum of Squares            df         Mean Square    F                      Sig.

Between groups            2.370                           1          2.370               12.035           .004

Within groups                 2.758                         14        .197

Total                                 5.128                         15

_______________________________________________________________
 

Second Data Analysis and Results

The Impact of Doing Research on One's Students on SEI Ratings: Courses A and B (Same Students)

To further work with the data, a colleague suggested that data be run only on courses that had exactly the same students.  So, two similar courses that contained exactly the same students enrolled in exactly the same education program (students were required to take course A (methods) their first semester of their senior year followed by course B (student teaching seminar in the second semester of their senior year) were compared. 

Courses A and B data. Data was retrieved on the assistant professor's 164 SEI ratings over a period of eleven semesters with the two courses.  The data was then sorted into two groups; namely, Group One (class sections of courses A and B, when research was not being done on the assistant professor's college students) and Group Two, when research was done on the professor's college students. 

Comparing the means of the two groups revealed statistically significant higher SEIs scores given by the group that had no research done on them (Group One) even though the students were exactly the same in both groups. Conversely, the SEI scores were statistically significantly lower (Group Two) if research was done with those same students:

_______________________________________________________________

Groups                   N                 Mean            Std. Deviation        Std. Error Mean

One                         3                 4.183           .6286                     .3630

Two                       12                3.186           .4218                     .1218

_______________________________________________________________
 

A one-way T-test between the mean items 1-14 (on the SEI instrument) between the classes that had no research done on them (Group One) and the classes that did (Group Two) revealed that the difference in mean scores was statistically significant, t (13) =3.50, p=.005 (2-tailed), d=.9942.  Again, the SEI ratings were higher with students who had no research was being done on them:

_______________________________________________________________

t-test for Equality of Means

Scores                                       t         df       Sig. (2-tailed)         Mean Difference

Equal variances assumed             3.35   13      .005                       .9942

Equal variances not assumed       2.597 2.47   .098                     .9942

_______________________________________________________________
 

Each course consisted of students who were specializing in one of the following subjects: economics, geography, history, political science, psychology or sociology.  Because of this variety of subjects represented among the students being studied, a one-way ANOVA was done.  This revealed that the classes which had research done on them scored the professor statistically significantly lower than the classes which did not have research done on them, F (1/13)=11.224, p= .005:

_______________________________________________________________

Scores                       Sum of Squares      df       Mean Square           F                  Sig.

Between groups      2.372                       1        2.372                     11.224         .005

Within groups           2.747                     13      .211

Total                            5.120                     14

_______________________________________________________________
 

Third Data Analysis and Results

The Impact of Perception that Research is Done on Them on SEI Ratings: (Course B)

In the same spring 2006 course (student teaching capstone), two sections were compared.  Group Alpha was section 001 that believed research was being done on them and Group Beta was section 002 that knew no research was being done on them.  Perception appears to be reality as there were statistically significant higher SEI ratings in the section that knew no research was being done on them and statistically significantly lower SEI ratings in the section that thought research was being done on them, t (25)=8.072, (2-tailed) p=.000, df 23.:

_______________________________________

Groups         N                 Mean            Std. Deviation        

Alpha           12                3.4967                   .2067          

Beta             13                4.2600                   .2604                     .

Total            25                3.8936                   .4527

_______________________________________


_______________________________________________________________

t-test for equality of means

Scores                                       t         df                 Sig. (2-tailed)         Mean Difference

Equal variances assumed             8.072 23      .         000                        .7633

Equal variances not assumed      8.149 22.524       .000                       .7633

_______________________________________________________________
 

Each section consisted of students who were specializing in one of the following subjects: economics, geography, history, political science, psychology or sociology.  Because of this variety of subjects represented among the students being studied, a one-way ANOVA was done.  This revealed that the section which had research done on them still scored the professor statistically significantly lower than the section which knew no research was being done on them, F (1/23)=65.156, p= .000:

_______________________________________________________________

Scores                       Sum of Squares      df       Mean Square           F                  Sig.

Between groups      3.636                      1        3.636                     65.156         .000

Within groups           1.283                     23      5.580E-02

Total                           4.919                     24     

_______________________________________________________________
 

Comments

Surveys were done with the candidates a year after graduation.  One former student wrote that, while he did not appreciate the course while he was in it, he now realizes that what was done was "good" for him.  Nevertheless, he still "resented" the research.  Another student wrote that the students in the class were "under stress" and having research done with them “just added” to the stress.  Another former student stated that he always "resented" when professors were doing research because he felt that he was "used." Finally, a person in charge of our university's research compliance unit wrote, " Please address the discomfort that may be experienced by students in the class regarding how their participation or refusal to participate in the research study (that is, their decision to allow or not to allow their reflective essays to be used for research purposes) might influence the instructor’s interest in them as students and might influence their grade in the course."  When, asked, this administrator sent the following email message to clarify her position:

My statement was just a general statement, reflecting concerns that students might be uncomfortable with being asked to participate in research or might feel they don't truly have a choice to not participate in research conducted by a course instructor. I don't know of published research about negative student reactions to having research conducted in courses. However, it seems to me that there's a widespread belief in the IRB community that students function in an environment that is inherently coercive when it comes to participation in academic research and that care must be taken to reduce even the possible perception of coercion in order to ensure the voluntariness of student participation in research. I don’t know whether this belief is based in research, but I can tell you I've bought into it.  If you know of research that suggests it's not true, I'd be really interested in learning about it.

Limitations

Only one professor's students were used in this study. Second, this study's results might be impacted by the fact that constructivism was used in all research with the college students. It might have been the constructivism and not the actual research that led to negative SEI ratings. A review of the literature states that critical thinking is often an implicit characteristic of higher education (King & Kitchener, 1994; Browne & Freeman, 2000; Mentkowski et a., 2000). One colleague suggested that most college seniors fear their professors and having to use critical thinking (i.e, constructivist) techniques in a college classroom is perceived as too threatening. However, in the third data analysis and results (see pages 8 and 9), the same course was taught in two different sections – one section (Beta) knew that no research was being done one them and one section (Alpha) thought research was being done with them. Both sections were taught with traditional teaching methods; yet, the section that thought research was being done on them reported statistically significantly lower SEI scores. Recent research seems to indicate that college students react negatively on SEIs when constructivist teaching strategies are used because they find them threatening (Nilson, 2003). So, this research plans to duplicate the above study with just traditional means of instruction being done while doing research on one's students in order to compare the results with the above data, that used constructivism in the research.

Conclusion

Given the above data results, it is difficult to recommend research on one's own students who will eventually evaluate your teaching effectiveness. This case study has shown that such action results in lower SEI ratings. This finding is valid with both different students in different courses or the same students in different courses over time. Finally, this study also revealed that just the perception research is being done on them will lead students to give lower SEI ratings. More research needs to be done on this topic.

References

Aleamoni, L.M. (1999). Student ratings myths versus research facts: An update.  Journal of Personnel Evaluation. 13 (2), 11-16.

Arreola, R.A. (2000). Developing a Comprehensive Faculty Retention System. Bolton, MA: Anker Publishing.

Benton, Thomas. (2006, June 9). A tough-love manifesto for professors. The Chronicle of Higher Education. 52 (4), 1.

Brady, Peter. (1994, June). How likeability and effectiveness ratings of college professors by their students are affected by course demands and professors' attitudes.  Psychological Reports 74 (3), 197-204.

Browne, M.N. & Freeman, K. (2000). Distinguishing features of critical thinking classrooms.  Teaching in Higher Education.  5 (3), 301-309.

Cashin, W.E. (1988). Student ratings of teaching: A summary of the research. IDEA Paper No. 1, Center for Faculty Evaluation and Development, Kansas State University.

Cohen, P.A. (1981). Student ratings of instruction and student achievement: A meta-analysis of multi-section validity studies.  Review of Educational Research.  51, 281-309.

Fledman, K.A. (1998).  Identifying exemplary teachers and teaching: Evidence from student ratings.  In Teaching and learning in college classrooms.  K.A.Feldman & M.B. Paulsen (Eds.).  Needham Heights, MA: Simon & Schuster.

Gordon, Michael. (2006, August 11). When B's are better. The Chronicle of Higher Education. 52 (49), 10.

Grayson, J.P. (2004). The relationship between grades and academic program satisfaction. Journal of Higher Education. 24(2), 1-34.

Greenwald, A.G. & Gillmore, G.M. (1997). No pain, no gain?  The importance of measuring course workload in student ratings of instruction.  Journal of Educational Psychology. 90 (4), 743-751.

Howard, G.S. & Maxwell, S.E. (1982).  Do grades contaminate student evaluations of instructors? Research in Higher Education.  16 (2), 175-188.

King, P.M. & Kitchener, K.S. (1994). Developing reflective judgment: Understanding and promoting intellectual growth and critical thinking in adolescents and adults.  San Francisco, CA: Jossey-Bass.

Knox, John. (2005, September 2). Professor says rank better from students. The Augusta Chronicle. Metro, Bo5.

Lang, Susan.  (1997, fall). Student ratings soar when professor use enthusiasm. Human Ecology Forum. 25 (4), 24-25.

Maines, Sophia. (2006, September 18). Task force to review evaluation process. Journal World. State and Regional News section.

Mcilwain, Ben. (2006, April 25). The Diamondback. Column.

Mentkowski, M., Rogers, G., et al. (2000). Learning that lasts: Integrating learning, development, and performance in college and beyond.  San Francisco, CA: Jossey-Bass.

Murray, H.C. & Renaud, R.D. (1998). Disciplinary differences in classroom teaching behaviors.  In Teaching and learning in the college classroom. K.A. Feldman & M.H. Paulsen (Eds.).  Needham Heights, MA: Simon & Schuster.

Thorson, Robert M. (2006, August 24). Defining best professors. Hartford Courant. Editorial, A11.