Evaluative Summary of Article on

Two-way ANOVA CRF-pq Design

1. Background Information

Hector, M. A., Davis, K. L., Denton, E. A., Hayes, T. W., & Hector, J. H. (1979). Teaching counselor trainees how to respond consistently to client negative affect. Journal of Counseling Psychology, 26(2), 146-151.

2. Abstract

The authors used a 3 (modeling, didactic, and control group) x 2 (male and female) fixed-effect factorial design to examine the effectiveness of teaching methods in improving counselor’s consistency of responding to client’s negative affect. The necessity of teaching counselor trainees how to stay with client’s negative affect is implied by the authors who point that many practicum counselors have difficulties in responding to client’s negative affect, such as client’s anger or fear. In order to suggest an effective teaching method, the authors compare two kinds of methods, which are modeling and didactic teaching methods. According to the previous study (Perry, 1975) cited by the authors, the combination of the two teaching methods was more effective than either modeling or didactic method alone. However, a combined teaching method was not used in this study. The authors recruited 30 participants, who were counseling practicum students, and randomly assigned them into three groups, which are modeling, didactic, and a control group. After treated with one of the methods, the participants were assigned to counsel an angry role-playing client who was either male or female. The authors hypothesized that the participants treated with the modeling method would show more consistency in responding to client’s negative affect than the participants taught with a didactic method. They also hypothesized that the modeling and didactic group members would show more consistency in responding to client’s negative affect than the control group members. The results indicate that the modeling and didactic groups show more consistency of responding than the control group, but there was no significant difference between the modeling group and the didactic group.

3. Null hypothesis, alpha (or p) level, and sample size per group.

The three null hypotheses for the two-way ANOVA test are , , and . The first null hypothesis means that there is no treatment effect caused by the three types of methods. In other words, the modeling, didactic, and control groups will show no difference in consistency of responding to client’s negative affect. The second null hypothesis indicates that there is no treatment effect brought by the sex of the client, which means that client’s sex difference does not affect the counselor’s consistency of responding to client’s negative affect. The third null hypothesis indicates no interaction effect between the three types of methods and the client’s sexes. That is a specific combination of the treatment factors, e.g. modeling and female, does not cause more or less consistency of counselor’s response to client’s negative affect compared to other combinations of the treatment factors.

The three forms of null hypotheses are general for any kind of two-way factorial designs. However, the authors are specifically interested in testing the following null hypotheses, which are , , and . The forth hypothesis indicates that the modeling group and the didactic group are not different in the consistency of responding to client’s negative affect. The fifth hypothesis means that there is no difference of consistency of responding between the modeling group and the control group. Finally, the sixth hypothesis has the meaning that the didactic group and the control group are not different in the degree of consistency of responding to client’s negative affect. The authors did not make any hypotheses regarding the client’s sex effect or the interaction effect between the three types of methods and the client’s sexes.

The level was not stated, but the p level was reported in the ANOVA test. A significant F-ratio was reported at the method (or treatment A) effect with the p<.001 level.

The total number (N) of participants is 30. The authors describe that they assigned 10 participants to each of the three groups of methods. However, it is not clear if each cell of 3 x 2 factorial design contains 5 cell members because the authors failed to report the specifics by only saying that they failed to randomly assign the participants regarding to the client’s sexes (or treatment B) because of some difficulties in scheduling the meeting between the participants and the role-playing clients. If I assume that each cell has 5 members, each cell with 5 members can be described as the following: modeling () x male (), modeling () x female (), didactic () x male (), didactic () x female (), control () x male (), and control () x female ().

4. Independent and dependent variables

Two fixed independent variables are adopted, which are the kind of counseling skill teaching method and the client’s sex. There are three levels of the teaching method, which are modeling method, didactic method, and lecture on ethics (control). The participants of the modeling and the didactic groups were taught how to respond to client’s negative affect with modeling or didactic method, while the control group members were taught counseling ethics not the topics of dealing with client’s negative affect. The two levels of client’s sex are male and female.

Dependent variable is the degree of participants’ consistency of responding to role-playing client’s negative affect. The consistency was measured by two raters with stopwatches and the average of the two scores was reported as the ratio of the duration of counselor’s responding to client’s feeling and the total duration of the counseling session.

5. Instrument, comment on its reliability and validity.

The authors developed a new measurement tool to assess the counselor’s consistency of responding to client’s negative affect because there was not an appropriate measurement tool for this study. The authors used two raters and have them practiced in order to improve the reliability of the rating method. After the two raters rated the 30 videotaped counseling sessions independently, the reliability of the rating method was reported as a correlation of .83 on the time duration ratios. The correlation increased to .93 when the raters re-rated three divergent pairs of ratings. For the re-rating, the authors emphasize that the raters were not told about the discrepancy between the pairs of ratings, but just told to re-rate the three pairs.

To establish the validity of the measurement tool, authors had the two raters rate 20 of the 30 videotaped role-played counseling sessions (according to the authors 10 of the tapes were accidentally erased) with a recognized measure of empathic responding. For that rating, the reliability was .75 of Person correlation between the two raters’ ratings. For the validity of the instrument used in this study, the authors report .96 of correlation between the instrument used in the study and the recognized empathic responding measure.

6. Experimental procedure

The participants were all graduate students at the University of Tennessee, who were taking a counseling prepracticum course. The participants consist of 18 females and 12 males. They volunteered for the research study and obtained part of course credit. The researchers randomly assigned 10 participants to each of three groups. Modeling and didactic teaching method was applied for the first two groups and the third group was used as a control group.

The experiment was conducted as the following. Each participant came into an office individually and read a description about the experiment procedure. After reading the description, he or she followed one of the three tracks. First track: He/she was given a transcript of counselors’ statements that were used in two model counseling interviews. The counselor’s statements of the transcript were responses focusing on the client’s feelings. After reading the transcript, the participant watched a 10 minute-long video. In the video, two senior authors played as a counselor and a client for one scenario and then they changed the client-counselor roles in the second scenario. In the first scenario, a female college student client had feelings depressed and lonely because she did not have any dates. In the second scenario, a male client was scared and upset about the job interview that he was going to have. When the video was finished, the participant went to another room and counseled an angry role-playing client. Ten participants assigned into the modeling method group followed the first track. Second track: A participant was given an outline of short lecture on how to respond consistently to client’s negative affect. After reading the outline, the participant watched a videotape lecture by the first author. The lecture teaches various ways of consistently responding to client’s negative affect. When the video was finished, the participant went into another room, and counseled an angry role-playing client. Another 10 participants assigned into didactic method group went through the second track. Third track: A participant was given a lecture outline on four ethical situations that counselors might encounter. After finishing the outline, the participant watched a videotape lecture on the ethical issues. When the video was finished, the participant went into another room, and counseled an angry role-playing client. The rest 10 participants belonged to the control group followed the third track.

A male and a female played as the client. The authors conducted a pilot study in order to make sure that the both role players present as much similar role as possible.

The participants’ counseling sessions with the role-play client were video-taped.

7. Statistical analysis and conclusion

A 3 (modeling method, didactic method, and control group) x 2 (male and female) analysis of variance was conducted on the data gathered. The treatment A (counseling skill teaching method) was revealed to have a significant effect, F (2, 24)=10.35, p< .001. However, the treatment B (sex of role-playing client) showed no significant effect, nor did the interaction between the treatment A and the treatment B. The authors did a post-hoc with the Scheffé multiple comparison test, and found that both of the modeling group () and the didactic group () had a significant difference from the control group () in terms of the results of time duration of participants’ responding to role-plying client’s negative affect in each group, p< .01.

Although the modeling method was not superior to the didactic method, thus one of the author’s hypotheses was not supported by the results, the other hypothesis that both the modeling and the didactic method were more effective than the control group method was supported. The authors concluded that both modeling and didactic methods are effective in increasing the consistency of counselor trainee responding to client’s negative affect.

8. If you were the researcher, how would you improve the study?

I think the authors did a good job in describing the experiment procedure in detail and in making a new measurement tool to assess counselor’s consistency of responding to client’s negative affect. However, if I were the researcher I would improve the study in the following ways.

First, I would be more careful not to lose the data before the study is completed. The authors reported that 10 of the videotapes were accidentally erased.

Second, I would be more consistent in assigning the participants randomly to one of the angry-role playing clients. The authors stated that they failed because of the scheduling complex. Third, I would be clearer in stating the purpose of using the two-way ANOVA factorial design. Researchers usually use a factorial design because they suspect an interaction effect. In this study, however, the authors did not have any hypothesis about a possible interaction effect. They seemed to be interested only in the effect of the treatment A. Thus, if I were the researcher I would either set up a hypothesis about possible interaction effect while using the two-way design or use a one-way design to find out the A treatment effect only. Fourth, I do not understand why the authors used the Scheffé procedure while they seemed to be interested only in pair-wise comparisons. Maybe they tried to be more impressive by using a more conservative test, such as the Scheffé test. However, I would use the Tukey procedure or other pair-wise tests if I were interested only in the pair-wise comparisons like in this case. Fifth, I think a control group in this study was not treated in a right way. Researchers in social sciences usually use a control group to prove the effectiveness of a new method, e.g. a new math teaching method (treatment group) versus a traditional math teaching method (control group). Thus, the control group gets treatment based on the traditional method. For example, the control group members are taught math with the traditional method while the treatment group members are taught math with a new method. However, in this study the control group members were not taught how to consistently respond to client’s negative affect but they got lecture about other topics. Some one may argue that a control group may not get any treatment. That is true, but that is more applicable in other types of research, such as drug effect or counseling outcome research. However, in this research case the research was conducted to find out the effectiveness of teaching methods. Thus, I would get the control group taught, with a traditional way if there is any, about the same topic that was taught to the treatment groups, or I would let participants in the control group prepare themselves to see a angry role-playing client instead of watching unrelated videotape. Sixth, I think the authors did not make clear about the purpose of the research. The authors wrote that the purpose of the research was to compare the two teaching methods. However, the main purpose seemed to prove the two teaching methods were more effective than no-teaching because as I already pointed out the control group did not learn about how to deal with client’s negative affect. Thus, I would state a clearer purpose of the study. Seventh, I would report the effect size of the significant results such as the of the teaching method effect.