School of Education Building          Fall Semester 2012

EDUC-R690: Application of Research Methods to IST Issues

Credits:  3
Class hours: Thursday, 1:00 - 3:45 p.m. in ED 2275
Instructor:  Ted Frick
Office and phone number:  ED 2218, (812) 856-8460



Table of Contents



Goal and Objectives

The goal of R690 is:

To help you learn disciplined inquiry in Instructional Systems Technology through first-hand experience -- i.e., by doing representative research tasks and critquing research done by others. 

  1. Conduct interviews for needs assessment (qualitative method).
  2. Do content analyses (qualitative method).
  3. Conduct usability evaluations (qualitative and quantitative methods, problem diagnoses).
  4. Write report describing needs assessment, results, usability evaluation, results, and recommendations (for client and instructor).
  5. Analyze existing data with SPSS; present and interpret results (quantitative methods, tool skills).
  6. Critique research reports done by others.

We will only have time to do an introduction or overview of most of these tasks in this class. But you will get your hands on data, do data analysis by hand and with computers, and even collect some data yourself. Similarly, you will critique research articles in the IST field, but this will be at an introductory level.

You will take inquiry courses where you can study and apply these techniques in greater depth; you will likely have opportunities in working with faculty mentors in R695 seminars to use some of these techniques and tools skills and to refine them; and of course you will do research yourself culminating in a dissertation study at the end of your Ph.D. program in which you will very likely use one or more of the above techniques.

Overview of the Course

1. Do a needs assessment of stakeholders.

This will involve collection and analysis of mostly qualitative data from several sources in order to find out what tasks the stakeholders need to do on the School of Education (SoE) and IST Website.

1.1.  Interviews of Stakeholders.  Once we have a set of interview questions (which we will develop together in class), each R690 student will conduct two or three interviews and write up results of each interview (this a deliverable for R690).  R690 students will work in pairs; each one does each interview with his or her partner as an observer. Both must listen and write down what the interviewee says.

1.2.  Collect FAQs from information gatekeepers. What are the 10 most frequently asked questions:  how often; who asks, when asked, how asked; and what is the answer to each FAQ.

1.3.  Do content analysis of results from 1.1-1.2.  This will be a class activity led by the instructor. Results of this content analysis will be recorded and shared with the clients.

2.  Do usability tests of the SoE and IST Website. 

2.1. Use content analysis results from 1.3. to identify tasks for the usability tests. R690 students are paired.  Each pair conducts two usability tests and writes them up (this a deliverable). 

2.2.  Do a content analysis to identify usability problems with the current Website.  This will be a class activity.

None of the inquiry tasks above will require human subjects approval by the IRB, since these are class activities for learning purposes, and the results will not be published.  This is not research resulting in generalizable knowledge or results to be disseminated, according to the definition of research in the IRB requirements. 

The basic goals here are for students to practice interviewing skills, practice doing content analysis for synthesis, practice doing a usability test, and doing yet another content analysis in the problem identification. These are largely qualitative methods, but applied to an IST kind of issue.

3. Write reports for the client. (10 points per team + 5 points for individual contribution)

These reports will describe each study conducted above:

A. Goals of the needs assessment; selection and description of the sample from the target audience; procedures for conducting the interviews and interview questions for the needs assessment; results of needs assessment; content analysis of results; summary and conclusions (Team A).

B. Goals of the usability evaluation; selection and description of the sample from the target audience; tasks and procedures for usability evaluation; results of usability evaluation; and recommended changes in the SoE and IST Website based on the results (Team B).

The writing of each report will need to be done by dividing tasks and coordinating among the R690 students. Each member of the team will be graded according to the quality of the overall team report (10 points max.), and according to independent peer evaluations of that team member's contributions (5 points max.).

4. Analyze existing data. (15 points per individual)

The next project for R690 will involve analysis of existing data.  Students will be analyzing some existing survey data (already collected in a previous study -- with data on 460 subjects, which has been sanitized), in order for students to get practice in doing basic descriptive statistics and their interpretation. 

Each student will be given a set of analysis tasks as a deliverable. Each student will work with a unique subset of 50 cases to be done via SPSS and is expected to interpret the results in an individual written report. The analysis tasks will include use of the appropriate descriptive statistic and interpretation. Several relationships between variables will be described using the appropriate measure of association (e.g., Pearson Product Moment correlation, chi square, Spearman's rho). In addition, you will do a pattern analysis for combinations of 3 or more classifications.

5. Choose an existing research study in an area of interest in IST, present a summary of the research to the class, and lead the discussion in which the class together evaluates the study. (15 points per individual)

The presentation and discussion during an R690 class meeting will last approximately 45 minutes. The presentation should be done in Powerpoint, and include:

5.1. A summary of the purpose of the study, how or why it is relevant to IST, research questions addressed, methods used, and results or conclusions of the study by its author(s).

5.2. Identify major strengths and weaknesses of the research study according to criteria discussed in class, as well as in Fraenkel and Wallen (2008) for the type of research that is attempted.

6. Write a formal critique of that research study, using APA style. (25 points per individual)

The critique should follow the outline and include a:

6.1. Description of the purpose of the study, how or why it is relevant to IST, research questions addressed, methods used, and results or conclusions of the study by its author(s).

6.2. Description of each criterion used and justification of it according to philosophic principles and/or accepted values held by research methodologists regarding the respective research method(s) used in the study being evaluated.

6.3. Application of each criterion to the study being evaluated and description of how the criterion was or was not met.

About Plagiarism: Do not do it!

Originality of Work: all work you submit for R690 must be largely your own. Plagiarism will not be tolerated. If you plagiarize the work of others, you will receive a failing grade for R690 and be reported to the IST chairperson for disciplinary review. If you do use the work of others, it must be a minor portion of what you submit and the original creator must be fully credited and such credits must be clearly visible to the user.

Evaluation and Grading

Grading Scale




Readings for Reference in Critiquing Research Reports

Booth, W. C., Colomb, G. G., Williams, J. M. (2008). The Craft of Research (3rd. Ed.). Chicago: The University of Chicago Press (required). This is exceptionally clear and concise, includes a variety of examples, and will be helpful in your research critiques for evaluating the logic of arguments made in research reports. Available in print or Kindle. You can download the free Kindle Reader to your PC, Mac, iPad, iPhone, Android, or other devices and read the book without owning a Kindle device--or just use the free Cloud Reader.

Fraenkel, J., Wallen, N. & Hyun (2011). How to Design and Evaluate Research in Education. New York: McGraw-Hill (required). This book will be useful for citing and justifying criteria for your research critiques, especially for evaluating both quantitative and qualitative research methods. I recommend the Kindle edition, since it is much less expensive that the hardcover, you can search it electronically, and key terms have hyperlinks to other parts of the book where they are explained.

Milinki, A. (1999). Cases in qualitative research: Research reports for discussion and evaluation. Glendale, CA: Pyrczak Publishing. This book provides reprints of published research studies in which qualitative methods are used (optional). Through these examples, important criteria for evaluating such research can be discussed. Not only are qualitative cases provided, but also are cases that combine qualitative and quantitative methods (now called mixed methods). Furthermore, contrasting studies of the same topics are provided, where one uses a qualitative approach and theother a quantitative approach.

Patten, M. (2012). Understanding research methods: An overview of the essentials (8th Ed.). Glendale, CA: Pyrczak Publishing (optional). This, too, is an exceptionally clear and concise overview of research methods. Chapters include discussion of literature review, sampling, measurement, experimental design, statistics, effect size, meta-analysis, qualitative research, and preparing research reports. Understanding ideas in this text will be helpful in your research critiques.

Pyrczak, F. (2010). Making sense of statistics: A conceptual overview (5th Ed.). Glendale, CA: Pyrczak Publishing (optional). This is an exceptionally clear and concise overview of basic concepts in statistics, which gets right to the essence of ideas--empirical research, sampling, measurement, descriptive statistics, correlational statistics, inferential statistics and effect size. It covers only statistics included in all textbooks, and which are commonly used in empirical research. This text also includes examples, and practice with feedback. Understanding ideas in this text will be helpful in your research critiques of studies using quantitative methods.

Steiner, E. (1988). Methodology of Theory Building. Sydney: Educology Research Associates (required--available online for R690 students at no charge). This book will also be useful for citing and justifying criteria for your research critiques, especially for evaluating descriptive and explanatory theory.

Additional readings are listed in the R690 schedule and resources (IU network ID and password required for the latter).

Department of Instructional Systems Technology
School of Education
Indiana University Bloomington