FORUM for Case Study Learning

Analyzing Research Reports in a Computer Conferencing Environment

W.R. Klemm

Reprinted with permission from Journal of College Science Teaching, 31 (2002): 298-302.

 

Journal articles are the fundamental building blocks of the edifice of science. Students learn the necessary skills and confidence to read primary research literature by working in groups in a computer conference. This model for analyzing research papers combines a form of guided group inquiry in shared computer workspace.

Research reports in scientific journals are the fundamental building blocks of the edifice of science. Students in any scientific discipline should therefore learn to interpret primary research literature (Greene 1991). Many science teachers have doubts about teaching undergraduates how to master primary research literature. Renewed interest in this topic seems to have developed in the last few years (Janick-Buckner 1997; Herman 1999; Fortner 1999).

Using journal articles as case studies gives students experience in applying their memorized facts of science toward developing the more important analytical and creative capabilities. Science facts can be taught in many ways; much more challenging is teaching how to manage, comprehend, and apply the facts of science. Case studies based on journal articles provide a powerful mechanism for teaching these higher skills.

I use journal articles to complement and enrich my neuroscience course for senior-level undergraduates. Three elements make it work:

· Small-group teams;

· A step-by-step analytical process;

· An anytime-anyplace computer conferencing environment.

Working as a Group

Using the group approach when examining journal articles as case studies enables students to help each other surmount the difficulties. Fortner (1999) uses the "jigsaw puzzle" approach—each student in a group is assigned certain aspects of an article to master. Students teach the other students in their group what they learn. The risk of this method is that each student becomes expert in only a small part of the paper. Janick-Buckner (1997) teaches scientific content by using a dialog approach involving the whole class, not student groups.

Using Journal Articles in Teaching

The teaching approach with journal articles is typically unsystematic. Traditionally, students present a journal article, which consists of a verbal description of what is in the article, along with some overhead or slide show projections of key data. Students do this with great variability, and typically they do not do it very well.

The presented model guides students through the process one step at a time. Questions are provided at each step to help students do it correctly. Each student participates in each step of analysis.

Analytical Model for Case Study

This analytical model describes a tiered sequential process in which students understand what is described in the paper; assess the strategies, methods, and experimental results; integrate the case study paper with the general body of knowledge; and create new insights and research hypotheses. These four levels of inquiry can be visualized as an ascending staircase in which the tiers between steps involve a series of specific analytical tasks (Figure 1).

Figure 1. Analytical model for scientific research reports

 

For each tier of the model, student groups do the following:

Understand

When reading scientific journal articles, students may not understand many concepts, methods, and terms. This is a daunting challenge for newcomers to a field, so it is important for groups of students to work together in written format. These teams should be heterogeneous in terms of academic background, learning styles, and ability levels. The group’s first task is to identify learning issues, such as the tasks they need to complete to give them an adequate knowledge base to understand the journal article. They should ask themselves such questions as: What theories/methods/etc. in this article do I not understand? What do I need to look up to increase our perspective about this article? Students assign each other different learning issues and then they report back in  online conference, so the other students can annotate with in-context pop-up notes that provide questions, comments, and more answers. In the conferencing software that we use, these pop-up notes are hyperlinked in context. For example, one student may select a section of text written by another student and create a link to a pop-up note that is the equivalent of "writing in the margins" of paper-and-pencil documents. The group must understand the subject, before they are equipped to assess the paper.

Assess

Janick-Buckner (1997) argues that students must be told what to look for in scientific papers. Novices do not assess scientific work well without guidance. As part of the guidance model, students need to address specific issues, such as recognizing the underlying principles of the discipline that are involved in the paper. Asking students to identify core principles helps them focus on key issues, develop discernment between fact and principle, avoid bogging down in details, and remember the article's content.

Another integral part of the model is critiquing the methods, results, and conclusions in the reported research. Such analytical tasks include addressing bias; pitfalls; alternative interpretations; adequacy of controls; magnitude and statistic reliability of treatment effects, and the extent and quality of supporting evidence.

Integrate

Students evaluate the article in the context of what is already known about the subject, both what they already know and material that they have gleaned from extra reading. They benefit from examining the paper from diverse points of view. In the case of neuroscience, for example, students can consider how the article is perceived from such perspectives as anatomy, physiology, pharmacology, pathology, psychology, philosophy, sociology, and clinical medicine.

Students also should evaluate the article in terms of its theoretical and practical significance. They consider the context of the hypothesis, the relationship to alternative hypotheses, and the extent to which results seem to be advancements in the field.

Create

A critical skill for science students is conceiving ways to advance science. Students are challenged to become creative, to think of new ideas, and to create testable hypotheses that were not envisioned by the authors.

Students are required to ask an insightful question and then to try to answer it (see Klemm 1998). At first, students tend to raise rather trivial insights, but with experience and feedback from the teacher, their work improves. An example of a relatively trivial question for students at this level might be: How do we know that higher-level thought process arise in the cerebral cortex? A more challenging question might be: Do bad brains cause bad behavior or does bad behavior cause bad brains? When they see the work of other students in the computer conference, they get much better at such a task. This model is similar to the guided-direction approach of Janick-Buckner (1997) but differs by requiring students to define all terms they do not know, identify core principles of the discipline that are reflected in the paper, and create new hypotheses and experimental designs.

The working model describes a tiered sequential process in which students (1) understand what is described in the paper, (2) assess the strategies, methods, and experimental results, (3) integrate the case study paper with the general body of knowledge, and 4) create new insights and research hypotheses.

Using a Computer Conference Environment

When student groups can conduct their case studies in an anytime-anyplace computer conferencing environment, the quality of their work improves markedly. There are several reasons:

· Students do not have to coordinate their schedules, because all students can choose their own time and place for interacting with the group.

· Students have time to reflect, check the library, and "incubate" their thoughts.

· Students become more precise and more clear, because the work is written and needs to be understood by fellow team members.

· Contributions to the case study, because they are documented in writing, become easier to reference, search, and update.

· Flaws in understanding or thinking are more likely to be detected and corrected by someone in the group.

· The conference provides a "paper trail" showing who said what and in what context. Everybody sees who is doing outstanding work and who may be under performing.

· Individual accountability increases. Students are conspicuous by their absence. The teacher and every member of the group can see who is contributing what to the group effort. Pride of ownership of information and ideas becomes a powerful motivator. Last year, one student inserted a comment in his group's workspace that said, "You guys are embarrassing me by such good work. I WILL do better."

Students assign each other different learning issues and then they report back in an online conference, so the other students can annotate with in-context pop-up notes that provide questions, comments, and more answers.

Evaluating Specific Conferencing Software

The brand of computer conferencing environment used makes a difference in the quality of work that can be done. All well-known conferencing systems are essentially e-mail organizers that allow students to submit input as mail messages. When other students wish to comment on another student's message, they must submit a separate message in which all or part of the original message is repeated so that others can know the context for the comment. This leads to a confusing and inefficient set of mail messages, even if the system organizes them as topics in an outline, each of which has e-mail messages posted in chronological order under the relevant topic. Each piece of mail is independent, and has to be opened and closed in separate steps.  No shared workspace exists nor does the opportunity to annotate in context.

Our conferencing system, the latest version of which is called Forum MATRIX, overcomes these obstacles. Developed in our Knowledge Engineering Laboratory at Texas A&M University, our anytime/anyplace conferencing software has many advantages over ordinary web pages, such as allowing each student in a learning team to create, edit, and make in-context links in a group-created web of documents. The advantages for group-based case studies include:

  • Better collaboration—each student supplies input, annotation, and links to shared documents.
  • Ease of use—students create their input to the group pages directly, from menu choices. They are enabled to create all input and annotations as Web pages, without having to know anything about Web-page editors or Web-site management.
  • Hyperlinking—students make in-context links, to World Wide Web sites, other documents in the MATRIX database, or topop-up "sticky notes." Sticky notes are part of the document, and they do not have to ship back and forth to the server. These are especially useful for short comments, questions, and answers.
  • Preview of linked topics—students and teacher preview which links to follow because topic statements could be displayed without actually opening the note.
  • Easy administration. Assuming that the institution has a Web site and Webmaster, the conference leader (teacher) and students control everything about the conferences from pull down menus. This includes forming groups, creating and editing documents, and assigning access permissions to given documents.
  • Powerful access control—multiple groups can work on the same article, and each group's analysis can remain secure until the deadline for completion. Then, the permission settings can be changed so that each group can see, but not edit, the work of the other group.
Interested readers can get the software from the following Web site: www.foruminc.com. Instructions are posted for linking to a demo site that provides several activities for visitors. Also a series of publications on computer conferencing and group learning can be found at website: www.cvm.tamu.edu/wklemm/collab.htm .

Experiencing the Analytical Model

The model has been tested over four semesters in my undergraduate Honors class in Introductory Neuroscience. Although Texas A&M is a residential campus, the enrollment is high, and the many extracurricular activities and student work schedules often make face-to-face meetings difficult. Most students had never been required to evaluate original research literature before enrolling in the class. The biomedical science courses are typically focused on presenting the facts of science rather than on the process of science. In the first semester, two groups of five students each performed case study analyses on four contemporary journal articles. In the second semester, three groups of five students each performed case studies of three journal articles. Most recently, four groups of  six students each completed case studies of two journal articles. One article is assigned in week 4 of the semester, and the other in week 9 The reduction in number of papers occurred because students complained about excessive workload (Klemm 1998).

In accordance with generally accepted practice (Cuseo 1996), I made each group heterogeneous to maximize the breadth of skills and academic background available for the group. Each student in the group was instructed to help the group as much as they could with their existing knowledge and with information learned while conducting research. In the process, whenever someone raises a question, someone else in the group tries to clarify issues. Links to separate documents were made for short essays, but the pop-up "sticky notes" could be read immediately. Delays were minimal because these notes do not require communication with the fileserver computer. Rarely did a student take advantage of the opportunity to link to relevant World Wide Web sites, although everybody seemed to like it when this happened. For each case study, a group grade provided incentive to do good work , as defined in the explanation of what was expected for the main components of the exercise ("understand, assess, integrate, and create"). Work quality distinctly improved with each successive journal article assignment, as students learned how to help each other. More extensive interactions occurred when the assigned articles dealt with topics that were of personal interest to the students. Commonly, biomedical science students are interested in such topics as drug mechanisms, well-known neurological diseases, sexual behavior, and emotions.

Results with the first two classes led me to believe that student performance could be improved if I gave them more specific guidance on what to look for in the journal article. As a result, the latest class received a set of open-ended questions in the hopes that this would facilitate better insights and more rigorous analysis (Figure 2). I put the questions in the FORUM conference, and students could use as much space as needed to respond to them. Adding these explicit questions greatly improved the quality of student work. This was especially noticeable on the first journal article, which has always been the most difficult for students because this learning activity is new to them.

Figure 2. Questions to assist students with research

Understand

1. Indicate and define terms you do not know

2. Can you explain the theory and rationale for the paper?

3. How do the techniques used work?

Assess

1. Do authors have a bias? Has it prevented them ftom considering other alternatives?

2. Is the literature cited pertinent? Are there gaps that should have been addressed?

3. Do the methods and procedures seem appropriate? …sensitive?

4. Are there variables that are not controlled?

5. Is the magnitude of test effect statistically significant? ...practically important?

6. How strong is the evidence for the questions asked and interpretations given?

Integrate

1. How do these results agree with what you already know? …what seems reasonable?

2. Can the results be interpreted in other ways not considered by the authors?

3. What is value and relevance of the results to the concepts and theories involved? Is the research important—if so, from what perspective?

4. What new research (not suggested by the authors) might these lead to? Does the article raise new issues?

Create

1. What questions arise that were not considered by the authors?

2. Can you conceive of any new or different concepts or principles?

3. How would you speculate on the implications of this research?

4. How would you synthesize these results and other information into new ideas and insights?

5. What hypotheses, not foreseen by the authors, could be generated from this study?

 

Each group used a slightly different approach to linking commentary. As with Web pages, there are two basic options: using key words as link anchors, which lead to separate pages for student input and dialog; or using the key words as bookmark anchors, where all the materials for a given case are in the same document but accessible quickly by bookmark links. The second option is preferable if users have high-speed network connections that enable rapid, high-speed download of a large file. Student groups that wanted to work off-campus via modem connections preferred the first option because it divided the case study into smaller pieces.

Perhaps even better work could be achieved if specific roles were assigned to each member of the team. The group could assign roles such as leader, librarian, motivator, interaction facilitator, and on-task monitor." This approach would motivate every student to be familiar with the analysis of all aspects of a research paper, and make certain duties explicit for each team member.

What do the students say about this experience? Many avoid the course, knowing that it is different from what they are comfortable with and perhaps more demanding than many other elective courses. But those who do take the course give me rave reviews, which I never received using traditional modes of teaching. Word seems to be spreading about the value of the course. This coming semester, the class is full and a waiting list has been created.

Asynchronous computer conferencing avoids many of the problems with face-to-face group work and provides a powerful way to conduct journal article analyses. This analytical model is applicable to all academic disciplines and can enable rewarding case-study learning experiences that develop student competence and confidence in coping with primary scientific literature.

References

Cuseo, J. B. 1996. Cooperative Learning: A Pedagogy for Addressing Contemporary Challenges & Critical Issues in Higher Education. Stillwater, OK: New Forums Press.

Fortner, R.W. 1999. Using cooperative learning to introduce undergraduates to professional literature. Journal of College Science Teaching 28:261-265.

Greene, J. 1991. Making students think. The Teaching Professor 5(8): 1.

Herman, C. 1999. Reading the literature in the j argon-intensive field of molecular genetics. Journal of College Science Teaching 28:252-253.

Janick-Buckner, D. 1997. Getting undergraduates to critically read and discuss primary literature. Journal of College Science Teaching 27:29-32.

Klemm, W.R. 1996. Understanding Neuroscience. St. Louis: Mosby.

Klemm, W.R. 1998. New ways to teach neuroscience: Integrating two teaching styles with two instructional technologies. Medical Teacher 20:364-370.