Learning Versus Performance:
A Review and Critique of “Co-Evolution of Technological
Design and Pedagogy
in an Online Learning Community”
Chris Dent, L509 Final Paper
December, 2002
“Co-Evolution of Technological Design and Pedagogy in an Online
Learning Community” is the weighty title of a paper by Amy Bruckman researching
uneven learning performance in a constructionist online learning community
known as MOOSE Crossing. The paper describes quantitative measurement of
learning performance in the community, shows that performance is uneven and
proposes a possible solution to the uneven performance. The research, though
quite interesting and utilizing solid analysis, fails to adequately explain the
results, insufficiently explains the source of motivation in the learners, and
fails to test the provided solution. This review will summarize the research,
point out some of the flaws and suggest future research that will address some
of the failings.
The paper asks two questions:
The paper attempts to answer these questions by describing
how quantitative analysis of learning performance on MOOSE Crossing led to new
thoughts regarding the nature of motivation in learners using the community.
The important concepts in the MOOSE Crossing analysis are
well described. MOOSE Crossing itself is a text-based MUD (Multi-User
Dimension/Dungeon) where students are invited to learn creative writing and
computer programming by creating a consensual space in which they can
communicate and share objects they have created. A key aspect of the MOOSE
Crossing environment is that it was has been designed, from the outset to be a
constructionist learning setting. Constructionism, envisioned by Seymour Papert
is developed from Jean Piaget’s constructivism, extending that model of
learning being fully in the minds of the learner to the idea that learning is
literally in the hands of the learner: learning is facilitated by the
construction of external things.
Constructionism is important at MOOSE Crossing because the
online community facilitates many of the constructionist ideals. The entire
site is made up of programmatic objects that have been created by other
community members. These objects provide a library of examples, each with an
author that may be contacted as a readily available informal mentor.
Constructionism is a problem at MOOSE Crossing because there,
as in many settings, it has shown excellent initial results that have not been sustained
over the lifetime of the learning activity. It has been argued that this is due
in part to the reliance on self-motivation and the lack of structuring guidance
present in constructionism.
When researchers at MOOSE Crossing sensed uneven performance
in the environment they chose to measure learning performance. To ease their
study, learning performance was defined as computer programming performance and
did not include the more difficult to assess creative writing performance.
Computer programming performance at MOOSE Crossing was
measured through a portfolio scoring method. Two judges reviewed the collected
programming work of each of the participants in the sample and scored the work
based on a scale of 0-4. If the two judges did not agree, a third judge
reviewed the work, scored it and then reached a compromise score by reviewing
all three scores. The scale was:
50 users out of 803 were randomly selected. 23 were girls and
28 boys. All were students less than 18 years of age introduced to the system
either directly or indirectly through their schools.
Summary statistics were generated from the age, window of
system use, number of commands typed into the system (used as an indicator of
time on task), number of scripts created, and the judged portfolio score. That
information is displayed in the table below.
|
Summary Statistics |
Minimum |
Maximum |
Median |
Mean (std. dev.) |
|
Age |
7 |
17 |
12 |
12 (2.3) |
|
Period of Participation |
7 minutes |
4 years, 1month |
3 months, 25 days |
9 months, 12 days (1 year,
1 month) |
|
Commands Typed |
6 |
51,850 |
788 |
6,638 (11,958) |
|
Scripts Written |
0 |
234 |
2 |
19.5 (43.3) |
|
Portfolio Score |
0 |
4 |
1 |
1.36 (1.47) |
(Bruckman)
The summary statistics reveal that a small group of motivated
students are skewing the results to cause relatively high mean participation
and performance. Low median scores and high standard deviations show that the
means are not descriptive of the group. As expected, there is an uneven level
of achievement.
A Mann-Whitney test shows that previous experience has a
significant (p<0.05) influence on programming performance. A Mann-Whitney
test is required in situations where a t-test is desired but the original
population is not normal and the values can not be considered to be part of
continuum (Lowry).
The researchers conclude that some of the participants are
learning a great deal, but others are not. Such uneven performance is
considered undesirable. They choose to develop a system of merit badges to help
structure learning and motivate the students. The merit badges are achieved
with the help of a mentor in the community and by review of a board.
The researchers also conclude that informal suppositions need
to be confirmed with quantitative analysis. This is especially true in
situations where the researchers are closely involved with the participants:
regular association with the participants can skew perceptions.
Bruckman’s research is well structured and well conceived. The
quantitative work is a sound analysis of performance. The flaws in the research
occur perhaps as a result of an over eagerness to gain as much understanding
from a limited research project as possible. The central problem is that the
research does not directly answer the questions posed. Portfolio scoring of
programming performance—without a baseline for comparison and in a situation
where previous experience has been shown to be relevant—only describes
performance, it says nothing concrete about learning.
Further, the previous experience that is shown as an
influence of performance is nowhere quantified or recorded in the research. It
is only casually mentioned and thus not worthy of complete trust. There are
many types of programming (or even other) experiences that may impact
performance. If previous experience is a relevant factor, knowing what the
previous experience is would be helpful in determining solutions, such as the
merit badge system.
What is the genesis of the merit badge idea? There is no
apparent connection between the analysis of the uneven performance and
establishing the merit badge system. That idea could have come along without
the analysis; in fact it is not made explicit that it did.
How do we know the merit badge is an effective solution? The
researchers interview two long term participants to get a sense of their
impressions of the new system: They find it interesting and potentially
effective. However these participants are already acknowledged as strong
participants in the system. Their participation, performance or learning has
not changed.
Finally, it must not be left unsaid: the implementation of
merit badge system into the originally constructionist environment of MOOSE
Crossing introduces a subtle competition into the environment, competition
which is in some ways anathema to the model. The researchers attempt to address
this, but their efforts are weak. If iterative design processes are in fact
valuable in adjusting pedagogic theory it seems in this case they have a rather
radical effect.
The apparent mix in this paper of analysis and a work in
progress is jarring. Both the merit badge system and the hypothesis that
iterative design impacts pedagogy are left in an untested state. Further
research should be done to test the effectiveness of the merit badge system as
a learning aid; not solely as a performance aid. To establish learning is
present there must be a change in understanding. Such a chance might be
measurable through a change in performance.
The current results can be used as a baseline for comparison
in future analysis. After a window of several months has passed, another sample
of students can be selected and the same portfolio scoring done. It may be wise
to quantify previous experience in some fashion. In the analysis the number of
merit badges a participant has achieved can be tested (with another
Mann-Whitney test) to determine if that value influences programming
performance. Do the participants with more merit badges have stronger
portfolios? In addition any difference in the mean and median performance and
participation values between the two studies, before and after the introduction
of merit badges, will reveal if the badges have had any impact.
Only after making this comparison can we know if learning has
been encouraged on the system and if the educational changes inspired by the
study were chosen wisely.
Bruckman, Amy. (2002). Co-evolution of technological
design and pedagogy in an online learning community. Retrieved
Lowry, Richard. (2002). The Mann-Whitney Test. In Concepts and applications for inferential
statistics. Retrieved