Distributed Creative Cognition In Digital Filmmaking
Nicholas Davis, Boyang Li, Brian O’Neill,
Mark Riedl
School of Interactive Computing
Georgia Institute of Technology
Atlanta, GA 30308 USA
{ndavis35, boyangli, boneill, riedl}@gatech.edu
Michael Nitsche
School of LCC/ Digital Media
Georgia Institute of Technology
Atlanta, GA 30308 USA
michael.nitsche@gatech.edu
ABSTRACT
This paper reports on an empirical study that uses a
Grounded Theory approach to investigate the creative
practices of Machinima filmmakers. Machinima is a new
digital film production technique that uses the 3D graphics
and real time rendering capability of video game engines to
create films. In contrast to practices used in traditional film
production, we’ve found that Machinima filmmakers
explore and evaluate ideas in real time. These filmmakers
generate vague and underspecified mental images, which
are then explored and refined using the real time rendering
capabilities of game engines. The game engine assists the
filmmaker to fill in indeterminate details, which allows
creative exploration of scenes through playfully
experimenting with parameters such as camera angle and
position, lighting, and character position. Creative
exploration distributes the cognitive task of evaluation
between the human user and the Machinima tool to enable
evaluation through exploring possible scene configurations.
Author Keywords
Creativity, Digital Filmmaking, Creative Cognition
ACM Classification Keywords
H.5.m. Information interfaces and presentation (e.g., HCI):
Miscellaneous.
General Terms
Experimentation, Design
INTRODUCTION
This paper reports on an empirical investigation of the
creative practices of a relatively new digital filmmaking
technique called Machinima. Machinima is a digital film
production technique that uses the 3D graphics and real
time rendering capability of video game engines to create
films. It has been defined as “animated filmmaking within a
real-time virtual 3D environment” [12]. Machinima
filmmakers, or machinimators, are able to script character
actions and camera sequences to create a visually robust
animated film with relative ease. The genre has become so
popular in recent years that many computer games, such as
The Sims, The Movies, and Grand Theft Auto are released
with tools and interfaces to provide fine-grained control of
their 3D game engines to create Machinima. Likewise,
more generic game engines including Epic’s Unreal
engine, Valve’s Source engine, and Crytek’s CryEngine
ship with improved Machinima production tools and
editors. Machinima gradually evolved from a technical and
esoteric manipulation of video game engine code to more
user-friendly and widely accessible graphic user interfaces
designed exclusively for making films [13]. This has led to
specific Machinima engines that completely focus on the
cinematic elements and reduce the gameplay mechanics, as
is the case with Moviestorm and iClone.
Currently, little is known about how these new techniques
have affected the creative process of machinimators. We
conducted a study of the creative practices of five expert
machinimators through a qualitative methodology called
Grounded Theory. Using this approach, we interviewed
experts and coded transcripts in an iterative process that
exposed common creative practices.
The contribution of this paper is a novel model, Distributed
Exploratory Visualization that describes the creative
practices of Machinima digital filmmaking. Our analysis
indicates that the real time rendering capabilities of
Machinima tools enable filmmakers to distribute part of
their creative process onto these tools by creatively
exploring and evaluating images through active
manipulation. Consequently, our model elaborates on the
concept of Distributed Creative Cognition [6] by blending
the generation and exploration cycle of the Geneplore [2]
account of creativity with the basic premise of distributed
cognition [5]. Distributed Cognition proposes that humans
offload cognitive work onto technology to coordinate
action, and Distributed Creative Cognition proposes that
humans offload cognitive work in during creative ideation.
In the remainder of this paper, we first consider the
development of Machinima and its role as a medium. Next,
we will describe our study and summarize our results. In
the discussion, we will define Distributed Creative
Cognition in detail. We will then reflect on our findings
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. To copy
otherwise, or republish, to post on servers or to redistribute to lists,
requires prior specific permission and/or a fee.
C&C’11, November 3–6, 2011, Atlanta, Georgia, USA. Copyright 2011
ACM 978-1-4503-0820-5/11/11...$10.00.
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise,
or republish, to post on servers or to redistribute to lists, requires prior
specific permission and/or a fee.
C&C’11, November 3–6, 2011, Atlanta, Georgia, USA.
Copyright 2011 ACM 978-1-4503-0820-5/11/11...$10.00.
and offer some design recommendations based on the
technological strengths and limitations of current
Machinima tools. Finally, we conclude with a summary of
our contributions and findings.
BACKGROUND
Historically, the technological development of Machinima
has been traced back to the early hacker and demoscene
[19] and the practice of game modding and demo
recording. Lowood traces the history of Machinima and
identifies the video game DOOM as being a critical catalyst
in its development [10]. Id Software, developers of the
DOOM engine, allowed gamers to make modifications, or
‘mods’ of the main game. A community of practice evolved
as hackers shared their experiences and manipulations [1].
This included manipulations of so-called demo files that
allowed players to record and play back gameplay in the
game engine. Using these demos, players started making
recordings to demonstrate their skills, which eventually led
to the narrative based Machinima productions that many
people are aware of today.
Early on Machinima was recognized as an emergent form
of play [15] and artistic practice that evolved from a game-
like activity. Game companies realized the potential in
crafting a cinematic piece from a gameplay-based engine.
Stunt Island (1992) – a pioneer engine for Machinima
creation – offered little more than a staging ground for
virtual stunts that could be “filmed” from a range of
variable perspectives and shared with other players. The
game’s very design makes the distinction between “play”
and “filmmaking” difficult. In parallel, players seem to
easily move between both worlds. Many of the first
Machinima production groups evolved from player
communities, such as the ILL Clan or The Rangers, who
are credited with the first narrative Machinima. The
concept of video game play as a creative practice evolved
as games became more accepted as cultural artifacts;
examples include play as sport and performance [1, 11, 14].
While the growing role of Machinima as evolving art form
in a community of practice is noted, little research is
available regarding how the specific conditions of
Machinima support new forms of creativity.
Because the history of Machinima is closely tied to game
engine development like the evolution of their visual
capabilities and available editors, one can learn from work
on tool development, that reflects the evolution of
expressive range in Machinima [7]. But a technologically
deterministic approach like this lacks an analysis of the
underlying cognitive effects at work during the production
of a Machinima movie. Without denying the role of
technology, tools, and Machinima as a socio-cultural
phenomenon, this paper focuses on those neglected aspects.
Machinima differs from traditional uses of Computer
Graphic Imagery (CGI) animation in film. CGI animation
in film mirrors the traditional filmmaking process of
extensive planning and production to deliver a final moving
image. In CGI, the rendering process can take a very long
time and mistakes are costly. Consequently, in CGI and
other traditional film techniques, a process called
storyboarding is used to plan out films. Storyboarding is
the use of descriptive drawings and sketches that depict
how shots in a scene should appear. A storyboard helps to
optimize the production process as it includes which
characters are in a shot, as well as camera perspective and
shot distance, which define which part of the film set need
to be prepared. To minimize the final cost of rendering,
scenes have to be planned in detail, which leads to a linear
production cycle like that represented in Figure 1.
Large film productions using Computer Generated Imagery
(CGI) have used Machinima as a method of rapid and low-
cost pre-visualization [9, 15]. As a pre-visualization tool,
Machinima is a cheap way to quickly evaluate scene
configurations and technical decisions before full CGI
rendering is committed to. Developing CGI movies is a
high cost and time consuming activity. Machinima provides
an interactive, robust, and rapid pre-visualization tool that
minimizes the risk during the actual production. As such a
tool, Machinima provides a new form of film pre-
production that combines playful exploration with shot
planning [15]. But for a pure machinimators, this process
constitutes not a phase of pre-production but the core
creative activity. Machinimators work continuously on the
final render image as they re-adjust their visuals. This
constitutes a principle difference in production. We argue
that this difference is reflected in the creative engagement.
RELATED WORK
Distributed Exploratory Visualization builds off two key
cognitive models: Distributed Cognition and the Geneplore
model for creative cognition.
Distributed Cognition considers cognition as a feature of
the larger context in which humans exist, including
environmental conditions, socio-cultural practices, and
technological artifacts [5]. Cognition is embodied and
situated [19], which means that thought and action are
dynamically coordinated through interaction with one’s
environment. This ability is leveraged when humans
restructure their environment to aid cognition, such as
spatially arranging important notes so they are immediately
visible. In this example, a memory task is restructured into
a visual representation, changing the nature of the task from
remembering to interpreting a spatial representation. In this
way, representations are propagated through different
mediums to facilitate and optimize cognition [5].
Kerne and Koh [6] developed the Distributed Creative
Figure 1: Traditional Film Process
Cognition framework to describe creative cognition as a
system distributed among social collaboration, tool use, and
mental operations. These researchers used distributed
creative cognition to analyze a creativity support tool called
combinFormation [6]. With this tool, users construct and
manipulate a composition space containing visual images
generated through iterative searches and semantic grouping
procedures. The system employs an intelligent online
search to find items to stimulate new ideas. The researchers
conclude “the mixed-initiative capabilities of procedural
generation and human manipulability of visual information
representation in the composition space support distributed
creative cognition and situated creative learning” [6]. Users
report being able to make new connections and think about
concepts in different ways after being able to see and
manipulate concepts in their composition space. Distributed
Creative Cognition does not provide a specific account for
the cognitive processes underlying creative thinking.
Finke et al. [2] developed the creative cognition approach
to study cognitive mechanisms underlying creative process.
The Geneplore model of creative cognition describes
creativity in two basic phases where individuals generate
vague ideas and refine these mental constructs through
exploration. In the generation phase, the creative thinker
generates what Finke et al. refer to as pre-inventive
structures, which include visual patterns, object forms,
mental blends, category exemplars, mental models, and
verbal combinations. Pre-inventive structures are explored
through specific mental operations in the next phase.
The Geneplore model proposes that creative cognition
cycles between these two phases to gradually refine the
initial pre-inventive structure (See Figure 2). As one
explores the pre-inventive structure, the initial idea is
updated, and the cycle continues until a satisfactory result
is reached. Below we will argue that machinimators report
a similar process whereby they generate vague mental
images and gradually refine these images by exploring
them using their Machinima tools.
However, Geneplore does not account for the fact that, for
machinimators, a large portion of the creative process is
distributed between the system and the self. For example,
concepts and ideas are explored through the capabilities of
the tool, which provide additional information with which
to make informed evaluations. To remedy this shortcoming,
a new framework is proposed that considers creative
cognition as distributed across mental operations and the
technological tools used throughout the creative process.
STUDY DESIGN
In our study, we used qualitative data collection methods to
perform an exploratory study about the creative practices of
machinimators. We conducted five semi-structured
interviews with professional level machinimators lasting
approximately one hour each. We define a professional
machinimator as one who has had a significant impact on
the Machinima community, including film awards from the
community, important roles in organizing community
events, publishing books, and holding classes and
workshops. A questionnaire was distributed before the
main interview to control for experience. The questionnaire
addressed issues such as how many films they had released,
how many hours per week they spend per week making
films, and how long they have been a filmmaker.
The interviews were conducted over Skype in all but one
case where the participant was present in person. The first
half of the interview centered on the general creative
processes and motivations for developing Machinima films.
These questions included details about creative experiences
before Machinima, the Machinima environment they
typically use and limitations they often face, and the
planning and preparation process used when creating films.
This section was meant to identify the overall creative
strategy of the individual and understand their intentions
for making movies. What do they hope to achieve and how
do they achieve this goal? How does the technology shape
their efforts?
In the remainder of the interview, we conducted a film
walkthrough of one of the participant’s films using desktop
sharing technology. During a walkthrough, the interviewer
and director both watched the movie, and the director
explained what he or she was thinking in each of the
scenes. The movie was paused at times to provide
opportunities to elaborate on certain elements. The
filmmaker would point out any technical difficulties s/he
had with the tools in trying to make specific scenes. We
were particularly interested in the technical means
employed to achieve a sense of narrative and emotion in
their films. Technical methods in consideration included
how the filmmakers used scriptwriting, storyboarding,
camera, lighting, sound design, set design, staging and
directing performance, editing, and special effects to
achieve their goals. All interviews were video recorded.
Because of the lack of empirical work on the creative
practices of machinimators, it was important to conduct an
exploratory study to accurately assess the creative
processes of these digital filmmakers, which included their
often rather varied attitudes and interpretations of the tools
they use. We analyzed the transcript data using Grounded
Theory [3], which is a flexible and robust approach to data-
driven theory building. Grounded Theory encourages
iterative hypothesis generation and adaptive data collection
methods that are updated based on new research questions.
In this approach, analysis coincides with data collection,
and transcript data is coded into categories. The features
Figure 2: Geneplore Model of Creativity
and relations of these categories help form hypotheses to
systematically explain the data.
As interview data were analyzed, we introduced new
questions into the subsequent interviews to gather more
data about hypotheses emerging in our Grounded Theory,
while other questions were eliminated due to irrelevance.
Questions that were introduced include those about specific
filmmaking skills and strategies for developing films, such
as: What aspect of filmmaking are you most confident in?
How do you handle a problem in a domain with which you
are inexperienced? What are your favorite and least favorite
aspects of making movies with Machinima? How much of
the story do you typically have worked out before you start,
and in what form does it exist (i.e. mentally, story board,
text, etc.)? These questions helped us understand the
creative process of machinimators in relation to the specific
toolsets they use.
Grounded Theory originated as a method in sociology to
provide a systematic and uniform approach to study
qualitative data. The human centered movement in
Computer Science has recognized the inherent power of
such a method in understanding how users engage with
technology. For example, Grinter et al. [4] used Grounded
Theory to understand the social and technical dynamic of
software development teams. Grounded Theory yields an
explanatory framework that serves to elucidate central
features of the domain in question. It has become especially
relevant to Human Computer Interaction and its related
disciplines that investigate how users interact with systems.
Findings from Grounded Theory can lead to design
recommendations that support naturalistic interaction.
RESULTS
The results of the study are organized around four
categories that emerged as central to Machinima
filmmaking. These categories are (1) mental images, (2)
creation, (3) exploration, and (4) evaluation. The remainder
of this section will go into detail explaining each of these
categories, cross-referenced with the data collected from
the interviews. To ensure the anonymity of filmmakers, we
have coded the filmmakers with letters A-E.
Mental Images
The content of mental images used to create films vary
depending on the current intention, creative practice, and
task of the filmmaker. The most basic dimensions along
which mental images vary are their level of abstraction and
the extent of their precision. The visualizations of
filmmakers contain different levels of narrative abstraction
that vary along a spectrum ranging from the most abstract
global structure of narrative events, to less abstract
individual scenes, and finally to concrete features in those
scenes. The content of the mental image in each of these
layers of abstraction can be more or less determinate. A
machinimators might have a clear image regarding the
setting and lighting of a shot but that image might lack
details in staging and framing, for example. Precision
varies along a spectrum from none, to a vague requirement,
and finally to a more precise object or element that is fully
represented in the mental image.
Examples from each of these two dimensions will help
elucidate the distinction. The most abstract types of mental
images are those that pertain to the entire narrative. Usually
developed in the beginning of the creative process, these
ideations may include some sort of notion about the order
of critical narrative events and the overall aesthetic style for
the movie. Filmmaker A, for example, defines a general
notion for a kind of style s/he wants for the movie:
A: I remember seeing some videos of Unreal 2003 and
some of the stuff there looked just photorealistic to me
back in the day. But still I felt like that never really
translated to animation, and so very early on I decided
that it would be really important to have a layer of
abstraction
Filmmaker A has certain parameters for the way s/he wants
the film to appear, but these are rather vague. In reality,
s/he defines what not to do rather than explicitly stating
what to do. This is an example of an abstract mental image
that provides a set of guidelines of imagistic requirements
rather than one specific value. The director goes on to
describe the process by which s/he started working:
A: It’s not that I actually wrote it down. It’s just that I
had this thing in my idea, and I had an idea, and I had
some pictures in my head, and I just started working.
The environment in which the machinimator is working can
also influence aesthetic style. Oftentimes, directors want to
the push the capabilities of a game engine and impress their
audience.
B: The entire series was constructed from the start
because I knew I wanted to make a fancy film from the
Neverwinter Nights engine as an experienced
Machinima creator.
C: Shadows hadn’t really been used this way in
Moviestorm before this movie as far as I know…I just
thought that was artistically quite nice, and it was a bit
more creative than just having the camera on the main
characters.
These aesthetic motivations set up an overarching aesthetic
priority that is applicable to the entire narrative. However,
this priority does not mean one specific thing; it is a vague
notion of a general way they want the movie to look.
Mental images of a particular scene are less abstract than
overarching narrative images. Filmmaker C describes a
method for approaching the production of a scene:
C: For the scene I will sometimes have in my mind like
the money shot basically. There will be one particular
point in the scene I want the camera to be low looking
up and a close up of the guy’s face and it’s going to
zoom behind him and we’ll see someone come through
the door, so I’ll have it in my mind quite clearly that
there’s going to be certain angles and scenes that I’ll be
looking to play out, but I don’t, I certainly don’t draw it
on paper or anything like that because I’m not really
terribly good at drawing.
In the filmmaker’s mental image, s/he is focused on an
individual scene with a relatively high level of precision
because s/he is actually picturing individual shots and what
is included in those shots. The most concrete type of mental
image is that which is focused on individual objects in a
scene, and even this type of mental image can have
imprecision. For example, filmmaker D became concerned
about the lighting in a particular room, which led to the
inclusion of a candle in the room.
D: Something that’s always bugged me about TV in
particular… somebody would turn off the lights and
you’d see these other lights come on because obviously
they can’t have it pitch black, so there’s these other
lights that don’t emanate from anywhere, its just the set
lighting kicking in. Even when I was a kid I jarred by
that…it just reminds you you’re on a set…I wanted to
have very little lighting in this set where lighting was
very important to what I was going to convey. I wanted
the light to have a source, and I figured after what just
happened in this room, the guy is not going to have all
the lights on, he’s not going to have all the lamps turned
on, but there has to be light coming from somewhere,
so the candle is just what was available and seemed like
the good way to go.
In this case, the individual’s mental image consisted of a
vague requirement for one concrete aspect of a scene,
namely the lighting. There was a certain way that D wanted
the lighting in particular to behave. The mental image
contained a guideline of what a solution should look like,
and after checking what was “available” in the game; a
candle was deemed a proper solution.
Figure 3 depicts how the properties of mental images vary
along the dimensions of narrative abstraction and precision.
When mental images are precise, they deal with concrete
visual qualities, while the imprecise end of the spectrum are
visually barren and corresponds more to the verbal and
procedural information specified in the story.
Creation Practices
The capabilities of the video game engine do not always
align with the desires of the machinimator. As a result, they
often face technical difficulties in trying to craft elements
of their film. These problems can arise from limitations in
the tool itself, or a lack of knowledge about either the tool
or film practices.
The game engines used to create Machinima films are often
poorly documented. Online forums have grown up around
each of the game engines, but the knowledge generated on
these forums is fragmented and incomplete.
E: Knowledge can be a funny thing within the
Machinima community because it will be that half the
people will know something and the other half don’t.
And the half that know it don’t know that the other half
don’t. It just seems normal and natural to them. And
that’s a feature of being self-taught…if you are self-
taught you don’t know what the bases are, and you
don’t know if you’ve missed them.
The fragmented knowledge can be traced in part to the way
in which problems are solved during Machinima creation.
If the solution to a problem is unclear, directors translate
that problem to a domain they do know in order to solve it.
This is accomplished through what has been referred to as a
‘work-around’ or ‘hack.’ Depending on the background of
a director, s/he will revert to familiar techniques to try to
solve unfamiliar problems.
A: Whenever I encountered a creative decision
problem, usually my first approach was: can I somehow
work around it with the camera? Like can I hide it? Can
I create [or] do editing in a way that conveys that?
It often turns out that these workarounds help overcome
glitches or bugs in the video game engine itself. For
example, character animations sometimes refuse to execute
properly, resulting in characters moving through solid
objects. As a result, the director will have to direct the
camera away from the action at that point in order to hide
the bug. Although each of these scenarios represents an
individual case, the filmmakers have generated heuristics
for approaching technical difficulties, and these methods
usually rely on a familiar skill based on their knowledge of
the 3D engine that is applied in a creative manner.
Exploratory Practices
The Machinima environment is used as a tool to
experiment with and play with different ideas because of
the real time rendering capabilities of the game engine.
E: One of the really appealing things about Machinima
is that you can experiment quickly…you don’t have to
wait to see what you’ve done in many things.
Being able to explore scenes encourages a sense of play,
which impacts the overall creative practices of these
directors. From the outset, the directors have a playful
attitude in which they experiment and explore in order to
Figure 3: Properties of Mental Images
develop their ideas. This is in sharp contrast with the
practice of storyboarding in conventional filmmaking.
Storyboarding is the use of descriptive drawings and
sketches that depict how shots in a scene should appear.
Filmmaker C describes how he plays around and
experiments with different ideas for sets in the beginning
of his process as a replacement for storyboarding:
C: I don’t storyboard it really, although…before I’ve
got the script I’ve got an idea as to what the film is
about, I’ll quite often play around in the software and
build a few tests, sets sometimes. Usually, in fact, they
don’t make it into the final movie, but they let me play
around if I’ve got an idea for how I want to use new
textures or a different lighting, or a different way
maybe of framing shots with the camera. I’ll play
around and from that I’ll get an idea of what’s going to
be possible in the movie.
Exploring scenes in this way has the potential to facilitate
emergent creativity. The game engine allows one to see the
scene in a way that cannot be imagined. There is no way
for the mind to actively hold all the details of a scene in a
conscious image [8] and perform the kind of movements
that the Machinima tools do. Thus, filmmakers often make
creative discoveries by exploring the scenes in real time.
E: The way that the camera pans in under that beer tap,
again, that was just a happy coincidence because
originally the camera wasn’t there… it was when I was
building this set and I was getting the other beer tap
ready I suddenly caught an angle, and I just thought that
beer tap looks fantastic with the light just on one side
around the actually pump and it just looked really good.
So that gave me an idea that actually what I should do
is start off close up around the tap and then move down
the bar to show the characters… had I story boarded
this in any detail I probably wouldn’t have had this shot
because it wouldn’t have occurred to me. Its more like
having built this set its kind of like I’m now starting to
look around the set that I’ve built and think of how I
might actually use it.
Filmmakers C and E explored the set to refine their ideas of
how the scene should appear. The sets provide a good
mechanism by which to simulate different possibilities for
the final film and in production quality. It is used as a tool
to think. One may even say that they are using the game
engine as a kind of temporal sketching mechanism to work
out visual concepts and moving image ideas.
Evaluation Practices
Exploring scenes in real time also enables directors to
evaluate these explorations in real time. Cinematic ideas
such as camera movements and cuts can be rapidly tested
and evaluated on the screen. It is very beneficial to the
creative practices of these individuals to receive immediate
feedback on the decisions they make in the Machinima
environment.
E: I always find being able to see something is very
useful…the kind of what you see is what you get is one
of the things I really like about Machinima. That’s kind
of different from the classic CGI where you have to
imagine what you’re going to get…Being able to adjust
things in real time, so you can just move something and
change it, so you can see what the difference is, that
seems to be a theme for me, in the most popular tools
that people use.
Explorative evaluation of this nature can reveal tensions
that require solutions. Broadly speaking, tension can come
from an aesthetic or narrative discrepancy between what
appears in the picture in the mind and what appears on the
screen. Filmmaker C describes how the set that was created
as a reference to the original A Clockwork Orange (1971)
does not align with the narrative s/he has crafted:
C: The reveal of the characters, of this gang, wasn’t
working at all. In A Clockwork Orange …the opening
scene…we see them in this nightclub and…the camera
pulls back and we meet Alex and so I was trying to an
extent recreate that without it being a direct copy and so
I had this nightclub and I had all of the characters and
lots of space in it, and I just thought, ya know what, In
this dystopian world I’m working in, which isn’t the
same as… A Clockwork Orange, you wouldn’t have
nightclubs you would have little run down grotty little
places…I think I felt as I watched what I was working
with it just didn’t fit with the story and so that was the
source of my dissatisfaction. So it didn’t matter what
sort of camera pans and zooms or whatever I did, I just
wasn’t getting the result I was happy with, and so I just
thought I just tore it up and started again with this new
bar.
This example elucidates a situation in which there are both
narrative and aesthetic discrepancies. At the scene level, the
author has a clear idea that s/he wants the set to look like
the bar scene from A Clockwork Orange. When the scene
image was only present in his/her mind, s/he was not able
to realize it conflicted with the goals of his/her narrative.
However, s/he realized there was a problem once the scene
was built because s/he was able to look around the final
virtual set, evaluate it and realize that ‘it just didn’t fit with
the story.’
C employs the typical creative exploration techniques of
playful manipulation to try to overcome what s/he thinks is
an aesthetic discrepancy. ‘Hacks’ and manipulations, such
as tweaking the camera pans and zooms, can help to work-
around aesthetic discrepancies. However, in this case s/he
noticed the more fundamental problem once the entire set
was built. The problem was a narrative discrepancy
between the set and the overarching narrative. It was only
after s/he he ‘watched what [s/he] was working with’ that it
became apparent that there was an actual narrative
discrepancy. Even after this was realized, there was a
general reluctance to restart the entire set because so much
time and effort had already been invested in this set.
DISTRIBUTED EXPLORATORY VISUALIZATION
Machinima offers immediate feedback on any creative
choice made. This distinguishes it from other forms of
traditional filmmaking. The real time rendering offered in
Machinima supports distributed creative cognition by
encouraging ideas to be developed through exploration and
evaluation inside the tool. The data reveal a cyclical pattern
in the creative process of machinimators in which they
generate vague mental images and explore those images by
playing and experimenting with their tools.
Psychological theories of the creative process, such as
Wallas’ Stage Model [1] of creativity (preparation,
incubation, illumination, and verification) do not account
for the technologically mediated aspect of creativity this
study reveals. Specifically, the data reveal a cyclical pattern
in the creative process of machinimators in which they
generate vague mental images and explore those images by
playing and experimenting with their tools. We explain our
results through a new framework Distributed Exploratory
Visualization, which elaborates on distributed creative
cognition [6] with an account of creative cognition inspired
by the Geneplore model [2]. Our model is unique in that we
consider a practice-based and emergent creative process in
a pragmatic context.
Our findings show that the immediate feedback and
rendering offered through video game engines has made
on-the-fly evaluation more practical than elaborate pre-
planning. In Machinima, we see evidence of situated
cognition and action, whereby plans are dynamically
developed and updated on the fly [17]. In this context,
situated action leads to serendipitous creative discoveries.
Instead of planning scenes out in detailed sketches,
machinimators describe a process in which they begin their
projects by developing a script and creating mental images
of scenes and refine them throughout the production.
Machinima software supports this creative process because
the indeterminate features in the mental image can be
defined by playfully exploring the scene. The filmmaker
may rapidly navigate through the scene using various
camera positions and angles, lighting, and staging. In fact,
the fluidity with which scenes are explored has led to
creative discoveries that may have been overlooked if
scenes were explicitly defined in storyboards. In other
words, ambiguity has led to an increase in creative
potential, which agrees with an assertion in creative
cognition that states that ambiguity in pre-inventive
structures often promotes creativity [2].
Mental images are cyclically generated and explored by
playfully engaging the video game engine. In this context,
the Machinima tool can be used to think about and
elaborate vague images generated in the mind of the
filmmaker. It is not necessary to painstakingly articulate
each detail before a scene is shot. Thanks to affordable real
time 3D engines, more effort can be spent evaluating the
aesthetic value of a final scene configuration. Moving
around the scene and interacting with lighting and camera
settings can provide additional resources to help evaluate
the aesthetic quality and narrative accuracy of elements in a
scene. As a result, real time play and on-the-fly decision
making have become central elements to the creative
practices of successful machinimators.
The Geneplore model of creative cognition accounts for the
generation and exploration component of creativity, but it
does not consider how creating things in the real world
affects the creative process. As a result, the Geneplore
model has been expanded to consider the distributed
properties of creativity. Filmmaking requires the use of
external representation, and in Machinima we see a
scaffolding of tools to rapidly generate and test various
permutations of scene configurations. To account for the
importance of externalizing mental content and evaluating
that content, two additional components have been added to
the Geneplore model: creation and evaluation.
Our model, Distributed Exploratory Visualization, has four
central processes, structured creativity, creative
exploration, explorative evaluation, and comparative
evaluation (see Figure 4). Each of these processes will now
be presented in detail.
Structured creativity
Structured creativity is the creation of elements in the
Machinima environment based on the director’s envisioned
outcome. It is a goal-directed activity that strives to make
the images on the screen match the image in the head of the
author. The system has rules and limitations that constrain
which objects can be created and how those objects will act
within a scene. Additionally, the author may be lacking the
knowledge or skill required to achieve some goal. In this
stage the author uses the resources available to him/her to
create an object based on a desired goal. Difficulties in this
phase are typically technical in nature and are often solved
through a work-around.
D: When I engaged the animation to have him turn
around and open that door…it did this weird thing
where his body like twisted and went down through the
floor, just a bizarre bug, and I couldn’t fix it without
scrapping the whole scene, so I decided ok I’m just
going to start in just a little bit tight, and I’m going to
let the sound tell the story of him opening the door, and
the mind fills in that gap, and it ended up working out
just fine.
Instead of directly solving the technical difficulty, the
problem was translated into the domain of camera and
sound, and D employed a solution that would implicitly
achieve the same narrative effect.
Creative Exploration
Creative Exploration occurs after some form of an object
has been created in the Machinima environment. Although
the Machinima environment can be restrictive in some
respects, such as a limited array of objects and animations,
it is incredibly rich in others, such as texture details,
nuanced lighting schemes, a robust physics engine, and
many camera angles. These tools can be adopted to help the
object in consideration match the intended goal of the user.
C: If I’m not getting the results on screen that I want
then I’ll start playing around with the lighting…because
you can totally change the feeling of a scene by
repositioning the lighting, changing the intensity of
lighting, adding multiple light sources
This part of the creative process is distributed between the
mind of the individual and their interaction with the tool.
New ideas can emerge while playing around with the
configurations of a scene. While creatively exploring the
possibilities offered by the tool, directors may actually
make creative discoveries, such as new camera angles or a
new use for a familiar feature. For example, A discovered
an emergent feature in the game engine Unreal Tournament
that ended up significantly impacting the rest of the movie:
A: I noticed that with the particle system in Unreal
Tournament, and this was going a long way in my
future work…you can attach it to a moving object, so
that the particles stay static but the object moves, so
basically what you do is that you have the object paint
its way down.
Exploring the physics engine enabled a creative discovery
that would have been overlooked if the process were not
flexible and open to creative exploration. The same is true
when D created a set and discovered a new camera angle
by looking around in the scene. Creative discoveries are
made possible because the filmmaker’s creative process
balances vague requirements and creative explorations
that refine and add to mental images.
Explorative Evaluation
Explorative evaluation is an evaluative process in which the
real time rendering capabilities of the system are used to
navigate through a scene to gain additional information,
therefore helping to make a more informed evaluation.
Mental images can only contain so much detail [8].
However, when the scene is visually present, the filmmaker
is able to gain useful information that provides a clear idea
about how the final image is going to look.
E: The actual messing about when you’re in it, the
creative play element, where you can immediately see
what you’ve done and whether it works or not. I think a
lot of people really find that is one of the biggest things
that really appeal to them about this way of working.
You don’t have to wait and see what you’ve done in
many things.
Whereas creative exploration had the intention of
experimenting with the tool in order to create, explorative
evaluation uses camera angles and positions to navigate in
a scene and to gather more information for evaluation of
the aesthetics of the image. The immediate feedback
offered through Machinima tools enables rapid evaluation.
Comparative Evaluation
Comparative evaluation is a different type of evaluation in
which the current object is compared to mental images
residing on different levels of narrative abstraction. It is
largely a mental activity, but it draws upon the information
gathered through explorative evaluation. An element
created in the virtual studio may be aesthetically appealing,
but it could very well conflict with the overall goal of the
scene or narrative, like when C created an entire set only to
find out it did not match his narrative.
DESIGN RECOMMENDATIONS
The Distributed Exploratory Visualization model describes
how machinimators rely on their tools to creatively explore,
evaluate, and refine vague mental images. As such,
Machinima tools should strive to keep the cost of creative
exploration and explorative evaluation as low as possible.
The cost of exploration and evaluation is proportional to
the amount of time reqiured to create an element in the
Machinima environment because something has to be
created on the screen before it can be explored and
evaluated. The quality of the evaluation increases with the
detail of the object being created.
When creating an element, evaluation tends to be focused
on the aesthetics of the element itself, rather than how it
relates to the larger narrative. This is especially true in long
creation tasks such as sets and custom objects and
animations. In these cases, a significant amount of time
elapses before the element can be explored and compared
to overall narrative. For example, Filmmaker C had to build
an entire set before s/he realized it did not mesh with the
feeling of his narrative.
The cost of creating a scene should be kept as low as
possible to encourage rapid production and a low cost of
complete restart in the early stages of the creation process.
Machinima productions often use pre-build scenes provided
by the game developer, but the cost of creating an entire
scene can be very high. Committing time and resources to
scene fabrication prevents a filmmaker from wanting to
throw a scene away even though it simply does not work,
and it also prevents rapid evaluation because the majority
of a scene has to be built in order to accurately and
effectively evaluate it. A system that aims to support
creativity in digital filmmaking should offer possibilities to
help rapidly and automatically prototype a very rough
sketch of a scene. The user should be able to set some
parameters to generate a rough scene very quickly so as to
encourage evaluation early on in the creation process for a
particular item.
The different types of mental images, namely those focused
on the narrative, scene, or individual object, are connected
in interesting ways. Narratives are composed of many
scenes, which have a pluarity of objects within them; each
layer is interconnected and depends on the next. Scenes and
objects are ealuated with respect to the overall narrative,
but these aspects sometimes require a multi-step process to
create because they contain many pieces. Multi-step
processes that do not provide immediate exploration and
evaluation are more likely to cause a narrative discrepancy.
It is not that multi-piece elements (such as scenes or
complex objects) are more likely to conflict with narrative
structure, but rather that more work has to be done on these
objects before they can be evaluated with respect to the
global structure of the narrative, which means that mistakes
are not realized until late in the creation process. In terms
of Distributed Exploratory Visualization, these multi-step
creation processes require more time for the ‘create’ part of
the cycle because each element that is created contains
other components that also have to be created, and so on, as
seen in Figure 5.
Intra-layer problems are usually aesthetic discrepancies
between a desired mental image and the image present on
the screen. These problems can typically be resolved
through technical means such as the workaround solutions
generated through creative exploration and subsequently
evaluated through exploration. However, inter-layer
problems are more serious and can represent a conflict with
the narrative, which can be harder to detect and also harder
to resolve. The chances of having a narrative discrepancy
increase when an object cannot be evaluated early in the
cycle to determine whether the idea aligns with the goals of
the overall narrative. The multi-step creative process
described above plays one part in this, but there may also
be another contributing factor.
The mental images at the narrative level have no actual
representation in physical reality besides in the scenes or
the script in most Machinima production. There is no high
level visual breakdown of the scenes. Traditional film
production techniques use storyboards to visually define
the structure of a film, and they can also be used as an
evaluation reference throughout film production. Real time
manipulation has replaced this function and revolutionized
the creative process of digital filmmaking, but in the
process, it may have also contributed to the discrepancy of
narrative abstraction. This issue could be resolved through
a high level virtual sketch of the narrative structure that
includes support for multi-modal scene description, image
manipulation, and creative exploration.
Machinimators describe the prime benefit of the medium as
the immediate feedback and real time support for decision-
making. Therefore, the most important design
Figure 5: How layers of narrative abstraction affect
Distributed Exploratory Visualization
Generate
Pre-Inventive
Structures
Explore
Pre-Inventive
Structures
Create
Evaluate
Narrative Layer
Scene Layer
Object Layer
Generate
Pre-Inventive
Structures
Explore
Pre-Inventive
Structures
Create
Evaluate
Generate
Pre-Inventive
Structures
Explore
Pre-Inventive
Structures
Create
Evaluate
Scene 1
Generate
Pre-Inventive
Structures
Explore
Pre-Inventive
Structures
Create
Evaluate
Scene 2
Generate
Pre-Inventive
Structures
Explore
Pre-Inventive
Structures
Create
Evaluate
Generate
Pre-Inventive
Structures
Explore
Pre-Inventive
Structures
Create
Evaluate
Object 1 Object 2 Object 3
recommendations are those that maximize the relevance of
immediate feedback. This can be accomplished through
lowering the cost of exploration and increasing the fluidity
of evaluating different narrative layers.
CONCLUSIONS
Based on a Grounded Theory analysis of the data collected
during our interviews with expert machinimators, we have
presented a critical reading of some creative practices at
work during Machinima development. Modifying
Geneplore’s approach, we have developed a model of
Distributed Exploratory Visualization. Within this process,
player/ producers craft the final features of the film by
constantly exploring and evaluating their interaction with
the game system.
Whether the individual feature was planned or emergent,
elements added to the scene are evaluated by exploring
how they interact with the rest of the scene. Evaluation is
intermixed with creation – mental imagery with digital
crafting (in this case 3D modeling). As the filmmaker
explores the scene, new perspectives and ways of
interpreting the scene evolve. These new interpretations of
the scene alter the pre-inventive structure, which in turn
changes the parameters by which elements are evaluated.
Distributed creative cognition illustrates the way in which
creative exploration and explorative evaluation are
intimately connected.
Following our research with machinimators, we see a close
connection to Machinima’s unique capabilities that support
this creative cycle and can trace this cycle to an expanded
version of the Geneplore model. This form of playful
creative exploration of the final artifact through digital
technology can be seen as a form of digital crafting. As
Machinima provides this exploration for the moving image,
it serves as a unique example platform for creativity in
digital filmmaking that can and should inform traditional
filmmaking.
ACKNOWLEDGMENTS
This work was supported by the National Science
Foundation under Grant No. IIS-1002748. We would also
like to thank the machinimators for their time and insights.
REFERENCES
1. Cameron, D. and Carroll, J. Encoding liveness:
Performance and real-time rendering in Machinima.
Proceeding of DiGRA 2005 Conference: Changing
Views-Worlds in Play, (2009), 1-9.
2. Finke, R. A., Ward, T.B., Smith, S.M., Creative
Cognition. MIT Press, Cambridge, 1992.
3. Glaser, B. Strauss, A., The Discovery of Grounded
Theory, Aldine Transaction, New Brunswich, 1967.
4. Grinter, R.E. Decomposition: Putting It All Back
Together Again. ACM Conference on Computer
Supported Cooperative Work, (1998), 393-402.
5. Hutchins, E., Cognition in the Wild. MIT Press,
Cambridge, 1996.
6. Kerne, A. and Koh, E. Representing Collections as
Compositions to support distributed creative cognition
and situated creative learning. New Review of
Hypermedia and Multimedia 13, 2 (2007), 135-162.
7. Kirschner, Friedrich “Toward a Machinima Production
Studio" in: The Machinima Reader, eds. Henry Lowood
and Michael Nitsche (MIT Press: Cambridge, MA
2011), 53-71.
8. Kosslyn, S. M., Image and Mind, Harvard University
Press, Cambridge, 1980.
9. Lehane, Scott. July 2001. “Unrealcity: ILM Creates
Artificial Cities for Artificial Intelligence.” Film and
Video 18(7).
10. Lowood, H. Found Technology: Players as Innovators
in the Making of Machinima. Digital Media, (2008).
11. Lowood, Henry. 2007. High-Performance Play: The
Making of Machinima. In Videogames and Art, edited
by G. Mitchell and A. Clarke. Bristol, UK: Intellect
Books.
12. Marino, Paul. 3D Game-Based Filmmaking: The Art of
Machinima. Scottsdale: ParaglyphAnderson, 2004.
13. Nitsche, Michael, 'Film live: And Excursion into
Machinima' in: Developing Interactive Narrative
Content: sagas_sagasnet_reader, ed. by Brunhild
Bushoff (Munich: High Text, 2005), 210-243Conger.,
S., and Loch, K.D. (eds.). Ethics and computer use.
Commun. ACM 38, 12 (entire issue).
14. Nitsche, Michael. 'Games as Structures for Mediated
Performances,' in Logic and Structure of the Computer
Game, ed. by Stephan Guenzel, Michael Liebe, and
Dieter Mersch (Potsdam: University Press, 2010).
15. Nitsche, Michael. “Experiments in the Use of Game
Technology for Pre-Visualization,” in Proceedings of
Futureplay 2008. Eds. Bill Kapralos, Mike Katchabaw,
and Jay Rajnovich. New York: ACM, 2008. 160-66.
16. Salen, Katie. 2002. Telefragging Monster Movies. In
Game On. The History and Culture of Videogames,
edited by L. King. London: Laurence King Publ.
17. Suchman, L. Human Machine Reconfigurations: Plans
and Situated Actions. Cambridge University Press,
Cambridge, 2007.
18. Suwa, M. and Tversky, B. What do architects and
students perceive in their design sketches? A protocol
analysis. Design Studies 18, 4 (1997), 385-403.
19. Tasajärvi, Lassi. 2004. Demoscene. The Art of Real-
Time. Helsinki: evenlake studios.
20. Varela, F.J., Thompson, E., Rosch, E., The Embodied
Mind. MIT Press, Cambridge, 1993.
21. Wallas, G. The art of thought. New York, Harcout,
Brace, and World, 1926.