Abstract

How can computer intelligence be used in the theater? Imagine that a computer is given the ability to control electronically all of the media of the stage, and is able to sense and understand in an abstract way what is happening in that space. Furthermore, suppose that the computer is given the ability to reason about what was happening and could construct abstract responses through media. What would be possible for the computer to do? In an analogy to the human body, the theatrical space is the computer's body, the electronic media the limbs, cameras and microphones used as sensors are the eyes and ears, a speech generation program the mouth, and the CPU(s) and internal programming are the brains that are used to interact with the physical world. The space that holds the performance becomes an environment generated from behaviors of the computer, responding to and shaped by performers, designers, and technicians. These computer behaviors are created through speech and movement, light and sound, music and visuals. Until the space is shaped, it remains inert and waiting to be molded, not a virtual space, but an actual physical manifestation of media sculpted by the artist through the computer.

This kind of intelligent environment will expand the expressive potential of traditional theater in many ways. Additional power will be given to actors, to directors, to dramaturgs, and to set designers to manipulate lighting, sound, and visuals. How this will effect the viewers’ and performers’ perceptions is an open question. The computer will require some kind of structure in order to be an 'intelligent participant'. While intelligent performance technology often enables the creation of new ways of working and new effects, it is not always useful as an alternative to more robust techniques that are manually created and controlled. But, it will make more effects possible and easier to create as it already has in theatrical lighting.

This paper considers some of the pros and cons of the involvement of computer intelligence in performance settings. It explores what dreams may come when the computer is allowed to grow from its current state of infancy in its involvement in theater. But, its involvement is not about the death of traditional theater (or dance, or art), but rather about its growth into new realms of expressiveness. The theater event cannot be reduced to the status of a physical container; it is not the lighting, or the set, or even the actors. Theater is the communication of thought to an audience within a created context and intention.

1.0 Introduction

Three kinds of abilities are required for computer intelligence in the interactive arts: Perception, Reasoning, and Dexterity. Perception is, and can be measured, as a level of understanding about what is happening in the environment. The computer accomplishes this through abstractly created and stored representations of sensed information. Sensing occurs through analysis of video images, spoken text, and user interface interactions. Reasoning is the computer making decisions about what responses are necessary or appropriate for given contextual and sensed information. Dexterity is the computer’s ability to respond to, and carry out, abstract decisions using electronic media and to interact with artists. With these abilities, the computer should be able to become a new kind of participant (as a performer, technician, or assistant) within the performance of a theatrical work or an interactive art piece.

However, the computer can become an 'intelligent' participant only with the 'real world knowledge' of production techniques and the structure of a performance or installation. This structure is contained within the minds of the designers and participants and the computer must be able to understand these views in order to incorporate contextual responses that allow it to participate and interact. By giving the computer some understanding of the 'real world' and of commonly understood concepts and languages of theater, the designers and participants (performers or audience) will be able to more easily control the computer, and this will give them more expressive range of controls of the media. This already occurs in a limited way in many theater productions through the use of standard light boards, sound mixers, and other computer controlled equipment. What current theatrical technique lacks is an adequate way of interacting with, and coordinating the equipment.

A very active field of research being explored at the Institute for Studies in the Arts (ISA) at Arizona State University is in the area of the use of computer intelligence in the arts. This research is an extension of the technology of the Intelligent Stage developed at the ISA. That work has focused the attention of many artists and researchers on the tools that can be used in media rich live performance.

The Intelligent Stage is a mediated performance space that responds to the actions of performers as they move. The system’s primary sensing occurs through a program called EYES that analyzes video activity to understand what is happening on stage [Figure 1]. (Speech recognition is also being researched as a another means of sensing). Sensing by the computer allows performers to control electronic theatrical elements such as sound, lighting, video, and slides through digitized video, photoelectric switches, contact switches, and many types of activities. Media responses occur through several controller computers that manipulate the theatrical electronic media.

The research being conducting now at the ISA and the Institute for New Media Performance Research at the University of Surrey (INMPR) involves migrating and expanding the technology of the Intelligent Stage into a system that is not dependent on a particular location. Although it will be dependent on certain types of equipment, the idea is to create a performance-aware intelligence that can 'live' in any performance space and 'learns' to use the equipment that is resident. Instead of requiring artists to work within a particular place, a computer system travels to where the artist wants to work or perform and adapts to that environment.

Computer intelligence is defined as the ability of a computer to understand, reason, and apply knowledge. This involves sensing the environment, assimilating what is happening within that environment, and responding. The computer has a knowledge representation scheme that allows the computer to "understand" problems the artist is trying to solve in the creation of a theatre production. Artists, in other words, can use the technology without having to immerse themselves in the details of its operation. Instead of relating in unfamiliar computer terms, the artist will be able to relate ideas to the computer in more 'real world' terms, often giving instructions in written or spoken English.

The current Intelligent Stage is not really 'intelligent' in that it lacks the above-mentioned qualities. It senses the environment, but only reacts according to predefined procedures created for each production. This algorithmic knowledge is lost each time a new show is created. Because of this, the technology is cumbersome and difficult to use.

Giving a computer the ability to sense, and the dexterity to respond to perceived information allows it to make decisions that can influence, and even change the direction of performances. Computers already have this capability in a limited sense, yet the question of how such capabilities can be brought to bear on and to operate within structures of artistic practice, and indeed what new structures will become possible, remain to be studied and addressed through further research. The computer can, for instance, behave like a human performer, or can act like an observer monitoring an installation. Taken to the extreme, the computer could have the ability to make design choices based on past experiences.

In general, technologies have always been difficult to use in their infancy. But, if the technology is useful, then through new innovations, simplifications, and standardization of techniques, new technologies such as the Intelligent Stage can be positively integrated into performance traditions.

Some of the most frequently asked questions of this kind of research include: "Why do it?", and "Do these technologies belong in theater and dance?", and "How should these technologies be used?". To these questions there are several answers that might at this point be offered.

Using new techniques to accomplish technical goals always entails a search for easier ways of doing things. The computer has the ability to manipulate the media in response to performers at a rate, and in a way, that cannot be fully pre-programmed. While it is rarely the case for coordinated starts and stops of effects not to be able to be controlled by an operator, often the control of parameters that modify the effect are outside the range of an operator's abilities. (It should be noted that computer intelligence and interactivity are not a means of eliminating a technical person. If anything, computer technologies as integrated into live performances enhance the role of the technical person).

The use of a computer in integrated media performance may pose a set of fruitful design/direction questions, rather than merely a question pertaining to the ‘legitimacy’ of the computer’s function. To offer a parallel example, we might say that if a new performer is added to a theatre production, this addition is not likely to be disputed on the grounds that "performers are not good for theater". But rather, (if at all) on the grounds that the addition of a new performer may not serve the needs of that particular play, production or performance. In the same way, when the use of interactivity is considered as an addition to or augmentation of live performance, this addition of new technology is best considered on a case by case basis, rather than in the larger terms of what may be ‘best’ for performance in general.

Of course many kinds of effects can be accomplished with traditional techniques, but the level and complexity for creating and controlling effects using several controller devices (sound, lights, and sensing) is sometimes too difficult to have any utility. A performance-aware computer however, has the expert knowledge to translate higher level descriptions of effects into programming.

Another important reason for the advancement and integration of computer intelligence into and among artistic forms is to build a bridge between the different art forms, music, visual arts, dance, and theater. The nature of interactive multi-electronic media fosters collaboration between people from different backgrounds and traditions because of the demand for content for many types of media and devices. Through the use of computer intelligence these collaborative processes will be enhanced.

Another reason for the use of computer intelligence in the arts is related to the idea of emergent phenomenon. Emergent phenomenon influence the processes through which a work is created by providing options that might not otherwise enter the designer’s mind. By the creation of simple mechanisms for interaction, a complicated, and hopefully also interesting, phenomenon occurs through the interaction. The computer becomes a catalyst for the creation of a performance or installation work based upon actions set in motion by the artists.

Finally, as when performers must interact with each other, interaction with computer intelligence creates the need for the performer to be aware on a different level than the personally physical and mental. That awareness extends to what is happening within the media, both visual and auditory, because the action of the media is the performer’s link to the awareness of the computer’s ‘perception’. This awareness becomes recognizable by the audience members in the performance event.

When answering the question why, it is important to keep focused on what's really important: the artistic result. Technology is very seductive and often becomes the end in itself. It has often been the case that the technology 'was there' and the artist found new ways to use it. That is not to say that experimenting with technology is not important as a means of experimentation and testing the limits of computer intelligence in performance. But in the end, the art is the most important aspect of the creation of an artistic.

How computer intelligence can be used is a more difficult question to answer because there are so many ways in which it can be used. How it is eventually used depends upon what particular methodologies are successful: a system of trial and error, performance, integration and testing of reception.

4.0 Possibilities

Computer intelligence’s place in theater is to augment, assist, and participate in performance. Participation as an actor in the "human sense" is a real possibility. But a computer can also make producing a show smoother, and it is capable of making it easier to create difficult computer controlled effects.

It is tempting to think that there is an ‘non-technological’ theatre tradition against which the kinds of computer-assisted performance discussed here might be measured. In most contemporary theater, however, computers control the media. Lighting is the most computerized of the theatrical media, followed closely by sound, sound sources, and electronic music. With video and other effects the computer is less involved, but its influence is growing. Sound and video will eventually migrate toward standard processes and technologies in the same way lighting technology has already done. The more the computer is involved in coordinating activities in media, the more difficult it becomes for the human participant to orchestrate what the computer does.

MIDI show control and SMPTE (these are technical communication protocols used by computers to control and synchronize media) and other protocols are effective techniques for manipulating and coordinating media in this environment, but they need to be placed in a context where they are not the only means of communicating coordination of media to the computer. The computer needs to operate at a higher communication level so that the humans are not required to think in unfamiliar patterns. With performer and audience interactive effects, this is even more important because the complexity of what the computer does can greatly increase.

The key to making integrated media performances and installations easier to produce is in creating a tradition of processes that work. Technology has a way of producing too many options, and too many ways of doing things, so that the artist becomes embroiled in the details of mechanics rather than in the overall form.

What are the elements of Computer Intelligence that are useful for theatrical situations? Intelligence requires the ability to make responses based on current sensory input. Sensory experience for the computer includes typed words, spoken sentences, mouse input, video and other types of motion sensed information and contact switches. Responses are manipulations of electronic media: lights, sound, visuals, robotics, etc.

What is the glue that allows the computer to transform what it "perceives" into an "appropriate" action? This glue is a whole system of inter-related elements that interact as a kind of primitive brain. The most important structure of this system is the representation of knowledge (information) and the mechanisms for manipulating that knowledge.

Information may be represented in many different ways, depending upon how it will be used. Several forms exist: accumulated knowledge, common knowledge, and contextual information. Sensed information is incorporated into these systems and decisions are made based on the actions of inference engines, advice taker/inquirers, and algorithmic behaviors.

Taking each knowledge form one at a time, what do they enable designers and artists to do? Accumulated knowledge is the recorded information about what has been done while the computer is running. In current systems, cue lists offer the closest approximation to accumulated knowledge. In a cue list, the computer knows what light levels, sound levels, etc need to be set for a given cue in a given show. In a more general sense, accumulated knowledge also includes other pieces of information about a show like the light plot, or the dimmer assignments for a particular building, or what DAT tape needs to be inserted in the machine at what time.

Common knowledge is defined as the computer’s a-priori knowledge of the theatrical environment, both as a concept and as an example of the particular situation it is placed within. This knowledge allows the computer to understand the language used in the theater: the computer can then be spoken to or written to, in that language domain. That is, it knows what words are spoken, and how they are spoken and can make judgements about the possible meanings that words might imply if actualized in performance.

Contextual information is what allows the computer to interpret what is happening at the moment. For instance if a light cue is being created, and someone says to the computer "bring 2 to full", the computer knows that this means bring channel 2 on the light board to full, and not channel 2 on the sound mixer. Contextual information also allows the computer to know how to adapt to new or already experienced situations.

Knowledge together with inference provides a base for translating descriptions of what must be accomplished into technical instructions for the devices controlling the media. The inference engine accomplishes the action of deciding something. This is a program that takes a set of rules and examines the current situation to see if and what action needs to take place. For instance, instructions by a designer to bring up the lights stage right could be interpreted as "bring channels three through ten and twenty six up by ten percent", or further as "send 0x5f to MIDI controller numbers 15,41,12,6,1,76,24,10, and 2 on channel 15". A sophisticated inference engine will allow the computer to be able to reference past types of interactions and adapt them to new spaces or situations.

An advice taking program obtains accumulated knowledge through queries for information or by simply taking statements of facts from the users. This program is used to tell the computer the configurations of people and objects in the theater and to clarify issues or instructions the computer does not understand.

Algorithmic behavior is similar to an inference engine and represents information given to the computer so that it can make conditional decisions about what to do in specific situations. These are rule-like statements such as "if cue 6 is over wait 10 seconds and then run cue 10", or "if there is movement stage left, play a video clip". The difference between inference and algorithm is that in an algorithm there is a specific cause and effect. With inference there are many actions that can be taken for a given situation, and they are not specific.

In a performance called Time in the Eye of the Needle several examples present what is possible. In the piece, which consists of eleven interactive sections, and a non-interactive finale, five sections stand out as particularly helpful in illustration of this discussion.

In one section, a woman dances a solo where two interactive musical effects are possible [Figure 2]. One interactive element consists of two locations on stage where she is able to access some background sounds of birds. Another consists of a sensor that looks at a floor projection of water; when the dancer enters the floor projection the computer responds with bamboo chimes.

In this example a video camera looks from overhead at the stage space and when motion is detected in certain locations in the video (corresponding to locations on stage), the computer plays a sample of sound.

A soloist enters the space and is illuminated by lights that "walks" toward him. He takes hold of the lights and turns them into a red sphere that he can then move around the space with gestures. On all sides of him are sensors that control the position of where the light moves. An overall sensor detects how fast he moves and adjusts the speed of the light proportionally. In this section the dancer is also responsible for the sound score by layering sounds generated by the sensors [Figure 3].

In another section four women dance a quartet, but stay within the same vicinity of each other [Figure 4]. Three projections of tombstones, two upstage and one stage left, "watch" the dancers as they move, turning stage left and right

or upstage and downstage depending upon the location of the four dancers. This effect was created using an overhead camera. The contrasting image of the dancers against the floor was used to calculate an average location within an image. This location was translated into a three dimensional stage location through knowledge of the camera geometry. Video on laser disc was manipulated by translating the x and z locations of the dancers into directions of movement within video sequences.

In another section a lone figure approaches cages, and as she approaches each cage, spoken text repeats and increases in volume as she moves closer to a cage. Video sensors that "look at" the floor in the area close to each cage accomplishing this effect. The location of the dancer within the sensor determines the volume of the playback of the spoken text.

Finally in a fifth section a solo dancer moves about the stage either travelling widely or huddling on the floor. Four robotic lights moved in a circle around her while she performed. When she is on the floor and low, the robotic lights spin slower and closer to her. When she gets up, they move away from her to the outskirts of the stage. This effect was accomplished by placing a camera "looking from" the audience perspective. The height of the contrasting figure in the image determines the speed of the lights and the diameter of the pathway they follow.

These effects are simple and completely within the control of the performers and provide a means for improvisation within the structure of the work. The control mechanisms were not created through traditional means with light boards, and pre-recorded cues. Instead, many of the actions in the show were created through the use of algorithmic knowledge programmed into the computer. The computer, instead of following a cue list, has a list of conditional statements based on sensed information that become active at different points, depending upon the desired effect. This mechanism provides a more powerful way of controlling and coordinating integrated and interactive media that is different from the traditional event-based model.

In the future, these kinds of effects will be accomplished through more intelligent computers in what will become traditionally defined processes. Computer Intelligence, video based sensing, and speech understanding will become mechanisms used by performers and designers, and will find their place as tools of the trade.

Gorden Craig, in his book "On The Art of Theater" wrote:

Peter Brook says in "The Empty Space":

Robb Lovell is a Resident artist/technologist at the Institute for Studies in the Arts (ISA) at Arizona State University. He is co-creator of the Intelligent Stage, a theatrical space that registers sensory input through video and audio, and responds through lights, sound, video, animation, and robotics. Current research includes motion and gesture recognition, theatrical knowledge bases, speech driven control, digital video in performance, and autonomous artistic agents. Lovell is currently creating tools for artists and technicians based on the technology of the Intelligent Stage: tools that allow artists to create interactive mediated works. He is working on a practical Ph.D. in Interactive Theatre Design through the Institute for New Media Performance Research.