1. IOWA STATE UNIVERSITY Department of Electrical and Computer Engineering C omputer Control of Theater Performance Electronics Clients: Co-Motion Dance Company Iowa State Dance SD May06-18 Team Members:Faculty Advisors: Amanda FarniokDr. Julie Dickerson Sheng LyProf. Gerald Sheble* Alex SillsDr. David Stephenson Tarun Bhatia* *Only on project Fall 2005 April 26, 2006

2. Presentation Outline Project Overview Project Overview Research Activities Research Activities Hardware Configuration Hardware Configuration Software Development Software Development Implementation Implementation Resources and Scheduling Resources and Scheduling Commercialization Commercialization Future Work Future Work Lessons Learned Lessons Learned Closing Remarks Closing Remarks Questions Questions

3. Definitions Isadora - Isadora is a graphic programming environment that provides interactive control over digital media, with special emphasis on the real-time manipulation of digital video. Isadora - Isadora is a graphic programming environment that provides interactive control over digital media, with special emphasis on the real-time manipulation of digital video. SDK - Software development kit SDK - Software development kit Actor – Module for Isadora software developed as an end product. Actor – Module for Isadora software developed as an end product. Scene – A collection of user grouped actors in Isadora. Scene – A collection of user grouped actors in Isadora. TinyOS - TinyOS operating system designed for the Crossbow transceivers TinyOS - TinyOS operating system designed for the Crossbow transceivers nesC - an extension to the C programming language designed to embody the structuring concepts and execution model of TinyOS. nesC - an extension to the C programming language designed to embody the structuring concepts and execution model of TinyOS. OSC – Open Sound Control OSC – Open Sound Control

4. Acknowledgement Janice Baker – Iowa State Dance Janice Baker – Iowa State Dance Valerie Williams – Co-Motion Dance Company Valerie Williams – Co-Motion Dance Company Dr. Julie Dickerson – faculty advisor Dr. Julie Dickerson – faculty advisor Dr. Gerald Sheble – faculty advisor Dr. Gerald Sheble – faculty advisor Dr. David Stephenson – emeritus faculty Dr. David Stephenson – emeritus faculty Jason Boyd – ECpE departmental technician Jason Boyd – ECpE departmental technician Daji Qiao – associate professor Daji Qiao – associate professor Wen-Chieh – HCI 572 student Wen-Chieh – HCI 572 student

5. Problem Statement Control dancer environment without the help of stagehands or technicians Control dancer environment without the help of stagehands or technicians Need dancer-operated sensor pack which sends input to be received by a computer program called Isadora Need dancer-operated sensor pack which sends input to be received by a computer program called Isadora Isadora can be programmed to control lighting, video projection, audio, and other characteristics of the dancer’s stage environment Isadora can be programmed to control lighting, video projection, audio, and other characteristics of the dancer’s stage environment Similar commercial technologies are not able to differentiate between the inputs Similar commercial technologies are not able to differentiate between the inputs

6. Solution-approach Statement Crossbow Technology Components: Sensor Board, Transceivers, Serial Interface Board Crossbow Technology Components: Sensor Board, Transceivers, Serial Interface Board Flex sensors (variable resistors) used to monitor movement Flex sensors (variable resistors) used to monitor movement Used nesC code in TinyOS (Crossbow program) to program components and forward data Used nesC code in TinyOS (Crossbow program) to program components and forward data Used C code to write serial-to-OSC relay Used C code to write serial-to-OSC relay

7. Operating Environment Theatrical dance settings: Betty Toman Dance Studio in Forker Hall, Fischer Theater, Stephens Auditorium, Ames City Auditorium Theatrical dance settings: Betty Toman Dance Studio in Forker Hall, Fischer Theater, Stephens Auditorium, Ames City Auditorium Possibility of ambient electromagnetic noise caused by the multitude of electrical equipment in the immediate vicinity Possibility of ambient electromagnetic noise caused by the multitude of electrical equipment in the immediate vicinity Substantial abuse caused by the motion of the dancers, their impact with the floor or wall, and the sweat and heat that their bodies produce during performance Substantial abuse caused by the motion of the dancers, their impact with the floor or wall, and the sweat and heat that their bodies produce during performance

8. Intended Users and Uses Users Members of Iowa State Dance Members of Iowa State Dance Members of Co-Motion Dance Company Members of Co-Motion Dance Company Directors and colleagues of the companies Directors and colleagues of the companiesUses Practices, performances, and productions of the users Practices, performances, and productions of the users Usable in any situation analogous to the conditions of these groups’ activities Usable in any situation analogous to the conditions of these groups’ activities Transmitter is within the requested range of 60 feet from the receiver Transmitter is within the requested range of 60 feet from the receiver Impacts inflicted on the transmitter and sensor by the dancers are minor Impacts inflicted on the transmitter and sensor by the dancers are minor

9. Assumptions and Limitations Assumptions Flex sensor data transmission only Flex sensor data transmission only No clear line of sight from the transmitter to the receiver No clear line of sight from the transmitter to the receiver Receiver and computer, as well as the visual aids, will run on power supplied from an external source Receiver and computer, as well as the visual aids, will run on power supplied from an external source Sufficient power to operate for at least the duration of an ISU/Co-Motion dance production Sufficient power to operate for at least the duration of an ISU/Co-Motion dance productionLimitations End-product will be only one transmitter/receiver pair but will allow for expansion for up to four transmitters End-product will be only one transmitter/receiver pair but will allow for expansion for up to four transmitters Small and compactly packaged to allow for full range of motion by the user Small and compactly packaged to allow for full range of motion by the user Distance from the transmitter to the receiver will be at most 60 feet Distance from the transmitter to the receiver will be at most 60 feet

10. End-Product and Deliverables Wireless transmitter with four sensors Wireless transmitter with four sensors Receiver connected through Serial/USB adapter to computer Receiver connected through Serial/USB adapter to computer Sensor watcher written to communicate between TinyOS and Isadora Sensor watcher written to communicate between TinyOS and Isadora User Manual for client to recreate and use full functionality of system User Manual for client to recreate and use full functionality of system

11. Data Flow - Functionality Hardware Flow Diagram Flex Sensor, Analog Voltage Signal MDA300 Analog Input Board MICAz Transmitter Mote MICAz Receiver Mote MIB510 Serial Interface 2.4 GHz Wireless Transmission HARDWARE CONNECTED TO COMPUTER HARDWARE WORN BY DANCER

12. Previous Research Project VrJuggler was previously used on a research project to manipulate input from sensors and display 3D video on projectors. VrJuggler was previously used on a research project to manipulate input from sensors and display 3D video on projectors. Certain aspects of previous research were taken into consideration Certain aspects of previous research were taken into consideration –Flex sensors –Crossbow Technology Inc.

13. Present Accomplishments Transmitter/receiver tested for range and use with sensors, being used in theater Transmitter/receiver tested for range and use with sensors, being used in theater Hardware programmed through nesC for correspondence between transmitter and computer Hardware programmed through nesC for correspondence between transmitter and computer Packaging purchased, installed Packaging purchased, installed Able to get sensor readings into Isadora Able to get sensor readings into Isadora

14. Future Tasks Complete User Manual Complete User Manual Purchase Crossbow system for client’s possession Purchase Crossbow system for client’s possession

15. Hardware Design - Research Analog audio wireless transmission method Analog audio wireless transmission method –Problems: Modulation/compression issues, capacity/channel issues, possible interference problems

16. Hardware Design - Research Industrial SCADA-type wireless transmission method Industrial SCADA-type wireless transmission method –Problems: Bulky packaging, insufficient channels/capacity, possible software (Isadora) compatibility issues

17. Hardware Design - XBow Crossbow “Mote” Transmission Method Crossbow “Mote” Transmission Method –Reasons for choice: Compact sizing, 2.4GHz band avoids interference and allows for multiple transmitters, TinyOS accompanying software suited for project goals –Possible problems: TinyOS/Isadora communication, line of sight issues

18. Hardware Design - XBow

19. Analog Output Circuit Crossbow motes accept analog inputs Crossbow motes accept analog inputs Flex sensor variable resistors used in a simple analog circuit to give 1.31 - 1.95 Volt output to mote board Flex sensor variable resistors used in a simple analog circuit to give 1.31 - 1.95 Volt output to mote board

20. Circuit Diagram - Four Sensor Output

21. MDA300 Sensor Board Accepts analog inputs via screw terminals Accepts analog inputs via screw terminals Provides excitation voltages Provides excitation voltages Converts analog to digital signal for wireless transmission Converts analog to digital signal for wireless transmission

22. MICAz Transceivers Connects to sensor board and serial interface with 51-pin connector Connects to sensor board and serial interface with 51-pin connector Transmits on 2.4 GHz Zigbee compliant band (good range, multiple channels, minimal interference problems) Transmits on 2.4 GHz Zigbee compliant band (good range, multiple channels, minimal interference problems) Powers sensor board, simple AA battery power supply Powers sensor board, simple AA battery power supply

23. MIB510 Serial Interface Receives signal from transceiver and transmits to computer through TinyOS software Receives signal from transceiver and transmits to computer through TinyOS software Powered by external AC adapter Powered by external AC adapter Through TinyOS, programs/configures transceivers and sensor board Through TinyOS, programs/configures transceivers and sensor board

24. Hardware Software Integration TinyOS is a platform bridging the hardware serial input using nesC TinyOS is a platform bridging the hardware serial input using nesC –Provides Libraries for parsing sensor data –Environment for the Programming the Wireless Motes. Program for receiving wireless data and putting on the serial port provide by Xbow Program for receiving wireless data and putting on the serial port provide by Xbow

25. Wireless Transmitter Program Developed in nesC Developed in nesC

26. Isadora Module Development Provides overall design needed for working Isadora Plugin/Actor. Provides overall design needed for working Isadora Plugin/Actor. Provides data structures and functions that are needed to implement. Provides data structures and functions that are needed to implement.

27. General Flow of the Functions

28. Open Sound Control Solution –C backend solution

29. Isadora Actor Mockup Our Actor Mockup Actor Link Connection

30. Putting the Pieces Together Implementation Integration between hardware and software using the research and documentation Integration between hardware and software using the research and documentation Implementing in theater with components on loan from the Department of Electrical Engineering Implementing in theater with components on loan from the Department of Electrical Engineering

31. Testing Procedures and Results Measuring flex sensor voltage Measuring flex sensor voltage –1.31 to 1.95 volts Distance between transmitter/receiver and line of sight issues Distance between transmitter/receiver and line of sight issues –Work up to 60 feet and without line of sight –Reliability issues when in enclosed corners Packaging integrity Packaging integrity Currently beta testing by client for use in a performance Currently beta testing by client for use in a performance

32. Resources

33. Schedule

34. Project Evaluation Research of Hardware and Software Research of Hardware and Software –100% Complete –Using findings to implement system Testing Testing –95% Complete –Hardware works with software Implementation Implementation –98% Complete –Software designed and loaded onto hardware, currently in use Documentation Documentation –95% Complete –All formal reports complete –User Manual under revision

35. Commercialization Theatrical companies readily share technological findings Theatrical companies readily share technological findings Share findings with those that shared information with us Share findings with those that shared information with us Consult Crossbow Technology before any package sales could be done Consult Crossbow Technology before any package sales could be done Selling Price: $1,000 Selling Price: $1,000 Market: High demand in performing arts in near future Market: High demand in performing arts in near future

36. Future Recommendations Switch batteries from AA to 3 V coin Switch batteries from AA to 3 V coin Custom design casing for compact yet safe design Custom design casing for compact yet safe design Connectors for between sensors and wires and stability of sensor joint Connectors for between sensors and wires and stability of sensor joint Expand to six sensors per transmitter Expand to six sensors per transmitter Expand to four transmitters Expand to four transmitters Use different sensor input Use different sensor input

37. Lessons Learned Successes Group teamwork and communication Group teamwork and communication Group supervision by faculty Group supervision by faculty Attainable trial hardware Attainable trial hardware

38. Lessons Learned Setbacks Loss of team member Loss of team member Original circuit design not applicable Original circuit design not applicable Developing software before hardware was decided Developing software before hardware was decided

39. Lessons Learned Experience Gained nesC Programming nesC Programming RF communication RF communication Multitasking Multitasking Client relations Client relations Communicating different expertise Communicating different expertise Delegating responsibilities Delegating responsibilities

40. Lessons Learned If we did it all again… Hardware team more involved with software development Hardware team more involved with software development More research on software cross- platform development More research on software cross- platform development

41. Risk and Risk Management Procurement of Materials Procurement of Materials –Hardware borrowed from department for testing and implementation before purchase Loss of team member Loss of team member –Encountered and overcome by members gathering completed progress and taking on more tasks –Involved other resources – HCI 572 Design and Evaluation of HCI Data Loss Data Loss –All documentation easily transferable between group members Physical Damage Physical Damage –Kept components in tackle box for safe keeping

42. Closing Remarks Successfully provided means for dancers to control own environment Successfully provided means for dancers to control own environment Human computer interaction Human computer interaction Leading other companies in theatrical technology Leading other companies in theatrical technology Expansion is possible Expansion is possible

43. Questions