1. 1 MMI Man Machine Interface Ohad Stein Sefi Epel Technion - IsraelSummer 2002

2. 2 Introduction Technion - IsraelSummer 2002 It is known for years that stress, press, fear, happiness and many other feelings, can be measured by several techniques. Based on this knowledge, we would like to build an interface, that will enable people to communicate without using any muscle, and without making any physical effort. Our interface, will enable communication by being a “mind- reader”. Detection of voltage deflections measured from subjects arms, will enable us to estimate the subject’s will.

3. 3 How can we do it? One way, is to measure the resistance of the skin, to the passage of a very small electric current. There are several ways to “speak” with the machine: Our way was to measure the change in DC potential across neurones of the autonomic nervous system connected to the sensori-motor strip of the cortex. Technion - IsraelSummer 2002 The main disadvantage of this technique is that it is hard to calibrate the system. The measured values changes between different subjects, and between different times with the same subject. In this way – we will detect a signal only after causing it.

4. 4 Part 1: Experimental background Summer 2002 On the following slides, 3 basic experiments are shown. In order to gain some basic knowledge about the MMI abilities we have done some basic experiments, in several different ways. Technion - Israel The first and second experiments have a strong emotional effect. We did them in order to have a basic indication about the ability of building a MMI. The third experiment do not have any emotional aspect. The subject should try to control the voltage deflections measured from his arm. The third experiment is actually the prototype for our project.

5. 5 Exp #1: “Pictures Experiment” – several pictures where shown to the subject, changes on the skin voltage ratio where measured according to the pictures emotional effect. Be aware that the fall of the voltage ratio is a short time after the “strange/unusual” picture is shown. Pictures experiments

6. 6 5 1015202530 Seconds Technion - IsraelSummer 2002

7. 7 5 1015202530 Seconds Technion - IsraelSummer 2002

8. 8 Exp #2: “Voice Experiment” – several words were announced to the subject, different voltage ratio was measured according to the words emotional effect. The words that did not affect the subject were words like “silence”, ”beautiful”, ”sunrise”, and the words that did affect the subject were “war”, “accident” and etc. Note that in this experiment the result can be affected by screaming, changing tone, speaking quietly and etc. Voice experiment Technion - IsraelSummer 2002

9. 9 5 1015202530 Seconds GOOD BAD!!! The person heard 5 different good/bad words. Here are the results: Technion - IsraelSummer 2002

10. 10 5 15202530 Seconds GOOD BAD!!! The person heard 5 different good/bad words. Here are the results: Technion - IsraelSummer 2002

11. 11 Exp #3: “Letters Experiment” – several letters were shown to the subject, one by one. Different voltage ratio was measured according to the picture emotional effect. Exp #3 is a much more complicated experiment. The emotional effect is dependent on the assistant decision and concentration. Letters experiment Technion - IsraelSummer 2002

12. 12 Based on Exp# 3 – we will try to prove the ability of the man to write down a word without using a pen, pencil, keyboard or anything else. Just by measuring the voltage ratio on his arm ! ! ! The red letter shown in the following graphs, is the letter that the subject was waiting to be shown. Technion - IsraelSummer 2002

13. 13 0 61117222733 MQKFTC Second 61117222733 MQKFTC Second Technion - IsraelSummer 2002

14. 14 0 Second 61116222733 BOLSW U 0 Second 61116222733 BOLSW U Technion - IsraelSummer 2002

15. 15 The experiments success As expected, not all the experiments succeeded. There were some reasons that caused it: 1.The training session was not long enough. 2. The session training was not led by Bio-feedback experts. 3. Not always the system was calibrated. 4. The low-frequency signal is not reduced by the system. We had about 70% successful experiments.An experiment is characterized “successful”, if a simple algorithm can detect which simbol the subject chose.

16. 16 0 Second 51116222833 TQKMC F Technion - IsraelSummer 2002 In this slide, you can see the importance of reducing the low-frequency signal.

17. 17 0 Second 61116222733 BOLSW U Technion - IsraelSummer 2002 Here the calibration system is missing, you see the “initializing” effect.

18. 18 0 61117222733 MQKFTC Second In this slide it is hard to detect the effect. Technion - IsraelSummer 2002

19. 19 5 1015202530 Seconds Technion - IsraelSummer 2002 In this slide there isn’t any effect.

20. 20 Part 1 Summary Basic experiments indicate that there is a linkage between the emotions and the voltage deflections measured from our skin. Technion - IsraelSummer 2002 Emotions can be evoked by pictures and sounds. Emotions can be controlled by the subject decision.

21. 21 Part 2 – Bio-Feedback Every hospital uses, nowadays, biofeedback experts in the pain clinics, in order to help the people help themselves. The technique is based on a long training session, led by experts, in order to learn the “patient emotionalism”, and the special biofeedback way that can be useful for him. The meaning of “Bio-Feedback” is that man can “learn” his reflexes, can control them, and can use them as a way of therapy. Technion - IsraelSummer 2002

22. 22 There could be some differences between patients: 4. In the time of training session which is needed. 3. In the psychological ways of the training. 2. In the detection way. 1.In the recording site. Technion - IsraelSummer 2002

23. 23 In order to improve our biofeedback abilities, and after consulting with some experts from the academy and from the medicine area, we have built 3 different train tools for the training. 1.A balloon. The balloon radius is affected by the measured voltage ratio, of the subject skin. The subject should try to increase the balloon radius. Technion - IsraelSummer 2002

24. 24 The balloon radius changes, due to voltage ratio change

25. 25 The balloon radius changes, due to voltage ratio change

26. 26 The balloon radius changes, due to voltage ratio change

27. 27 2,3. A beeper and a colored column. The column height and the beeper tone are the feedback in this tool. Technion - IsraelSummer 2002

28. 28

29. 29

30. 30

31. 31

32. 32 Part 3 – Visual Basic Real-Time implementation In this part of the project we have build an real-time user-interface, for our implementation. Stage #1 – a matrix is shown to the subject. 5 lines of 5 letters each. Each line become bold for 5 seconds. The subject choose one of the 25 letters, therefore he will “like” one line more than the others (the one that includes the chosen letter…) The following slides will show the interface and the effect of the “chosen line” to our mind… Technion - IsraelSummer 2002

33. 33 The user interface Technion - IsraelSummer 2002

34. 34 3 69121518 Second s Line 1Line 4 Line 5 Line 3Line 2 Technion - IsraelSummer 2002

35. 35 3 69121518 Second s Line 1Line 4 Line 5 Line 3Line 2 Technion - IsraelSummer 2002

36. 36 3 6912151818 Secon ds Line 1Line 4 Line 5 Line 3Line 2 Technion - IsraelSummer 2002

37. 37 How does it work? We can see, that the voltage ratio measured from the subject skin, is affected from the bold line. There are some stages on the interface process: 1.Calibration. On the first 15 seconds the system calibrates. Technion - IsraelSummer 2002

38. 38 2. After calibrating, the system starts with bolding each line in a random order for 5 seconds. The screen appearance is shown in the following slide. Technion - IsraelSummer 2002

39. 39 We can see here the bolded line. Line #3 Technion - IsraelSummer 2002

40. 40 3. After choosing a line, by the changes of the voltage ratio on the subject arm, we stop bolding lines and start bolding letters on the chosen line. Now, the subject will be able to choose the wanted letter and print it in the right place. The screen will look as followed. Technion - IsraelSummer 2002

41. 41 Bolding letters one by one, in random order. Technion - IsraelSummer 2002

42. 42 The chosen letters are written here. Technion - IsraelSummer 2002

43. 43 Technion - IsraelSummer 2002 Summary The main target was to be able to write a whole word in the real-time implementation. We have not completely reached this target. We can point at some areas for improvement: 1. A much longer training session. 2. Try to use other Bio-feedback tools. 3. The algorithm should solve some problems that our algorithm did not solve, like reducing the low-frequency signal. In accuracy of 70% we are able to write one chosen letter. The real-time implementation functionality is not completed.