1. Using Lego Mindstorms NXT in the Classroom
Gabriel J. Ferrer
Hendrix College

2. Outline NXT capabilities Software development options
Introductory programming projects
Advanced programming projects

3. Purchasing NXT Kits Two options (same price; $250/kit)
Standard commercial kit
Lego Education kit
Advantages of education kit
Includes rechargeable battery ($50 value)
Plastic box superior to cardboard
Extra touch sensor (2 total)
Includes NXT-G visual language

4. NXT Brick Features 64K RAM, 256K Flash 32-bit ARM7 microcontroller
100 x 64 pixel LCD graphical display
Sound channel with 8-bit resolution
Bluetooth radio
Stores multiple programs
Programs selectable using buttons

5. Sensors and Motors Four sensor ports Three motor ports Sonar Sound
Light
Touch
Three motor ports
Each motor includes rotation counter

6. Touch Sensors Education kit includes two sensors
Much more robust than old RCX touch sensors

7. Light Sensor Reports light intensity as percentage Two modes
Active
Passive
Practical uses
Identify intensity on paper
Identify lit objects in dark room
Detect shadows

8. Sound Sensor Analogous to light sensor Practical uses
Reports intensity
Reputed to identify tones
I haven’t experimented with this
Practical uses
“Clap” to signal robot

9. Ultrasonic (Sonar) Sensor
Reports distances
Range: about 5 cm to 250 cm
In practice:
Longer distances result in more missed “pings”
Mostly reliable
Occasionally gets “stuck”
Moving to a new location helps in receiving a sonar “ping”

10. Motors Configured in terms of percentage of available power
Built-in rotation sensors
360 counts/rotation

11. Software development options
Onboard programs
RobotC
leJOS
NXC/NBC
Remote control
iCommand
NXT_Python

12. RobotC Commercially supported Not entirely free of bugs
Not entirely free of bugs
Poor static type checking
Nice IDE
Custom firmware
Costly
$50 single license
$250/12 classroom computers

13. Example RobotC Program
void forward() {
motor[motorA] = 100;
motor[motorB] = 100; }
void spin() {
motor[motorB] = -100;

14. Example RobotC Program
task main() {
SensorType[S4] = sensorSONAR;
forward();
while(true) {
if (SensorValue[S4] < 25) spin();
else forward(); }

15. leJOS Implementation of JVM for NXT Reasonably functional
Threads
Some data structures
Garbage collection added (January 2008)
Eclipse plug-in just released (March 2008)
Custom firmware
Freely available

16. Example leJOS Program sonar = new UltrasonicSensor(SensorPort.S4);
Motor.A.forward();
Motor.B.forward();
while (true) {
if (sonar.getDistance() < 25) {
Motor.B.backward();
} else { }

17. Event-driven Control in leJOS
The Behavior interface
boolean takeControl()
void action()
void suppress()
Arbitrator class
Constructor gets an array of Behavior objects
takeControl() checked for highest index first
start() method begins event loop

18. Event-driven example class Go implements Behavior {
private Ultrasonic sonar =
new Ultrasonic(SensorPort.S4);
public boolean takeControl() {
return sonar.getDistance() > 25; }

19. Event-driven example public void action() { Motor.A.forward();
Motor.B.forward(); }
public void suppress() {
Motor.A.stop();
Motor.B.stop();

20. Event-driven example class Spin implements Behavior {
private Ultrasonic sonar =
new Ultrasonic(SensorPort.S4);
public boolean takeControl() {
return sonar.getDistance() <= 25; }

21. Event-driven example public void action() { Motor.A.forward();
Motor.B.backward(); }
public void suppress() {
Motor.A.stop();
Motor.B.stop();

22. Event-driven example public class FindFreespace {
public static void main(String[] a) {
Behavior[] b = new Behavior[]
{new Go(), new Stop()};
Arbitrator arb =
new Arbitrator(b);
arb.start(); }

23. NXC/NBC NBC (NXT Byte Codes) NXC (Not eXactly C)
Assembly-like language with libraries
NXC (Not eXactly C)
Built upon NBC
Successor to NQC project for RCX
Compatible with standard firmware

24. iCommand Java program runs on host computer Controls NXT via Bluetooth
Same API as leJOS
Originally developed as an interim project while leJOS NXT was under development
Big problems with latency
Each Bluetooth transmission: 30 ms
Sonar alone requires three transmissions
Decent program: 1-2 Hz

25. NXT_Python Remote control via Python Similar pros/cons with iCommand
Similar pros/cons with iCommand

26. Developing a Remote Control API
Bluetooth library for Java
Opening a Bluetooth connection
Typical address: 00:16:53:02:e5:75
Bluetooth URL
btspp:// e575:1; authenticate=false;encrypt=false

27. Opening the Connection
import javax.microedition.io.*;
import java.io.*;
StreamConnection con = (StreamConnection) Connector.open(“btspp:…”);
InputStream is = con.openInputStream();
OutputStream os = con.openOutputStream();

28. NXT Protocol Key files to read from iCommand: NXTCommand.java
NXTProtocol.java

29. An Interesting Possibility
Programmable cell phones with cameras are available
Camera-equipped cell phone could provide computer vision for the NXT

30. Introductory programming projects
Developed for a zero-prerequisite course
Most students are not CS majors
4 hours per week
2 meeting times
2 hours each
Not much work outside of class
Lab reports
Essays

31. First Project (1) Introduce motors Introduce subroutines Simple tasks
Drive with both motors forward for a fixed time
Drive with one motor to turn
Drive with opposing motors to spin
Introduce subroutines
Low-level motor commands get tiresome
Simple tasks
Program a path (using time delays) to drive through the doorway

32. First Project (2) Introduce the touch sensor Interesting problem
if statements
Must touch the sensor at exactly the right time
while loops
Sensor is constantly monitored
Interesting problem
Students try to put code in the loop body
e.g. set the motor power on each iteration
Causes confusion rather than harm

33. First Project (3) Combine infinite loops with conditionals
Enables programming of alternating behaviors
Front touch sensor hit => go backward
Back touch sensor hit => go forward

34. Second Project (1) Physics of rotational motion
Introduction of the rotation sensors
Built into the motors
Balance wheel power
If left counts < right counts
Increase left wheel power
Race through obstacle course

35. Second Project (2)
if (/* Write a condition to put here */) { nxtDisplayTextLine(2, "Drifting left");
} else if (/* Write a condition to put here */) { nxtDisplayTextLine(2, "Drifting right");
} else {
nxtDisplayTextLine(2, "Not drifting"); }

36. Third Project Pen-drawer Limitations of rotation sensors
First project with an effector
Builds upon lessons from previous projects
Limitations of rotation sensors
Slippage problematic
Most helpful with a limit switch
Shapes (Square, Circle)
Word (“LEGO”)
Arguably excessive

37. Pen-Drawer Robot

38. Pen-Drawer Robot

39. Fourth Project (1) Finding objects Light sensor Sonar sensor
Find a line
Sonar sensor
Find an object
Find freespace

40. Fourth Project (2) Begin with following a line edge
Robot follows a circular track
Always turns right when track lost
Traversal is one-way
Alternative strategy
Robot scans both directions when track lost
Each pair of scans increases in size

41. Fourth Project (3)
Once scanning works, replace light sensor reading with sonar reading
Scan when distance is short
Finds freespace
Scan when distance is long
Follow a moving object

42. Light Sensor/Sonar Robot

43. Other Projects “Theseus” Robotic forklift Perimeter security robot
Store path (from line following) in an array
Backtrack when array fills
Robotic forklift
Finds, retrieves, delivers an object
Perimeter security robot
Implemented using RCX
2 light sensors, 2 touch sensors
Wall-following robot
Build a rotating mount for the sonar

44. Robot Forklift

45. Gearing the motors

46. Advanced programming projects
From a 300-level AI course
Fuzzy logic
Reinforcement learning

47. Fuzzy Logic
Implement a fuzzy expert system for the robot to perform a task
Students given code for using fuzzy logic to balance wheel encoder counts
Students write fuzzy experts that:
Avoid an obstacle while wandering
Maintain a fixed distance from an object

48. Fuzzy Rules for Balancing Rotation Counts
Inference rules:
biasRight => leftSlow
biasLeft => rightSlow
biasNone => leftFast
biasNone => rightFast
Inference is trivial for this case
Fuzzy membership/defuzzification is more interesting

49. Fuzzy Membership Functions
Disparity = leftCount - rightCount
biasLeft is
1.0 up to -100
Decreases linearly down to 0.0 at 0
biasRight is the reverse
biasNone is
0.0 up to -50
1.0 at 0
falls to 0.0 at 50

50. Defuzzification Use representative values: Left wheel:
Slow = 0
Fast = 100
Left wheel:
(leftSlow * repSlow + leftFast * repFast) / (leftSlow + leftFast)
Right wheel is symmetric
Defuzzified values are motor power levels

51. Q-Learning Discrete sets of states and actions Q-values
States form an N-dimensional array
Unfolded into one dimension in practice
Individual actions selected on each time step
Q-values
2D array (indexed by state and action)
Expected rewards for performing actions

52. Q-Learning Main Loop Select action Change motor speeds
Inspect sensor values
Calculate updated state
Calculate reward
Update Q values
Set “old state” to be the updated state

53. Calculating the State (Motors)
For each motor:
100% power
93.75% power
87.5% power
Six motor states

54. Calculating the State (Sensors)
No disparity: STRAIGHT
Left/Right disparity
1-5: LEFT_1, RIGHT_1
6-12: LEFT_2, RIGHT_2
13+: LEFT_3, RIGHT_3
Seven total sensor states
63 states overall

55. Action Set for Balancing Rotation Counts
MAINTAIN
Both motors unchanged
UP_LEFT, UP_RIGHT
Accelerate motor by one motor state
DOWN_LEFT, DOWN_RIGHT
Decelerate motor by one motor state
Five total actions

56. Action Selection Determine whether action is random If random: If not:
Determined with probability epsilon
If random:
Select uniformly from action set
If not:
Visit each array entry for the current state
Select action with maximum Q-value from current state

57. Q-Learning Main Loop Select action Change motor speeds
Inspect sensor values
Calculate updated state
Calculate reward
Update Q values
Set “old state” to be the updated state

58. Calculating Reward No disparity => highest value
Reward decreases with increasing disparity

59. Updating Q-values Q[oldState][action] = Q[oldState][action] +
learningRate *
(reward + discount * maxQ(currentState) - Q[oldState][action])

60. Student Exercises Assess performance of wheel-balancer
Experiment with different constants
Learning rate
Discount
Epsilon
Alternative reward function
Based on change in disparity

61. Learning to Avoid Obstacles
Robot equipped with sonar and touch sensor
Hitting the touch sensor is penalized
Most successful formulation:
Reward increases with speed
Big penalty for touch sensor

62. Other classroom possibilities
Operating systems
Inspect, document, and modify firmware
Programming languages
Develop interpreters/compilers
NBC an excellent target language
Supplementary labs for CS1/CS2

63. Thanks for attending! Slides available on-line:
Currently writing lab textbook
Introductory and advanced exercises