1. Learning and Control of Biped Locomotion
Dr Changjiu Zhou
School of Electrical & Electronic Engineering
Singapore Polytechnic
Development of Humanoid Soccer Robots

2. Development of Humanoid Soccer Robots
Outline
Introduction
Biped Walking Cycles
How to Control Biped Locomotion
How to Plan/Learn Biped Gaits
Biped learning by reinforcement
Some Research Topics
Development of Humanoid Soccer Robots

3. Biped Gait (Frontal Plane)
Single Support
Double Support
Time
Biped Gait (Frontal View)
Development of Humanoid Soccer Robots

4. Biped Gait (Sagittal Plane)
Development of Humanoid Soccer Robots

5. Finite State Machine for Biped Walking Control
Right
Support
Left
Left-to-Right
Transition
Right-to-Left
Swing time completed
Left foot touches down
Right foot touches down
Development of Humanoid Soccer Robots

6. Development of Humanoid Soccer Robots
Static Walking
In static walking, the biped has to move very slowly so that the dynamics can be ignored.
The biped’s projected center of gravity (PCOG) must be within the supporting area.
Double Support
Single Support
Development of Humanoid Soccer Robots

7. Development of Humanoid Soccer Robots
Dynamic Walking
In dynamic walking, the motion is fast and hence the dynamics cannot be negligible.
In dynamic walking, we should look at the zero moment point (ZMP) rather than PCOG.
The stability margin of dynamic walking is much harder to quantify.
Development of Humanoid Soccer Robots

8. Why is Biped Robotics Hard?
Unpowered DOF between the foot and ground
This constraint limits the trajectory tracking approaches used commonly in manipulators research.
Development of Humanoid Soccer Robots

9. Biped Control: Model-based
Feet position and ZMP (PCOG)
Inverse kinematics model
Desired joint angles
Biped Robot
Development of Humanoid Soccer Robots

10. Biped Control: Model-based
Except for certain massless leg models, most biped models are nonlinear and do not have analytical solutions.
Massless leg model is the simplest model. The body of the robot is usually assumed to be point mass and can be viewed to be an inverted pendulum.
When the leg inertia and other dynamics like that of the actuator, joint friction, etc. are included, the overall dynamic equations can be very nonlinear and complex.
Development of Humanoid Soccer Robots

11. Example: Massless leg model
The simplest biped model
Some assumptions, e.g.,
From D’Alembert’s principle
Development of Humanoid Soccer Robots

12. Biped Control: Biologically Inspired
Since none of the humanoid robots match biological humanoids in terms of mobility, adaptability, and stability, many researchers try to examine biological bipeds so as to extract certain algorithms that are applicable to the robots.
Reverse Engineering
Development of Humanoid Soccer Robots

13. Biped Control: Biologically Inspired
Two Main Research Areas
Central Pattern Generators (CPG)
Passive Walking
Development of Humanoid Soccer Robots

14. ZMP-based Gait Planning
Plan the hip and ankle trajectories according to walking constraints and ground constraints.
Derive all joint trajectories by inverse kinematics.
Development of Humanoid Soccer Robots

15. Example: Gait Planning for Walking on Slope
- Plan gait using 3rd order Spine which guarantees the continuity of both 1st derivative and 2nd derivative.
Development of Humanoid Soccer Robots

16. Example: Planning Results
Consecutive walking gait along slope
Joint angles
Development of Humanoid Soccer Robots

17. IP-based Gait Planning
The dynamic equation of the IP model L v 
2wf
If the angle is small, it can be simplify as a linear homogeneous 2nd order differential equation
Development of Humanoid Soccer Robots

18. 3D Linear Pendulum Model
Development of Humanoid Soccer Robots

19. Example: IP-based Gait Planning
Development of Humanoid Soccer Robots

20. Development of Humanoid Soccer Robots
Biped Kicking
Kicking constraints:
Kicking range
Friction …
Development of Humanoid Soccer Robots

21. Development of Humanoid Soccer Robots
Kicking Pattern
Development of Humanoid Soccer Robots

22. Biped Learning by Reinforcement (1)
A humanoid robot aims to select a good value for the swing leg parameters for each consecutive step so that it achieves stable walking.
A reward function that correctly defines this objective is critical for the reinforcement learning.
Supporting foot
Unstable
r = -1 (punishment)
Stable
r = 0 (reward)
Development of Humanoid Soccer Robots

23. Biped Learning by Reinforcement (2)
The control objective of the gait synthesizing for biped dynamic balance can be described as
To evaluate biped dynamic balance in the frontal plane, a penalty signal should be given if the biped robot falls down in the frontal plane

24. Biped Learning by Reinforcement (3)
Good
Very Bad
Supporting foot
Excellent
Bad OK
Reinforcement Learning with Fuzzy Evaluative Feedback
Development of Humanoid Soccer Robots

25. Development of Humanoid Soccer Robots
The RL Agent
AEN - the action-state evaluation network
ASN - the action selection network
SAM - the stochastic action modifier
Both the AEN and ASN are initialized randomly.
Learning starts from scratch.
It needs a large number of trials for learning.
Development of Humanoid Soccer Robots

26. Development of Humanoid Soccer Robots
The FRL Agent
Neural fuzzy networks are used to replace the neuron-like adaptive elements.
The expert knowledge can be directly built into the FRL agent as a starting configuration.
The ASN and/or AEN could house available expert knowledge to speed up its learning.
Development of Humanoid Soccer Robots

27. The FRL Agent with Fuzzy Evaluative Feedback
The numerical evaluative feedback is not the biological plausible.
The fuzzy evaluative feedback is much closer to the learning environment in the real world.
The fuzzy evaluative feedback is based on a form of continuous evaluation.
Development of Humanoid Soccer Robots

28. Comparison of FRL Agents
Types
Action Network
(ASN)
Critic Network
(AEN)
Evaluative Feedback
RL agent
neural
numerical
FRL agent
(Type A)
neuro-fuzzy
(Type B)
(Type C)
Fuzzy
Development of Humanoid Soccer Robots

29. Information Available for Biped Gait Synthesizing
The Description of the Information
Case A
No expert knowledge is available. Only numerical reinforcement signal is used to train the gait synthesizer.
Case B
Only the intuitive biped balancing knowledge is used as the initial configuration of the gait synthesizer.
Case C
Both the intuitive biped balancing knowledge and walking evaluation knowledge are utilized.
Case D
Besides all the information used in case C, the fuzzy evaluative feedback, rather than numerical evaluative feedback, is included.
Development of Humanoid Soccer Robots

30. The Gait Synthesizer Using Two Independent FRL Agents
Development of Humanoid Soccer Robots

31. Before and After Learning
Ankle joint
Knee joint
Development of Humanoid Soccer Robots

32. Results (1) The ZMP trajectory after FRL (type C)
Development of Humanoid Soccer Robots

33. Development of Humanoid Soccer Robots
Results (2)
Walk (Backward)
Development of Humanoid Soccer Robots

34. Development of Humanoid Soccer Robots
Some Research Topics
Online gait generating
Online footprint planning
Constraints
ZMP constraint for stable walking
Friction constraint for stable walking …
Current Challenges
Knee bending
Body shifting
Development of Humanoid Soccer Robots

35. Development of Humanoid Soccer Robots
References
C. Zhou, “Robot learning with GA-based fuzzy reinforcement learning agents,” Information Sciences 145 (2002)
C. Zhou, “Fuzzy-arithmetic-based Lyapunov synthesis to the design of stable fuzzy controllers: a computing with words approach,” Int. J. Applied Mathematics and Computer Science 12(3) (2002)
C. Zhou and Q. Meng, “Dynamic balance of a biped robot using fuzzy reinforcement learning agents,” Fuzzy Sets and Systems 134(1) (2003)
C. Zhou, P.K. Yue, Z. Tang and Z. Sun, “Development of Robo-Erectus: A soccer-playing humanoid robot,” Proc. IEEE-RAS Intl. Conf. on Humanoid Robots, CD-ROM, 2003.
Z. Tang, C. Zhou and Z. Sun, “Gait synthesizing for humanoid penalty kicking,”  Dynamics of Continuous, Discrete and Impulsive Systems, Series B, (2003)
D. Maravall, C. Zhou and J. Alonso, “Hybrid fuzzy control of inverted pendulum via vertical forces,” Int. J. of Intelligent Systems, 2004 (in press).
Development of Humanoid Soccer Robots

36. Development of Humanoid Soccer Robots
Acknowledgements
Staff Member
P.K. Yue, F.S. Choy, Nazeer Ahmed
M.F. Ercan, Mike Wong, H. Li
Research Associate
Z. Tang (Tsinghua U.), J. Ni (Shanghai Jiao Tong U.)
Technical Support Officer
H.M. Tan, W. Ye
Students
P.P. Khing, H. W. Yin, H.F. Lu, H.X. Tan, J.X. Teo,
Stephen Quah, H.M. Tan, Y.T. Tan
Development of Humanoid Soccer Robots

37. Thanks! Dr Changjiu Zhou
School of Electrical and Electronic Engineering
Singapore Polytechnic
Development of Humanoid Soccer Robots