1. Hybrid simulation of the use of products by controlling continuous behaviour with state machines
EDIProD’ 2006, Gronów, Poland
Wilfred van der Vegte,
September 2006
Faculty of Industrial Design Engineering

2. Concept of system architecture Pilot implementation and demo
Overview
Objective and scope
Research issues
Reasoning model
Concept of system architecture
Pilot implementation and demo
Discussion and conclusions
November 19, 2018

3. Objective of the research
simulation (definition):
performing experiments on
virtual models
New
computer tool
development
my goal:
enable designers
to perform behavioural simulations with
fully virtual systems
to simulate use processes of
consumer durables
the human involved in the use process is also virtual:
even his/her decision-making is simulated
simulation of autonomous virtual humans involved in use processes
November 19, 2018

4. Scope of the research New computer tool development
behaviour of artefacts
behaviour of products
to simulate use processes of
consumer durables
behaviour of humans
behaviour of surroundings
application field:
behaviour prediction in
conceptual design
November 19, 2018

5. Research issues
Integration of artefact/engineering aspects and human/ergonomics aspects
Integration of the various different types of artefact behaviour and human behaviour in simulation
Dealing with the variety of use processes:
Variety in artefacts (products, surroundings)
Variety in humans (size, capabilities, habits)
Variety in course of action: “designing for use” try to anticipate all possible forms of use, including “user errors”. Simulation can support the designer in this.
<phone idle>
pick up phone
<dial tone>
on hook
dial 5
<phone idle>
<internal call>
on hook
enter digit
<phone idle>
<third>
on hook
enter digit
<fourth>
<phone idle>
enter digit
on hook
<phone idle>
<validating>
error message
on hook
(Hsia et al., 1999)
ring
busy
<phone idle>
<connecting>
<disconnected>
<try again>
on hook
callee pickup
on hook
on hook
<phone idle>
<connected>
<phone idle>
on hook
talk
<phone idle>
<state>
<phone idle>
<finished>
on hook
callee on hook
path
operation
<phone idle>
<disconnected>
<state>
on hook
<phone idle>
Scenario tree for the use of a phone: “IF-THEN” production rules control the decision points
November 19, 2018

6. Reasoning model
November 19, 2018

7. A simple example: simulating the use of a pedal bin
state:
process without intervention by human or electronics:
is simulated as a continuous process based on laws of
physics 9 n
transitions (10, .. , n-1) not elaborated
outside bin 8 2 2b 3
4,5 1 2a 7 6
pedal
inside bin 2b 3 4
arm/hand/garbage object
transition: triggers use state machine to intervene (e.g. human decision to activate muscles);
part of discrete simulation
inclusive
(AND)
branching
exclusive
(OR) 7
garbage object 5
arm/hand 6 1
informal representation of a scenario structure: extension of the scenario tree with inclusive branching (parallel paths) and loops
physical situation change:
different physical laws start to become valid;
part of continuous simulation 2a
November 19, 2018

8. A common formal FSM representation: the Petri Net
transitions (10, .. , n-1) not elaborated 2 2b 3
4,5 1 2a 6 7 2b 8 9 7 3 4 5 6 1 2a
November 19, 2018

9. Concept of a system architecture for modelling and simulation
Hybrid resource integration module
Use state machine
module
Nucleus module
Modelling
Nucleus-based
object model
Link between
models
Use state machine
(behavioural model)
Discrete-event
based simulation of
interpreted physical
behaviour
Nucleus-based
simulation of phys-
ical behaviour
Link between
simulations
Simulation UI
User input, modelling feedback and simulation results
Physically-based modelling and simulation with particle clouds (Horváth, 2004; Rusák et al., 2004). Includes deformations, kinematics and multiphysics.
In this pilot implementation, rigid 3D volumetric modelling and simulation with MSC visualNastran 4D is used instead.
Focus of this paper: pilot implementation
using Simulink block diagrams and Simulink Stateflow
November 19, 2018

10. Pilot implementation of a simple RI model
Object modelling & continuous simulation (MSC VisualNastran 4D)
Link between models (SIMULINK block diagram)
Use state machine (Stateflow)
Simulation time
USM
MainControl 1
Thumb 4
ShoulderY 6
Elbow 2
Fingers 3
ShoulderZ 7
ShoulderX 5
Rest
entry: yArmspeed=0
StopForNow
Forward
entry: zArmspeed=2
Backward
entry: yArmspeed=-2
entry: Fingerspeed=0
entry: Elbowspeed=89
entry: Fingerspeed=90
Throw
entry: Fingerspeed=-90
entry: Elbowspeed=-89
InsideBin
AfterRelease
entry: Thumbspeed=90
OutsideBin
entry: xArmspeed=0
entry: zArmspeed=0
entry: Thumbspeed=-90
entry: Thumbspeed=0
entry: Elbowspeed=0
entry: xArmspeed=2
WatchLitter z
ArmControl xy
xyDisplacement
LitterFalling
zDisplacement
Wait
GoBack x y
yDisplacement
xDisplacement
Released
exit(Throw)
enter(Throw)
LitterHalted
BackInStartPosition
enter(WatchLitter.Wait)
enter(ArmControl.Wait)
enter(GoBack)
exit(GoBack)
[zlitter < 0.025]
exit(xDisplacement)
enter(xDisplacement)
yHandEqualsyLitter
xHandEqualsxLitter
zHandEqualszLitter
enter(y.Wait)
enter(x.Wait)
enter(xy.Wait)
enter(z.Wait)
exit(yDisplacement)
enter(yDisplacement)
exit(zDisplacement)
enter(zDisplacement)
Clock
Start of simulation
Litter distance
Distance
exceeded
Elbow Angle
xlitter
ylitter
zlitter
SimTime
Original angle
Events
reached
x_hand
Meters used in conditions
zero
Elbowspeed
Fingerspeed
Thumbspeed
xArmspeed
yArmspeed
zArmspeed
Meter values
Controls
y_hand
zero1
vNPlant
z_hand
zero2
x position of litter
y position of litter
z position of litter
November 19, 2018
absolute
Controls
velocity
Below threshold
of litter

11. Zooming in on the use state machine
process without intervention by human or electronics:
is simulated as a continuous process based on laws of
physics
USM
MainControl 1
Thumb 4
ShoulderY 6
Elbow 2
Fingers 3
ShoulderZ 7
ShoulderX 5
Rest
entry: yArmspeed=0
StopForNow
Forward
entry: zArmspeed=2
Backward
entry: yArmspeed=-2
entry: Fingerspeed=0
entry: Elbowspeed=89
entry: Fingerspeed=90
Throw
entry: Fingerspeed=-90
entry: Elbowspeed=-89
InsideBin
AfterRelease
entry: Thumbspeed=90
OutsideBin
entry: xArmspeed=0
entry: zArmspeed=0
entry: Thumbspeed=-90
entry: Thumbspeed=0
entry: Elbowspeed=0
entry: xArmspeed=2
WatchLitter z
ArmControl xy
xyDisplacement
LitterFalling
zDisplacement
Wait
GoBack x y
yDisplacement
xDisplacement
Released
exit(Throw)
enter(Throw)
LitterHalted
BackInStartPosition
enter(WatchLitter.Wait)
enter(ArmControl.Wait)
enter(GoBack)
exit(GoBack)
[zlitter < 0.025]
exit(xDisplacement)
enter(xDisplacement)
yHandEqualsyLitter
xHandEqualsxLitter
zHandEqualszLitter
enter(y.Wait)
enter(x.Wait)
enter(xy.Wait)
enter(z.Wait)
exit(yDisplacement)
enter(yDisplacement)
exit(zDisplacement)
enter(zDisplacement)
USM
MainControl 1
Thumb 4
ShoulderY 6
Elbow 2
Fingers 3
ShoulderZ 7
ShoulderX 5
Rest
entry: yArmspeed=0
StopForNow
Forward
entry: zArmspeed=2
Backward
entry: yArmspeed=-2
entry: Fingerspeed=0
entry: Elbowspeed=89
entry: Fingerspeed=90
Throw
entry: Fingerspeed=-90
entry: Elbowspeed=-89
InsideBin
AfterRelease
entry: Thumbspeed=90
OutsideBin
entry: xArmspeed=0
entry: zArmspeed=0
entry: Thumbspeed=-90
entry: Thumbspeed=0
entry: Elbowspeed=0
entry: xArmspeed=2
WatchLitter z
ArmControl xy
xyDisplacement
LitterFalling
zDisplacement
Wait
GoBack x y
yDisplacement
xDisplacement
Released
exit(Throw)
enter(Throw)
LitterHalted
BackInStartPosition
enter(WatchLitter.Wait)
enter(ArmControl.Wait)
enter(GoBack)
exit(GoBack)
[zlitter < 0.025]
exit(xDisplacement)
enter(xDisplacement)
yHandEqualsyLitter
xHandEqualsxLitter
zHandEqualszLitter
enter(y.Wait)
enter(x.Wait)
enter(xy.Wait)
enter(z.Wait)
exit(yDisplacement)
enter(yDisplacement)
exit(zDisplacement)
enter(zDisplacement)
transition: triggers use state machine to intervene (e.g. human decision to activate muscles);
November 19, 2018

12. Demonstration video The video is available as a separate download.
It shows two simulations of the use of a basic example product: an open litter bin. In the first simulation, an irregularly shaped object is thrown into the bin. Once the object has left the hand, the USM commands the muscles of the simulated human to rotate the arm back to its starting position. Once the object lies still on the bottom of the bin, the simulation stops.
In the second simulation, the throwing speed is reduced. The object lands outside the bin. Once the object lies still, the USM commands the arm to move the hand to the position of the object and the simulation ends. A next step would have been pick up the object, move the arm back to the starting position and try again with increased throwing speed. However, effective simulation gripping turned out to be difficult without the possibility to include deformations of the hand.
November 19, 2018

13. Discussion and conclusions
Proof of ideas: for a basic application example, it is possible to create and simulate a hybrid model
More complex examples are needed to show the value of this approach for designers
Complex examples need simulation of more types of physical behaviour – not just rigid-body kinetics  nucleus-based modelling
Statecharts may not be the ideal FSM representation for information processing in a use process : even for this basic example case a quite complex behavioural model is needed.
‘Robotic’ human-motion patterns need to be replaced by realistic motion patterns from biomechanics and human-motor research
November 19, 2018