1. Design and Implementation of Adaptive Signal Processing Systems Using Markov Decision Processes
Marilyn Wolf1
With contributions from:
Lin Li2, Adrian E. Sapio2, Jiahao Wu2, Yanzhou Liu2, Kyunghun Lee2, and Shuvra S. Bhattacharyya2,3
1. Georgia Institute of Technology, USA
2. University of Maryland, USA
3. Tampere University of Technology, Finland

2. Outline Motivation and Contribution Background
Framework for Design and Implementation of Adaptive Signal Processing Systems
Case Study of Channelizer/receiver Application
Conclusions

3. Motivation and Contribution
Modern signal processing applications impose increasing demands of:
Adaptivity
Efficiency
Reconfigurability
Flexibility
Challenges at many levels of system design, implementation and optimization:
Dynamically-changing working scenarios
Stringent constraints on energy-efficiency and real-time performance
Multidimensional design space

4. Motivation and Contribution
This research aims at developing a framework for design and implementation of adaptive embedded signal processing systems that integrates
automated, MDP-based generation of optimal reconfiguration policies
dataflow-based application modeling
implementation of embedded control software that carries out the generated reconfiguration policies

5. Background Markov Decision Processes (MDPs)
Probabilistic transitions combined with inputs.
Given an input at a state, next state is chosen probabilistically.
A policy p defines the actions in each state s.
Optimal policy maximizes rewards.

6. Background Dataflow Modeling
Basic concepts of dataflow modeling:
Digital Signal Processing (DSP) system directed dataflow graph
Computational functions nodes (actors)
Communication channels between actors edges (FIFOs)
Actor Firing: Actor execution as a discrete unit of computation
Token: The encapsulation of some well-defined amount of data.
Consumption/Production Rate: Number of tokens consumed/produced from/to the input/output FIFO during one actor firing.

7. Background LightWeight DataFlow (LWDF)
LightWeight DataFlow (LWDF): a programming methodology for integration, experimentation, and optimization with dataflow modeling approaches.
Actor Mode: Determines the dataflow behavior of the actor.
Enable function: Checks actor firing condition according to its current mode. This function can be bypassed at run time if static scheduling analysis can ensure the result.
Invoke function: Executes an actor firing according to its current mode.
Lightweight Dataflow Environment (LIDE):
Provides a compact set of application programming interfaces (APIs) that is used for constructing, connecting, and executing dataflow components such as actors, edges, and graphs.
LIDE APIs have been implemented in a variety of implementation languages, including C, Verilog, and CUDA

8. Framework for Adaptive Signal Processing Systems
We propose a novel framework, called Hierarchical MDP framework for Compact System-level Modeling (HMCSM).

9. Framework for Adaptive Signal Processing Systems
Hierarchical MDP Subsystem
A single Markov decision process (MDP) is transformed into a hierarchy of multiple MDPs that can be independently solved.
Such decomposition into a collection of simpler MDPs leads to more efficient design optimization.
The stochastic models of the environment and system include, for each of the environment and system, the definition of the state space and the state transition matrix (STM).
Control Actions represent the set of possible reconfiguration operations.
The Reward Function maps state-action pairs into scores that assess the utility of performing the associated control action during the given state.

10. Framework for Adaptive Signal Processing Systems
Parameterized LIDE Implementation
Dataflow graph implementation of the application developed using LIDE
Parameter updates are made by setting (at design time or run time) appropriate variables in this implementation
Configuration Control Machine (CCM)
Determines, based on the current environmental state and system state, whether or not to perform a dynamic reconfiguration operation
Determines the specific reconfiguration operation that is to be applied to the system if reconfiguration is to be performed
Policy Mapping Engine (MPE)
Translates control actions into updates to dynamic parameters in the embedded software

11. Case Study of Channelizer/receiver Application (1)

12. Case Study of Channelizer/receiver Application (2)
Init Graph: modeling construct in parameterized synchronous dataflow (PSDF) for reconfiguration functionality (determination
and propagation of new parameter values).
Body Graph: modeling construct in PSDF for core signal processing functionality associated with an application.
PSDF specification of channelizer subsystem.

13. Case Study of Channelizer/receiver Application (3)
Average processing power of all the available configurations.
Platform for Channelizer: Raspberry Pi 3 Model B
Device for Power Measurement: Tektronix Keithley Series 2280 Precision Measurement DC Power Supply

14. Case Study of Channelizer/receiver Application (4)
Comparison among MDP-generated policies and fixed-configuration designs.
Simulation results for MDP-I.
In addition to providing better energy efficiency compared to the fixed configuration designs, our MDP approach
can be configured systematically to generate a much larger set of trade-off options (Pareto-optimized fronts)
ensures optimality (with respect to the given reward function)

15. Solver running time Model size
MDP Solver: MATLAB-based open source solver, named MDPSOLVE
Platform for MDP Solver:
Processor: i7-4710HQ running at 2.50GHz
RAM: 12.0 GB
OS: 64-bit Windows 10
MDP-I solver running time: 294ms
MDP-II-a solver running time: 50.8ms
MDP-II-b solver running time: 41.5ms
In a deployment with a fixed processing system (MDP-II-b) and changing external environment (MDP-II-a), the hierarchical MDP scheme reduces the solver time from 294ms to 50.8ms, which is a factor of over 5.7X smaller.
Model size
MDP-I: 1.63MB
MDP-II: 265kB
MDP-II reduces model size by a factor of over 6.1X.

16. Conclusions
We propose the HMCSM framework for design and implementation of adaptive embedded signal processing systems.
HMCSM stands for Hierarchical MDP framework for Compact System-level Modeling.
HMCSM provides a structured methodology that integrates dataflow methods; MDP formulation using compact, hierarchical models; optimal policy generation at design time; and dynamic, system-level reconfiguration at run time.
We demonstrate the effectiveness of our new MDP-based system design framework through experiments with an adaptive wireless communications receiver.

17. Thanks!