Memory Network: Enabling Technology for Scalable Near-Data Computing Gwangsun Kim, John Kim Korea Advanced Institute of Science and Technology Jung Ho Ahn Seoul National University Yongkee Kwon SK Hynix

Memory Network I/O port … Vault controller I/O port Intra-HMC Network Vault controller … Logic layer High-speed link DRAM layers Vault Hybrid Memory Cube (HMC) 2/10  “Near”-data processing with multiple memories?  “Far”-data?  Memory network enables scalable near-data computing. Data AData B “compute A+B” Data B

DIVA Processing-in-Memory (PIM) Chip Draper et al., “The architecture of the DIVA processing-in-memory chip”, ICS’02  For multimedia and irregular applications.  Proposed memory network for PIM modules.  Simple low-dimensional network (e.g., ring )  High packet hop count  performance & energy inefficiency  Advanced technology is available – high off-chip bandwidth 3/10

Memory Networks from Micron D. R. Resnick, “Memory Network Methods, Apparatus, and Systems,” US Patent Application Publication, US A1, D Mesh topology 4/10 Local memories Network-attached memories

Memory Network Design Issues  Difficult to leverage high-radix topology – Low-radix vs. high-radix topology – High-radix topology  smaller network diameter – Limited # of ports in memory modules.  Adaptive routing requirement – Can increase network cost – Depends on traffic pattern, memory mapping, etc. 5/10 Low-radix networks High-radix networks

Memory-centric Network  Host-memory bandwidth still matters. – To support conventional applications while adopting NDP. – NDP involves communication with host processors.  MCN Leverage the same network for NDP. Network … Processor-centric Network (PCN) (e.g., Intel QPI, AMD HyperTransport) … … CPU Network … … … CPU Memory-centric Network (MCN) [PACT’13] Memory BW Processor-to processor BW Flexible BW utilization Separate network required for NDP The same network can be used for NDP 6/10

Memory Network for Heterogeneous NDP  NDP for not only CPU, but also for GPU.  Unified memory network for multi-GPU systems [MICRO’14].  Extending the memory network for heterogeneous NDP. CPUGPU … … Unified Memory Network … … … … … …… … FPGA … … … 7/10

Hierarchical Network  With intra-HMC network, the memory network is a hierarchical network.  NDP requires additional processing elements at the logic layer.  Need to support various types of traffic – Local (on-chip) traffic vs. global traffic – Conventional memory access traffic vs. NDP-induced traffic DRAM (stacked) Hybrid Memory Cube Vault controller On-chip channel I/O port Concentrated Mesh-based intra-HMC network [PACT’13] 8/10

Issues with Memory Network-based NDP  Power management – Large number of channels possible in memory network – Power-gating, DVFS, and other circuit-level techniques.  Data placement & migration – Optimal placement of shared data – Migration within memory network  Consistency & coherence – Direct memory access by multiple processors – Heterogeneous processors 9/10

Summary  Memory network can enable scalable near-data processing.  Leveraging recent memory network researches – Memory-centric network [PACT’13] – Unified memory network [MICRO’14]  Intra-HMC design considerations  Further issues 10/10