A look at memory issues Data-transfers must occur between system memory and the network interface controller.
Published byModified over 4 years ago
Presentation on theme: "A look at memory issues Data-transfers must occur between system memory and the network interface controller."— Presentation transcript:
A look at memory issues Data-transfers must occur between system memory and the network interface controller
Typical Chipset Layout MCH Memory Controller Hub (Northbridge) ICH I/O Controller Hub (Southbridge) CPU Central Processing Unit DRAM Dynamic Random Access Memory NIC Network Interface Controller HDC Hard Disk Controller AC Audio Controller Graphics Controller TimerKeyboardMouseClock Multimedia Controller Firmware Hub USB controller
Typical Chipset Layout MCH Memory Controller Hub (Northbridge) ICH I/O Controller Hub (Southbridge) CPU Central Processing Unit DRAM Dynamic Random Access Memory NIC Network Interface Controller HDC Hard Disk Controller AC Audio Controller Graphics Controller TimerKeyboardMouseClock Multimedia Controller Firmware Hub USB controller DMA
PCI Bus Master DMA 82573L i/o-memory RX and TX FIFOs (32-KB total) Host’s Dynamic Random Access Memory Descriptor Queue packet-buffer DMA on-chip RX descriptors on-chip TX descriptors
Memory-mapped I/O We mentioned that Intel’s x86 architecture originally was designed with two separate address-spaces, one for memory and the other for I/O ports, unlike the designs for CPUs by many of Intel’s competitors in which I/O access was “memory-mapped” But now the newer Intel processors also can support memory-mapped I/O as well
Address-bus widths The Intel Core-2 Quad processors in our classroom and Lab machines potentially could address 2 36 physical memory cells (i.e., 64GB), although only 4GB of RAM actually are installed at the present time Some PCI-compliant hardware devices were designed for a 32-bit address-bus, thus they must be “mapped” below 4G
Physical-address assignments The CPU’s physical address-space Devices’ registers must be mapped to addresses in the bottom 4G Dynamic Random Access Memory
Virtual addresses Software running on the x86 processor is unable to use actual memory addresses, but instead uses ‘virtual’ addresses that map to physical addresses by means of mapping-tables which Linux dynamically defines for each different process it runs This complicates the steps software must take to arrange for the DMA to take place
Our ‘dram.c’ module To help us confirm that our hardware-level network software is working as we intend, or to diagnose our ‘bugs’ if it isn’t, we can use an LKM we’ve written that implements a character-mode device-driver for system memory, allowing us to view the contents of physical memory as if it were a file; for example, by using our ‘fileview.cpp’ tool
How to view system memory Download ‘dram.c’ from course website Compile it using our ‘mmake’ utility Install ‘dram.ko’ by using ‘/sbin/insmod Insure the ‘/dev/dram’ device-node exists Download ‘fileview.cpp’ from website and compile it with ‘g++’ (or with ‘make’) Execute ‘fileview /dev/dram’ and use the arrow-keys to navigate (or hit )
4-level address-translation The CPU examines any virtual address it encounters, subdividing it into five fields offset into page-frame index into page-table 63 48 47 39 38 30 29 21 20 12 11 0 16-bits9-bits12-bits index into page- directory index into page- directory pointer table index into level 4 page-map table 9-bits sign- extension Any 48-bit virtual-address is sign-extended to a 64-bit “canonical” address Only “canonical” 64-bit virtual-addresses are legal in 64-bit mode
Reserved (must be 0) Format of 64-bit table-entries 63 62 52 51 40 39 32 31 12 11 9 8 7 6 5 4 3 2 1 0 Page-frame physical base-address[31..12] Page-frame physical base-address [39..32] EXBEXB PWU PWTPWT PCDPCD A avl Meaning of these bits varies with the table Legend: P = Present (1=yes, 0=no)PWT = Page Cache Disable (1=yes, 0=no) W = Writable (1=yes, 0=no)PWT = Page Write-Through (1=yes, 0=no) U = User-page (1=yes, 0=no) avl = available for user-defined purposes A = Accessed (1=yes, 0=no) EXB = Execution-disabled Bit (if EFER.NXE=1)
Our ‘mem64.c’ module We wrote an LKM to create a pseudo-file that will let us see how the virtual memory is being utilized by an application program Download this file, compile it with ‘mmake’ and install ‘mem64.ko’ in the Linux kernel Then view the virtual-memory mapping that is being used by the ‘cat’ program: $ cat /proc/mem64
The NIC’s PCI ‘resources’ Status Register Command Register DeviceID 0x109A VendorID 0x8086 BIST Cache Line Size Class Code Class/SubClass/ProgIF Revision ID Base Address 0 Subsystem Device ID Subsystem Vendor ID CardBus CIS Pointer reserved capabilities pointer Expansion ROM Base Address Minimum Grant Interrupt Pin reserved Latency Timer Header Type Base Address 1 Base Address 2Base Address 3 Base Address 4Base Address 5 Interrupt Line Maximum Latency 31 0 16 doublewords Dwords 1 - 0 3 - 2 5 - 4 7 - 6 9 - 8 11 - 10 13 - 12 15 - 14
Mechanisms compared kernel memory-space NIC i/o-memory CPU’s ‘virtual’ address-space io user memory-space Each NIC register has its own address in memory (allows one-step access) addrdata Access to all of the NIC’s registers is muliplexed through a pair of I/O-ports (requires multiple instructions) CPU’s ‘I/O’ address-space
‘nicstatus.c’ Here’s an LKM that creates a pseudo-file (called ‘/proc/nicstatus’) which will allow a user to view the current value in our Intel 82573L Network Interface Controller’s ‘DEVICE_STATUS’ register It uses the I/O-port interface to the NIC’s registers, rather than a ‘memory-mapped’ interface to those device-registers
0 Device Status (0x0008) ?0 0 000 000000 GIO Master EN 000 0000 PHY RA ASDV ILOSILOS SLUSLU 0 TX OFF 0 FDFD 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0 Function ID LULU 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 SPEED FD = Full-Duplex LU = Link Up TXOFF = Transmission Paused SPEED (00=10Mbps,01=100Mbps, 10=1000Mbps, 11=reserved) ASDV = Auto-negotiation Speed Detection Value PHYRA = PHY Reset Asserted 82573L some undocumented functionality?
‘82573.c’ This is a more elaborate example of an LKM which not only creates a pseudo-file (i.e., ‘/proc/82573’) that we can view using the Linux ‘cat’ command and that lets us see our NIC’s PCI Configuration Space, but also implements some device-driver functions that let us view the NIC’s device registers by using our ‘fileview.cpp’ tool
In-class exercise Two of the NIC’s 32-bit device registers are used to hold its 48-bit Ethernet MAC address – which will be a different value for each of the hosts in our classroom These two registers are located at offsets 0x5400 and 0x5404 in device-memory The six bytes occur in network byte-order Write code to show the MAC address!