Slide 1

OpenStack NUMANova OpenStack

Slide 2

1 2 3 4 Name Rui Chen OpenStack Employer Huawei Weibo @kiwik Blog http://kiwik.github.io Who?

Slide 3

OpenStack A. NUMA B. Nova & NUMA C. Nova & NUMA & ? CONTENTS

Slide 4

OpenStack NUMA Non-Uniform Memory Access [1][1] OpenStack

Slide 5

NUMA SMP, NUMAMPP 1. SMP 2. NUMA 3. MPP Symmetric Multi-Processor Non-Uniform Memory Access Massive Parallel Processing

Slide 6

OpenStack Sharing RAM SMP

Slide 7

OpenStack Sharing RAM, local faster NUMA

Slide 8

OpenStack Sharing nothing MPP

Slide 9

OpenStack NUMA Topology

Slide 10

OpenStack Nova & NUMA guest NUMA node placement & topology [2][2] OpenStack

Slide 11

NUMA IcehouseNovalibvirt.xml host NUMAlibvirt NUMA nodecpuRAM guest Nova NUMA

Slide 12

OpenStack Use Case Use Case #1 guestvcpuRAMhost NUMA node NUMA nodeRAMguest 1hw:numa_nodes=1

Slide 13

OpenStack Use Case Use Case #2 guestvcpuRAMhost NUMA node guestguest NUMA node host NUMA topologyguest OSguest NUMA DB 2hw:numa_nodes=N

Slide 14

OpenStack Use Case Use Case #3 RAMguesthost NUMA nodeguestNUMA NFV/ 3hw:numa_nodes=N

Slide 15

OpenStack What is guest in QEMU/KVM? 1. Guest 2. vCPU ComputeNodeLinux Guesthost OS How to place a guest into a NUMA node? 1. Schedule vCPU threads to pCPU set of a NUMA node 2. Allocate RAM from local

Slide 16

OpenStack flavor & image hw:numa_nodes=NN guest NUMA node hw:numa_mempolicy=preferred|strict RAMJuno hw:numa_cpus.0= - guest NUMA node0vcpu hw:numa_cpus.1= - guest NUMA node1vcpu hw:numa_mem.0= - guest NUMA node0RAM hw:numa_mem.1= - guest NUMA node1RAM

Slide 17

OpenStack Nova 12 34 Nova Workflow nova-api guest NUMA topology instance_extra nova-scheduler host guest NUMA topology nova-compute instance_claim hostNUMA guest NUMA host NUMA libvirt driver libvirt.xml 5 resource tracker host numa

Slide 18

OpenStack Host NUMA Host&Guest NUMA topology 8 vcpus 2 numa nodes 512M RAM 2 NUMA nodes 40G RAM per node CPU&RAM distance Nova stack@openstack:/home/devstack$ [master]$ numactl --hardware available: 2 nodes (0-1) node 0 cpus: 4 5 6 7 12 13 14 15 node 0 size: 40232 MB node 0 free: 36833 MB node 1 cpus: 0 1 2 3 8 9 10 11 node 1 size: 40318 MB node 1 free: 37822 MB node distances: node 0 1 0: 10 20 1: 20 10 Guest NUMA +----------------------------+------------------------------------------------------------+ | Property | Value | +----------------------------+------------------------------------------------------------+ | OS-FLV-DISABLED:disabled | False | | OS-FLV-EXT-DATA:ephemeral | 0 | | disk | 1 | | extra_specs | {"hw:numa_cpus.0": "0,1,2,3,4,5", "hw:numa_cpus.1": "6,7", | | "hw:numa_nodes": "2", "hw:numa_mem.1": "128", | | "hw:numa_mem.0": "384"} | | id | 11 | | name | chenrui_f | | os-flavor-access:is_public | True | | ram | 512 | | rxtx_factor | 1.0 | | swap | | | vcpus | 8 | +----------------------------+------------------------------------------------------------+

Slide 19

OpenStack Libvirt.xml pin vcpu to pcpu set in cputune export guest numa in cpu/numa 524288 8

Slide 20

OpenStack Double check stack@openstack:~/logs$ $ sudo numastat -c qemu-system-x86_64 Per-node process memory usage (in MBs) PID Node 0 Node 1 Total --------------- ------ ------ ----- 7161 (qemu-syste 703 12 714 11839 (qemu-syst 700 13 712 12774 (qemu-syst 162 550 713 15115 (qemu-syst 12 698 711 16379 (qemu-syst 111 5 117 17493 (sudo) 2 1 2 --------------- ------ ------ ----- Total 1689 1279 2969 stack@openstack:~/data/nova/instances/3722ba51-cefa-48e1-b11e-fd6ec14865ee$ $ virsh vcpuinfo 10 VCPU: 0 CPU: 4 State: running CPU time: 1.9s CPU Affinity: ----yyyy----yyyy VCPU: 1 CPU: 15 State: running CPU time: 0.4s CPU Affinity: ----yyyy----yyyy VCPU: 2 CPU: 5 State: running CPU time: 0.0s CPU Affinity: ----yyyy----yyyy VCPU: 3 CPU: 12 State: running CPU time: 0.0s CPU Affinity: ----yyyy----yyyy VCPU: 4 CPU: 13 State: running CPU time: 0.0s CPU Affinity: ----yyyy----yyyy VCPU: 5 CPU: 14 State: running CPU time: 0.1s CPU Affinity: ----yyyy----yyyy VCPU: 6 CPU: 2 State: running CPU time: 0.0s CPU Affinity: yyyy----yyyy---- VCPU: 7 CPU: 0 State: running CPU time: 0.1s CPU Affinity: yyyy----yyyy---- numastat virsh vcpuinfo

Slide 21

OpenStack Where is RAM? Why? the kernel will always try to allocate memory from the NUMA node that matches the one the guest CPUs are executing on. 524288 8

Slide 22

OpenStack Nova & NUMA & ? I/O NUMA scheduling, vcpu pin, large pages, etc. [3][3] OpenStack

Slide 23

NFV 1 NUMA-placement https://blueprints.launchpad.net/nova/+spec/virt-driver-numa-placement 2 vCPU topology https://blueprints.launchpad.net/nova/+spec/virt-driver-vcpu-topology 3 IO based NUMA scheduling https://blueprints.launchpad.net/nova/+spec/input-output-based-numa-scheduling 4 cpu pinning https://blueprints.launchpad.net/nova/+spec/virt-driver-cpu-pinning 5 Large pages https://blueprints.launchpad.net/nova/+spec/virt-driver-large-pages 6 SR-IOV https://blueprints.launchpad.net/nova/+spec/pci-passthrough-sriov 7 vhost user https://blueprints.launchpad.net/nova/+spec/vif-vhostuser

Slide 24

OpenStack Thanks!