admin-guide/mm/numaperf.rst

7 Some platforms may have multiple types of memory attached to a compute
8 node. These disparate memory ranges may share some characteristics, such
12 A system supports such heterogeneous memory by grouping each memory type
14 characteristics.  Some memory may share the same node as a CPU, and others
15 are provided as memory only nodes. While memory only nodes do not provide
16 CPUs, they may still be local to one or more compute nodes relative to
18 nodes with local memory and a memory only node for each of compute node::
20  +------------------+     +------------------+
21  | Compute Node 0   +-----+ Compute Node 1   |
23  +--------+---------+     +--------+---------+
25  +--------+---------+     +--------+---------+
27  +------------------+     +--------+---------+
29 A "memory initiator" is a node containing one or more devices such as
30 CPUs or separate memory I/O devices that can initiate memory requests.
31 A "memory target" is a node containing one or more physical address
32 ranges accessible from one or more memory initiators.
34 When multiple memory initiators exist, they may not all have the same
35 performance when accessing a given memory target. Each initiator-target
36 pair may be organized into different ranked access classes to represent
37 this relationship. The highest performing initiator to a given target
38 is considered to be one of that target's local initiators, and given
41 memory targets.
43 To aid applications matching memory targets with their initiators, the
44 kernel provides symlinks to each other. The following example lists the
45 relationship for the access class "0" memory initiators and targets::
47 	# symlinks -v /sys/devices/system/node/nodeX/access0/targets/
48 	relative: /sys/devices/system/node/nodeX/access0/targets/nodeY -> ../../nodeY
50 	# symlinks -v /sys/devices/system/node/nodeY/access0/initiators/
51 	relative: /sys/devices/system/node/nodeY/access0/initiators/nodeX -> ../../nodeX
53 A memory initiator may have multiple memory targets in the same access
54 class. The target memory's initiators in a given class indicate the
55 nodes' access characteristics share the same performance relative to other
59 The access class "1" is used to allow differentiation between initiators
68 Applications may wish to consider which node they want their memory to
72 memory node's access class 0 initiators as follows::
82 	# tree -P "read*|write*" /sys/devices/system/node/nodeY/access0/initiators/
84 	|-- read_bandwidth
85 	|-- read_latency
86 	|-- write_bandwidth
87 	`-- write_latency
93 The values reported here correspond to the rated latency and bandwidth
96 Access class 1 takes the same form but only includes values for CPU to
97 memory activity.
103 System memory may be constructed in a hierarchy of elements with various
104 performance characteristics in order to provide large address space of
105 slower performing memory cached by a smaller higher performing memory. The
106 system physical addresses memory  initiators are aware of are provided
107 by the last memory level in the hierarchy. The system meanwhile uses
108 higher performing memory to transparently cache access to progressively
111 The term "far memory" is used to denote the last level memory in the
113 initiator access, and the term "near memory" represents the fastest
117 L1, L2, L3) uses the CPU-side view where each increased level is lower
118 performing. In contrast, the memory cache level is centric to the last
119 level memory, so the higher numbered cache level corresponds to  memory
120 nearer to the CPU, and further from far memory.
122 The memory-side caches are not directly addressable by software. When
124 near memory cache if it is present. If it is not present, the system
125 accesses the next level of memory until there is either a hit in that
126 cache level, or it reaches far memory.
128 An application does not need to know about caching attributes in order
129 to use the system. Software may optionally query the memory cache
130 attributes in order to maximize the performance out of such a setup.
131 If the system provides a way for the kernel to discover this information,
132 for example with ACPI HMAT (Heterogeneous Memory Attribute Table),
133 the kernel will append these attributes to the NUMA node memory target.
135 When the kernel first registers a memory cache with a node, the kernel
141 a memory-side cache, or that information is not accessible to the kernel.
152 software to query::
156 	|-- index1
157 	|   |-- indexing
158 	|   |-- line_size
159 	|   |-- size
160 	|   `-- write_policy
162 The "indexing" will be 0 if it is a direct-mapped cache, and non-zero
163 for any other indexed based, multi-way associativity.
170 The "write_policy" will be 0 for write-back, and non-zero for
171 write-through caching.
178 - Section 5.2.27