1Introduction 2============ 3 4This document descibes a collection of device-mapper targets that 5between them implement thin-provisioning and snapshots. 6 7The main highlight of this implementation, compared to the previous 8implementation of snapshots, is that it allows many virtual devices to 9be stored on the same data volume. This simplifies administration and 10allows the sharing of data between volumes, thus reducing disk usage. 11 12Another significant feature is support for an arbitrary depth of 13recursive snapshots (snapshots of snapshots of snapshots ...). The 14previous implementation of snapshots did this by chaining together 15lookup tables, and so performance was O(depth). This new 16implementation uses a single data structure to avoid this degradation 17with depth. Fragmentation may still be an issue, however, in some 18scenarios. 19 20Metadata is stored on a separate device from data, giving the 21administrator some freedom, for example to: 22 23- Improve metadata resilience by storing metadata on a mirrored volume 24 but data on a non-mirrored one. 25 26- Improve performance by storing the metadata on SSD. 27 28Status 29====== 30 31These targets are very much still in the EXPERIMENTAL state. Please 32do not yet rely on them in production. But do experiment and offer us 33feedback. Different use cases will have different performance 34characteristics, for example due to fragmentation of the data volume. 35 36If you find this software is not performing as expected please mail 37dm-devel@redhat.com with details and we'll try our best to improve 38things for you. 39 40Userspace tools for checking and repairing the metadata are under 41development. 42 43Cookbook 44======== 45 46This section describes some quick recipes for using thin provisioning. 47They use the dmsetup program to control the device-mapper driver 48directly. End users will be advised to use a higher-level volume 49manager such as LVM2 once support has been added. 50 51Pool device 52----------- 53 54The pool device ties together the metadata volume and the data volume. 55It maps I/O linearly to the data volume and updates the metadata via 56two mechanisms: 57 58- Function calls from the thin targets 59 60- Device-mapper 'messages' from userspace which control the creation of new 61 virtual devices amongst other things. 62 63Setting up a fresh pool device 64------------------------------ 65 66Setting up a pool device requires a valid metadata device, and a 67data device. If you do not have an existing metadata device you can 68make one by zeroing the first 4k to indicate empty metadata. 69 70 dd if=/dev/zero of=$metadata_dev bs=4096 count=1 71 72The amount of metadata you need will vary according to how many blocks 73are shared between thin devices (i.e. through snapshots). If you have 74less sharing than average you'll need a larger-than-average metadata device. 75 76As a guide, we suggest you calculate the number of bytes to use in the 77metadata device as 48 * $data_dev_size / $data_block_size but round it up 78to 2MB if the answer is smaller. The largest size supported is 16GB. 79 80If you're creating large numbers of snapshots which are recording large 81amounts of change, you may need find you need to increase this. 82 83Reloading a pool table 84---------------------- 85 86You may reload a pool's table, indeed this is how the pool is resized 87if it runs out of space. (N.B. While specifying a different metadata 88device when reloading is not forbidden at the moment, things will go 89wrong if it does not route I/O to exactly the same on-disk location as 90previously.) 91 92Using an existing pool device 93----------------------------- 94 95 dmsetup create pool \ 96 --table "0 20971520 thin-pool $metadata_dev $data_dev \ 97 $data_block_size $low_water_mark" 98 99$data_block_size gives the smallest unit of disk space that can be 100allocated at a time expressed in units of 512-byte sectors. People 101primarily interested in thin provisioning may want to use a value such 102as 1024 (512KB). People doing lots of snapshotting may want a smaller value 103such as 128 (64KB). If you are not zeroing newly-allocated data, 104a larger $data_block_size in the region of 256000 (128MB) is suggested. 105$data_block_size must be the same for the lifetime of the 106metadata device. 107 108$low_water_mark is expressed in blocks of size $data_block_size. If 109free space on the data device drops below this level then a dm event 110will be triggered which a userspace daemon should catch allowing it to 111extend the pool device. Only one such event will be sent. 112Resuming a device with a new table itself triggers an event so the 113userspace daemon can use this to detect a situation where a new table 114already exceeds the threshold. 115 116Thin provisioning 117----------------- 118 119i) Creating a new thinly-provisioned volume. 120 121 To create a new thinly- provisioned volume you must send a message to an 122 active pool device, /dev/mapper/pool in this example. 123 124 dmsetup message /dev/mapper/pool 0 "create_thin 0" 125 126 Here '0' is an identifier for the volume, a 24-bit number. It's up 127 to the caller to allocate and manage these identifiers. If the 128 identifier is already in use, the message will fail with -EEXIST. 129 130ii) Using a thinly-provisioned volume. 131 132 Thinly-provisioned volumes are activated using the 'thin' target: 133 134 dmsetup create thin --table "0 2097152 thin /dev/mapper/pool 0" 135 136 The last parameter is the identifier for the thinp device. 137 138Internal snapshots 139------------------ 140 141i) Creating an internal snapshot. 142 143 Snapshots are created with another message to the pool. 144 145 N.B. If the origin device that you wish to snapshot is active, you 146 must suspend it before creating the snapshot to avoid corruption. 147 This is NOT enforced at the moment, so please be careful! 148 149 dmsetup suspend /dev/mapper/thin 150 dmsetup message /dev/mapper/pool 0 "create_snap 1 0" 151 dmsetup resume /dev/mapper/thin 152 153 Here '1' is the identifier for the volume, a 24-bit number. '0' is the 154 identifier for the origin device. 155 156ii) Using an internal snapshot. 157 158 Once created, the user doesn't have to worry about any connection 159 between the origin and the snapshot. Indeed the snapshot is no 160 different from any other thinly-provisioned device and can be 161 snapshotted itself via the same method. It's perfectly legal to 162 have only one of them active, and there's no ordering requirement on 163 activating or removing them both. (This differs from conventional 164 device-mapper snapshots.) 165 166 Activate it exactly the same way as any other thinly-provisioned volume: 167 168 dmsetup create snap --table "0 2097152 thin /dev/mapper/pool 1" 169 170Deactivation 171------------ 172 173All devices using a pool must be deactivated before the pool itself 174can be. 175 176 dmsetup remove thin 177 dmsetup remove snap 178 dmsetup remove pool 179 180Reference 181========= 182 183'thin-pool' target 184------------------ 185 186i) Constructor 187 188 thin-pool <metadata dev> <data dev> <data block size (sectors)> \ 189 <low water mark (blocks)> [<number of feature args> [<arg>]*] 190 191 Optional feature arguments: 192 - 'skip_block_zeroing': skips the zeroing of newly-provisioned blocks. 193 194 Data block size must be between 64KB (128 sectors) and 1GB 195 (2097152 sectors) inclusive. 196 197 198ii) Status 199 200 <transaction id> <used metadata blocks>/<total metadata blocks> 201 <used data blocks>/<total data blocks> <held metadata root> 202 203 204 transaction id: 205 A 64-bit number used by userspace to help synchronise with metadata 206 from volume managers. 207 208 used data blocks / total data blocks 209 If the number of free blocks drops below the pool's low water mark a 210 dm event will be sent to userspace. This event is edge-triggered and 211 it will occur only once after each resume so volume manager writers 212 should register for the event and then check the target's status. 213 214 held metadata root: 215 The location, in sectors, of the metadata root that has been 216 'held' for userspace read access. '-' indicates there is no 217 held root. This feature is not yet implemented so '-' is 218 always returned. 219 220iii) Messages 221 222 create_thin <dev id> 223 224 Create a new thinly-provisioned device. 225 <dev id> is an arbitrary unique 24-bit identifier chosen by 226 the caller. 227 228 create_snap <dev id> <origin id> 229 230 Create a new snapshot of another thinly-provisioned device. 231 <dev id> is an arbitrary unique 24-bit identifier chosen by 232 the caller. 233 <origin id> is the identifier of the thinly-provisioned device 234 of which the new device will be a snapshot. 235 236 delete <dev id> 237 238 Deletes a thin device. Irreversible. 239 240 trim <dev id> <new size in sectors> 241 242 Delete mappings from the end of a thin device. Irreversible. 243 You might want to use this if you're reducing the size of 244 your thinly-provisioned device. In many cases, due to the 245 sharing of blocks between devices, it is not possible to 246 determine in advance how much space 'trim' will release. (In 247 future a userspace tool might be able to perform this 248 calculation.) 249 250 set_transaction_id <current id> <new id> 251 252 Userland volume managers, such as LVM, need a way to 253 synchronise their external metadata with the internal metadata of the 254 pool target. The thin-pool target offers to store an 255 arbitrary 64-bit transaction id and return it on the target's 256 status line. To avoid races you must provide what you think 257 the current transaction id is when you change it with this 258 compare-and-swap message. 259 260'thin' target 261------------- 262 263i) Constructor 264 265 thin <pool dev> <dev id> 266 267 pool dev: 268 the thin-pool device, e.g. /dev/mapper/my_pool or 253:0 269 270 dev id: 271 the internal device identifier of the device to be 272 activated. 273 274The pool doesn't store any size against the thin devices. If you 275load a thin target that is smaller than you've been using previously, 276then you'll have no access to blocks mapped beyond the end. If you 277load a target that is bigger than before, then extra blocks will be 278provisioned as and when needed. 279 280If you wish to reduce the size of your thin device and potentially 281regain some space then send the 'trim' message to the pool. 282 283ii) Status 284 285 <nr mapped sectors> <highest mapped sector> 286