1 00:00:00,090 --> 00:00:10,800 ‫And let's talk about durability, durability, in a nutshell, is the process of persisting the rights 2 00:00:10,800 --> 00:00:21,330 ‫that clients make to the database to a nonvolatile system storage system that means anything that we 3 00:00:21,330 --> 00:00:23,190 ‫write is stored. 4 00:00:23,760 --> 00:00:32,640 ‫So that I mean, even if I lost power or even if I crashed as a client after my committing that transaction, 5 00:00:32,970 --> 00:00:35,460 ‫if I come back, I see the change. 6 00:00:35,580 --> 00:00:39,210 ‫If the database crashed after the commit, I restore the database. 7 00:00:39,210 --> 00:00:40,220 ‫I see that change. 8 00:00:40,230 --> 00:00:43,470 ‫I see it persisted in desk if my computer crashed. 9 00:00:43,860 --> 00:00:45,360 ‫I come back, I see this change. 10 00:00:45,450 --> 00:00:46,950 ‫That's what durability is. 11 00:00:47,520 --> 00:00:49,720 ‫And you might think of that as this, OK? 12 00:00:49,740 --> 00:00:51,090 ‫Of course, everything is durable. 13 00:00:51,510 --> 00:00:52,200 ‫Not really. 14 00:00:52,230 --> 00:00:55,260 ‫A lot of database system actually play it with that. 15 00:00:55,710 --> 00:01:02,490 ‫They play with the durability says OK, durability is slow because to make things durable, that means 16 00:01:02,490 --> 00:01:05,520 ‫you have to be persistent to disk and disk are inherently slow. 17 00:01:05,820 --> 00:01:08,670 ‫So a lot of databases play around. 18 00:01:08,670 --> 00:01:10,020 ‫This durability is okay. 19 00:01:10,290 --> 00:01:14,340 ‫I'm going to write my stuff to memory and occasionally on the background. 20 00:01:14,340 --> 00:01:16,440 ‫I'm going to snapshot and write a disk. 21 00:01:16,440 --> 00:01:18,600 ‫So I make I'm going to make my race faster. 22 00:01:18,690 --> 00:01:21,210 ‫Can you get the database that does that right is right? 23 00:01:21,210 --> 00:01:22,950 ‫Is does that right as an option? 24 00:01:22,950 --> 00:01:23,490 ‫Obviously. 25 00:01:23,730 --> 00:01:23,970 ‫Right. 26 00:01:24,450 --> 00:01:30,030 ‫But durability is one of the stronger relational databases must have right. 27 00:01:30,040 --> 00:01:31,020 ‫You have to be durable. 28 00:01:31,170 --> 00:01:31,380 ‫Right. 29 00:01:31,730 --> 00:01:33,690 ‫And we're going to define what that means, really. 30 00:01:34,320 --> 00:01:40,590 ‫So durability changes made by a committed transaction, emphasis on committed has to be committed if 31 00:01:40,590 --> 00:01:41,370 ‫it's not committed. 32 00:01:42,080 --> 00:01:49,290 ‫Really, all bets are off must be persistent in and under an adorable, non-volatile storage like SSD 33 00:01:49,290 --> 00:01:50,040 ‫or hard drive. 34 00:01:52,770 --> 00:01:55,410 ‫Durability techniques, there are many durability techniques. 35 00:01:57,330 --> 00:01:58,080 ‫There is the wall. 36 00:01:58,230 --> 00:02:05,190 ‫The right to head log, which really you don't want out when I show you all the tables and indexes and 37 00:02:05,190 --> 00:02:05,940 ‫stuff like that. 38 00:02:06,600 --> 00:02:13,110 ‫These are really the representation of the data, which is what people read, what would what clients 39 00:02:13,110 --> 00:02:13,380 ‫read? 40 00:02:13,860 --> 00:02:16,980 ‫But these are really large if you think about it. 41 00:02:17,430 --> 00:02:27,120 ‫So if you if you rewrite these things to disk, which we, well, durability, committing a transaction 42 00:02:27,120 --> 00:02:28,860 ‫takes time because this is huge. 43 00:02:29,220 --> 00:02:32,970 ‫These data and the data structures and the B trees and stuff like that. 44 00:02:33,430 --> 00:02:33,960 ‫Stakes done. 45 00:02:34,230 --> 00:02:40,320 ‫So what databases improvement is a version of the changes that happen only, which is built as effectively 46 00:02:40,320 --> 00:02:41,820 ‫as you can think of it as the deltas? 47 00:02:42,210 --> 00:02:49,150 ‫And then this is a write only right to head log in what's called a wall and any changes. 48 00:02:49,150 --> 00:02:50,740 ‫It goes to this immediately. 49 00:02:50,760 --> 00:02:54,030 ‫First, we flush it immediately to disk. 50 00:02:54,480 --> 00:02:56,700 ‫And this guarantees persistence. 51 00:02:56,700 --> 00:03:03,900 ‫So yeah, we didn't really update the data table right on on another data structure yet, but we can. 52 00:03:03,900 --> 00:03:07,050 ‫If we have a crash, we have this beautiful wall. 53 00:03:07,350 --> 00:03:15,090 ‫We can go back and read all the wall entries and then rebuild this state effectively, easily from a 54 00:03:15,090 --> 00:03:15,930 ‫persistent estate. 55 00:03:16,080 --> 00:03:17,430 ‫We still persist this. 56 00:03:17,430 --> 00:03:23,040 ‫We still ride this durable right to disk like the actual table itself, the changes. 57 00:03:23,490 --> 00:03:26,280 ‫But we don't rely on that because that could be slow. 58 00:03:27,330 --> 00:03:31,950 ‫And there's another approach, which is asynchronous snapshots or even synchronous snapshots for that 59 00:03:31,950 --> 00:03:37,530 ‫matter, where we're as we as we write, we keep everything in memory. 60 00:03:37,950 --> 00:03:43,110 ‫But then asynchronously on the bag and on, we snapshot everything to desk at once. 61 00:03:43,500 --> 00:03:47,100 ‫And that is also another concept Regice uses. 62 00:03:47,100 --> 00:03:52,980 ‫Actually, both of these things append only file is another option, which is very similar to the wall 63 00:03:53,310 --> 00:04:00,270 ‫really that Redis uses, which is keeps tracks of the changes of what happens and then writes these 64 00:04:00,270 --> 00:04:00,570 ‫things. 65 00:04:01,080 --> 00:04:09,990 ‫This guarantees that we have a very, you know, lightweight way of storing data in a in a very fast 66 00:04:09,990 --> 00:04:17,910 ‫way, but also in case of a crash, I can always read back and construct the state of the table completely 67 00:04:17,910 --> 00:04:24,120 ‫without story, without the need of storing the likes and the idea and the string and the blob and all 68 00:04:24,120 --> 00:04:24,840 ‫that stuff, right? 69 00:04:25,020 --> 00:04:25,640 ‫We change all. 70 00:04:25,980 --> 00:04:29,370 ‫We store only these changes and eventually we're going to override that as well. 71 00:04:30,310 --> 00:04:31,440 ‫Let's talk about the wall a little bit. 72 00:04:32,230 --> 00:04:36,390 ‫Writing a lot of data to disk is expensive, as I talked about that a little bit. 73 00:04:37,200 --> 00:04:43,710 ‫We have indexes, we have data files, we have columns, rows, you know, we have system file system 74 00:04:43,710 --> 00:04:45,060 ‫tables, all this stuff. 75 00:04:45,060 --> 00:04:52,050 ‫Or I think if you imagine writing all the changes and the amplification of the rights itself, that's 76 00:04:52,050 --> 00:04:53,380 ‫going to take a long time. 77 00:04:53,400 --> 00:04:55,000 ‫You can do it, but it's going to be a lot. 78 00:04:55,530 --> 00:04:58,040 ‫So we need a better way. 79 00:04:58,050 --> 00:05:00,170 ‫We need a more optimized compact way. 80 00:05:00,930 --> 00:05:06,150 ‫So they compress this version of of these changes, known as wall right ahead log segments. 81 00:05:06,510 --> 00:05:14,280 ‫So the only released log log segment and this this better be persisted because this is what tells us 82 00:05:14,280 --> 00:05:16,200 ‫what to change effectively. 83 00:05:16,920 --> 00:05:17,220 ‫Right. 84 00:05:17,220 --> 00:05:20,610 ‫So these are just tell you, Hey, we changed this value to this. 85 00:05:20,610 --> 00:05:24,990 ‫We change this value to this, which is this value to this and we we even compress that story. 86 00:05:25,110 --> 00:05:28,600 ‫So we get it in a very tiny segments. 87 00:05:28,770 --> 00:05:36,660 ‫So there is something very critical in database system, and this is one of the problem the operating 88 00:05:36,660 --> 00:05:37,410 ‫systems do. 89 00:05:37,680 --> 00:05:44,880 ‫And this is called the OS cache when when you actually write to the OS, you ask the operating system, 90 00:05:44,880 --> 00:05:47,070 ‫what are the Windows, a Windows or Linux? 91 00:05:47,580 --> 00:05:52,410 ‫When you win the Davis, ask something to ask the OS to write to disk. 92 00:05:52,740 --> 00:06:00,000 ‫The OS actually does something shady, it writes not to this rights, to its own memory cache. 93 00:06:00,540 --> 00:06:00,900 ‫Right? 94 00:06:01,590 --> 00:06:09,870 ‫And the reason is because it will want to batch these rides and flush it to disk at once for for performance 95 00:06:09,870 --> 00:06:10,320 ‫reasons. 96 00:06:10,620 --> 00:06:16,650 ‫Because most applications do does a lot of right and they they want to kind of batch these supplies 97 00:06:16,650 --> 00:06:17,610 ‫and do it once. 98 00:06:17,680 --> 00:06:24,300 ‫So because if you batch it, you did, you do less Io and if you do a less ill, as I always talk about 99 00:06:24,480 --> 00:06:25,380 ‫this, it was better. 100 00:06:26,640 --> 00:06:27,810 ‫But what's the problem with that? 101 00:06:28,080 --> 00:06:32,370 ‫What if I actually you say commit and the database? 102 00:06:32,370 --> 00:06:33,350 ‫Asked the database. 103 00:06:33,630 --> 00:06:41,280 ‫Ask the the OS to write the wall segments to the disk, but always say, Hey, I'm going to write it 104 00:06:41,280 --> 00:06:42,420 ‫to the cache instead. 105 00:06:42,510 --> 00:06:48,990 ‫The OS will tell the database that, hey, I wrote successfully, despite it, only it only wrote to 106 00:06:48,990 --> 00:06:49,770 ‫the cache. 107 00:06:50,010 --> 00:06:56,400 ‫It didn't write to the disk wall till the database that hey database or application doesn't know. 108 00:06:56,560 --> 00:07:01,180 ‫The database, just not an applications that ask her to write something and it will ask the database, 109 00:07:01,180 --> 00:07:07,060 ‫Hey, by the way, yeah, I'm the voice and I wrote your changes successfully, but it only brought 110 00:07:07,060 --> 00:07:15,430 ‫it to the cache, so the database will say, OK, sure, the database will say your transaction committed 111 00:07:15,430 --> 00:07:20,230 ‫successfully, but the problem is what if you get a crash there? 112 00:07:21,070 --> 00:07:23,380 ‫The database told us it committed successfully. 113 00:07:24,070 --> 00:07:29,740 ‫You get a crash, your computer crashed, you restart and then the odds, obviously. 114 00:07:30,990 --> 00:07:36,000 ‫The data was in the cache and the oice cache when you restart, it was in the RAM, obviously. 115 00:07:36,000 --> 00:07:37,680 ‫So look, we lost the data. 116 00:07:38,070 --> 00:07:40,200 ‫And as a result, the database lied to us. 117 00:07:40,440 --> 00:07:47,070 ‫The database in that particular situation is not durable because you wrote the transaction because the 118 00:07:47,070 --> 00:07:51,450 ‫transaction told us that it was committed, but it wasn't durable. 119 00:07:52,290 --> 00:07:52,770 ‫Why? 120 00:07:53,220 --> 00:07:54,780 ‫Because if there was a problem here. 121 00:07:55,470 --> 00:08:03,060 ‫So when the rights go to the OS and the OS crashed machine restart could lead to loss of the data. 122 00:08:03,420 --> 00:08:05,850 ‫And that's a big problem that Davies's fight for. 123 00:08:06,630 --> 00:08:13,440 ‫So there is this command that bypasses the OS cache and ask the database to ask the operating system, 124 00:08:13,440 --> 00:08:16,500 ‫Hey, you dumb os, I don't trust you. 125 00:08:16,890 --> 00:08:21,750 ‫I tell you to flush these wall changes, flush them directly to the disk. 126 00:08:22,350 --> 00:08:27,360 ‫This is called the F Sync Command, so I've set command just forces the right always to do so. 127 00:08:27,360 --> 00:08:28,500 ‫It's going to be slow. 128 00:08:28,680 --> 00:08:30,120 ‫It's going to be very, very slow. 129 00:08:30,450 --> 00:08:35,070 ‫So we have to do it in a smart way because you're bypassing the OS gash. 130 00:08:35,790 --> 00:08:37,080 ‫And you might say why? 131 00:08:37,110 --> 00:08:37,570 ‫Why this? 132 00:08:37,590 --> 00:08:43,110 ‫The OS uses the cache right isn't the dangerous component of most of the situations, like you don't 133 00:08:43,110 --> 00:08:44,550 ‫really get a crash that often. 134 00:08:44,550 --> 00:08:51,890 ‫So if you save award file on a notepad or something like things like that, it's OK to use the OS cache. 135 00:08:51,900 --> 00:08:58,590 ‫Most of the applications use the OS cash, and it's very rare that this crash can happen. 136 00:08:59,730 --> 00:09:01,080 ‫Database system is a different. 137 00:09:01,350 --> 00:09:05,880 ‫They want to guarantee durability, and as a result, they have to bypass the OS cat and as a result, 138 00:09:06,030 --> 00:09:06,900 ‫it becomes slow. 139 00:09:08,190 --> 00:09:09,720 ‫You see the trade off, guys, right? 140 00:09:10,290 --> 00:09:11,730 ‫There is always a trade off. 141 00:09:13,290 --> 00:09:17,250 ‫So if it can be expensive and slow, does commence, right, this is another problem, right? 142 00:09:17,280 --> 00:09:19,860 ‫If you commit, you want to commit as fast as possible. 143 00:09:21,160 --> 00:09:26,950 ‫So let's summarize the final lecture here, durability is the ability to persist that changes when the 144 00:09:26,950 --> 00:09:29,900 ‫transaction commenced, the emphasis on the one transaction commits. 145 00:09:30,220 --> 00:09:33,520 ‫So when the transaction commence, I want you to have persisted on desk. 146 00:09:33,880 --> 00:09:34,750 ‫I don't care. 147 00:09:35,740 --> 00:09:37,090 ‫I don't care where he persisted. 148 00:09:37,240 --> 00:09:43,660 ‫Here's what I want when I when you tell me it's commit, I can shut off the power at that exact moment. 149 00:09:44,320 --> 00:09:47,200 ‫And when I come back, my data better be there. 150 00:09:47,230 --> 00:09:48,340 ‫That's what durability means. 151 00:09:49,060 --> 00:09:51,900 ‫I don't care going to store it on, on, on the desk. 152 00:09:51,910 --> 00:09:56,170 ‫And this is the only storage on, I don't know, on the air. 153 00:09:56,320 --> 00:09:57,020 ‫Who cares? 154 00:09:57,040 --> 00:09:57,830 ‫I don't care. 155 00:09:57,850 --> 00:09:59,380 ‫I want my data back. 156 00:09:59,560 --> 00:10:00,490 ‫That's what durability is. 157 00:10:00,910 --> 00:10:08,770 ‫A lot of databases play a trade off with this properties because durability is slows, writes down because 158 00:10:08,770 --> 00:10:14,200 ‫now I'm going through this physical medium to process my changes, you know? 159 00:10:14,860 --> 00:10:17,800 ‫And as a result, it will be slower than if you brought to memory. 160 00:10:18,080 --> 00:10:24,730 ‫So riders, for example, plays some games or that gives you option actually, that none of the databases 161 00:10:24,730 --> 00:10:27,310 ‫that I know of even relational gives you that option. 162 00:10:28,300 --> 00:10:29,140 ‫Because guess what? 163 00:10:29,590 --> 00:10:30,460 ‫It gives you the option. 164 00:10:30,460 --> 00:10:35,710 ‫It says, Hey, you want to be completely durable, strongly durable. 165 00:10:36,040 --> 00:10:37,060 ‫Who do you want to be here? 166 00:10:37,510 --> 00:10:40,210 ‫Partially durable, not even partial posture was the wrong one. 167 00:10:40,870 --> 00:10:47,500 ‫Eventually, durable, I guess I just made up that word, but eventually is going to persist to disk. 168 00:10:47,680 --> 00:10:57,010 ‫So there is a slight few seconds delay that if you lose power off of or if you crash, you could lose 169 00:10:57,010 --> 00:10:57,520 ‫your data. 170 00:10:58,180 --> 00:11:06,850 ‫Some people trade that off their three seconds with a fast rights because, hey, I rather get fast, 171 00:11:06,850 --> 00:11:07,240 ‫right? 172 00:11:07,630 --> 00:11:10,190 ‫And OK, my data is not that important. 173 00:11:10,210 --> 00:11:14,830 ‫I'm stalling just logs off of my application IoT devices. 174 00:11:14,950 --> 00:11:16,390 ‫Who cares if I lose a few things? 175 00:11:16,840 --> 00:11:18,220 ‫But that's what right? 176 00:11:18,370 --> 00:11:19,030 ‫Most of the time. 177 00:11:19,450 --> 00:11:23,340 ‫Nobody reports loss because crashes are very rare. 178 00:11:23,350 --> 00:11:29,770 ‫If you think about it when it comes to the actual databases, those guys build the systems to not really 179 00:11:29,770 --> 00:11:30,010 ‫crash. 180 00:11:30,010 --> 00:11:35,830 ‫But you can't take that guarantee because if it's Murphy's Law, it says if it never happened, it will 181 00:11:35,830 --> 00:11:36,700 ‫eventually happen. 182 00:11:37,510 --> 00:11:38,120 ‫All right, guys. 183 00:11:38,140 --> 00:11:40,570 ‫That was the last lecture of this. 184 00:11:40,870 --> 00:11:43,480 ‫We're going to jump into many other lectures I've got to talk about. 185 00:11:44,860 --> 00:11:45,520 ‫I've enjoyed courses. 186 00:11:45,520 --> 00:11:47,050 ‫I'm going to show you examples here. 187 00:11:47,080 --> 00:11:49,450 ‫So go with another lecture. 188 00:11:49,880 --> 00:11:52,390 ‫It's called Essid example by practice. 189 00:11:52,390 --> 00:11:55,270 ‫I believe that's what I called it and show. 190 00:11:55,450 --> 00:11:59,040 ‫See how I do this in real time with the actual database. 191 00:11:59,050 --> 00:12:04,060 ‫Show you all these beautiful properties can also talk about eventual consistency, and I'm going to 192 00:12:04,750 --> 00:12:09,970 ‫take take, take a stab at the quiz and enjoy the rest of the course. 193 00:12:10,660 --> 00:12:11,320 ‫Thank you so much. 194 00:12:11,620 --> 00:12:11,980 ‫Goodbye.