‫MongoDB is a document based NoSQL database.
‫It became very, very popular for its schemaless way of storing documents.
‫You know, because the friction when it comes to writing code has gone.
‫Compared to SQL based databases.
‫But I get this question a lot, and I thought this video will be the perfect Segway to actually answer
‫that question.
‫What is really the difference between NoSQL and SQL?
‫So I'm going to address that in this video.
‫And but the main purpose of this video is actually going through the evolution of MongoDB internal architecture.
‫So this is a topic that very rarely people discuss because we're going into the bowels of the database,
‫not the front end, in a sense that how you interact with it and store data, right?
‫So we're talking about the actual architecture of the internals, right?
‫There's been evolution up until version 5.3.
‫A very interesting, uh, feature was added.
‫It's called the Clustered Collections.
‫So I'll go through this, the evolution of this.
‫So I'll discuss what is the difference between SQL and NoSQL and in a very deep way, and we'll discuss.
‫The first version of MongoDB started with their storage engine and V1, then the Wiredtiger acquiring
‫them back in 2014, I think, and then moving all the way to the recent changes, which is the clustered
‫collections.
‫And this is all going to make sense by the end of the video, hopefully.
‫How about we get started?
‫All right, so I'm going to use my medium article, at least the images in my medium article to illustrate
‫this.
‫Think of it as like as the slide shows, but go ahead and make sure to follow me on Medium.
‫I started posting a lot of content there.
‫If you're if you're like if you like more the written medium than actual videos.
‫But the first thing we're going to discuss is like the database internals.
‫If you look really at any database, any database almost always will have two pieces.
‫And the most piece that we actually deal with and interact with is actually the front end piece of the
‫database, which is the API.
‫You see the most popular databases API to actually communicate to the database, to tell it what to
‫fetch to, to, to actually ask it to store something.
‫Is the SQL language which state for structured query language, right?
‫And that is the API that we know and love.
‫And another piece of a different API could be like Redis or Mongo, right?
‫So hey, get this document and store this document.
‫There is no structured query language, There is no selecting tables and fields, right?
‫It's just its own different API.
‫So the API can actually change based on the database.
‫The second portion is actually the data format.
‫When I ask you to get something or I want to store something, what am I giving you and what am I taking
‫back from you?
‫And this is where really a database can shine.
‫It's it's the core of any database system, the data format.
‫So for the longest time, databases has always been tables and rows and columns.
‫Right.
‫And to interact with these rows and columns, you use the SQL language right.
‫To query it.
‫So when people design the database back in the 70 seconds or 60 seconds even, right, they thought
‫about it and always say it's always going to be tables and it's always going to be rows and always going
‫to be columns, and then the application can build on top of it, right?
‫We built it bottom up, if you will.
‫And then, uh.
‫So that's one data format, right?
‫But then people challenged this.
‫People came in the 2000 era and say, wait a minute, why do I have to be?
‫Really fix to these tables.
‫I don't know what my application has nothing to do with tables.
‫As the web evolved, as the as the evolution of the web, right.
‫Came in and JSON and documents, really, I want to deal with documents.
‫I don't even have a schema per se.
‫I don't have tables with a specific schema.
‫I want to be flexible.
‫Why are you forcing me to do tables?
‫And that's the idea of where documents came in Later, graphs came in later.
‫Other column based storage came in, right?
‫And instead of row storage, all of this, it really automatically the database became this two part
‫where the front end and the storage engine, which is the most important part, the storage engine here,
‫you see.
‫So the storage, once we discuss this, there's the data format, how I'm returning these things to
‫the user and the user here, I really mean the application.
‫And again, the front end, front end here, I'm talking about the actual database front end.
‫It's in the database right portion.
‫And then the second portion, which is the most important part, is how am I storing the data on disk,
‫Right?
‫And the storage engine doesn't really care what you're storing in it to the storage engine.
‫You have something called a page and you in the page you have bytes.
‫That's all what it cares about.
‫And the.
‫Front.
‫And part of the database will say, Hey, by the way, in this page there is a bunch of rows, right?
‫I have a row store right where I have a table and I put the row and all the columns.
‫And then right after the final column of the first row, I put the second row and you can see that it's
‫just a if you if you think of it like an actual page, a rectangle, the first row, and then followed
‫by the second row, followed by the third row and all its column, fourth row and all its column, fifth
‫row and all column until the page fills.
‫That's why the storage engine has a property called the page size in A.B.
‫MySQL.
‫That's 16 KB kilobyte in Mongo MongoDB.
‫I don't remember MongoDB but Postgres is eight K and you can change this page size so in mongo.
‫What we're storing is just a document.
‫It's a JSON document that the front end receives and it turns it into a bunch of bytes and then we flush
‫it to a page and it's the same thing, the document or key, the first key and the value, and then
‫we just write it to the storage engine.
‫So if you think really, really about it, it's always the storage, what you're storing and what how
‫the front end is actually extracting this information.
‫So document graph, right based database.
‫When I say, Hey, this is a graph based database, the storage engine doesn't care.
‫It's just how the front end part of the database actually organizes the bytes such that when I store
‫them.
‫I want to read that page and get as much efficiency in my read as possible.
‫So another piece of the storage engine is indexes, right?
‫Because now we're storing things in a bunch of pages, right?
‫How are we storing them is also another story, right?
‫Are they are they just a bunch of files each?
‫Does each file represent a table or a collection in MongoDB?
‫Right.
‫Or am I storing the actual data in the indexes itself?
‫We're going to talk about all that.
‫Right.
‫Indexes will help fast track what you're looking for, Right?
‫That's also part of the storage engine.
‫The type of indexes you're creating.
‫Is it just is it a b-tree?
‫And all really helps pinpoint exactly what page you are trying to read.
‫Okay.
‫So if you have a like a table or a MongoDB collection, these MongoDB collections is just a bunch of
‫JSON documents, right?
‫Storage engine can decide, you know what, this document is really large.
‫I'm going to decide to compress it.
‫So that's all property of the storage engine.
‫The front end doesn't even know that this document is compressed.
‫All it does is, hey, just give me that document and this part will just decompress it and give you
‫back.
‫Give it back to the front end and the front end will return it.
‫Right?
‫So this is like a clear separation between these two and they can share tasks as well.
‫Of course, the files like where is the actual full data, you know, because indexes only have part
‫of the data, right?
‫Like I'm indexing on the first name and give me all first name is a bad index and I don't know, salary
‫maybe, right?
‫Salary is another data structure we create and then we traverse it back to get back to exactly to that
‫data.
‫Right to that data file, which then pulls the entire document or row and then we return it.
‫And you can you can really be creative here.
‫And that's what people did, right?
‫The storage engine also responsible for transactions.
‫You know, when I'm changing this and this and this and this and this, I want it to do it as one unit
‫of work such that if there is a failure, please roll back All these changes don't persist anything.
‫Half way through.
‫I want to be consistent.
‫I want to be atomic and I want to be isolated.
‫My concurrent transactions, all of these are properties of the storage engine.
‫Really.
‫Right?
‫I want durability.
‫I want that.
‫If if I say commit and you told me the front end.
‫Right.
‫That's another thing, right?
‫The transaction will say, Hey, I want you to commit and the storage engine will say, yes, you committed
‫successfully.
‫If I get the success and return to the user and then later you crashed, that data better be there when
‫I come back because you told me you committed successfully.
‫Right.
‫All of these things.
‫Well, the write ahead log or journaling, as MongoDB calls it, right?
‫As I'm writing things, usually when you write things, it goes it needs to go to the data file.
‫That's where the the major storage lies lives.
‫But but then writing to the data file is really expensive because you're writing in pages these massive
‫pages, eight K and 16 K, So right.
‫If imagine like you're touching one column, one property, you need to write a whole page.
‫There is no writing one byte in databases.
‫No, no, no, sir.
‫We don't go to disk and say, Hey, just change that tiny byte or just change that tiny three byte or
‫just change that one K byte.
‫Nope, you can do that.
‫That's not how SSDs and hard drive works.
‫You have to write in chunks and big chunks for efficiency.
‫You an IO, you're going to write an all sector, you do an SSD, you can write a whole page or a block
‫or an erasable unit based on the new technology of SSDs.
‫Right.
‫That's where the shingled hard drives comes into the picture, where they increase the right portions
‫and stuff like that.
‫We don't have byte addressability on disk, unfortunately, until today 2022.
‫We have byte addressability on RAM.
‫You can definitely write single byte and RAM.
‫Definitely.
‫That's fine.
‫Right.
‫But on disk persisted.
‫No, you gotta write on pages.
‫Right.
‫And that's what we have today.
‫And because of that cost.
‫Right writing to hold data files what the data storage engine does like as you change all these changes
‫goes to ram.
‫Bup bup bup bup bup bup bup bup bup.
‫We call them dirty pages.
‫The moment you touch a page where you have a row or collection or document, we just mark it as dirty.
‫And again, the storage engine doesn't know it's a document.
‫It just knows it's bytes.
‫It knows it's a page with a bunch of bytes that you touched.
‫Right?
‫And then you write to the memory and so it's fast.
‫And then later the storage engine will collect as much changes as possible and then flush it once right
‫to the database.
‫All of this the job of the storage engine.
‫I still didn't come to the difference between school and SQL, but we'll get to get there, right?
‫You clearly are going to see it, I think, by by this time, if you're still watching or listening,
‫you're probably going to know the difference, right?
‫So we're not writing immediately.
‫We're collecting this change, you might say, Hussein, but wait a minute.
‫You're writing to Ram.
‫If I commit, you're writing to Ram.
‫What if I crash?
‫That's the problem, right?
‫So that's why in case to to recover from the crash, we create this called thing.
‫This thing is called Wall, the write ahead log.
‫So as we write to the RAM to this data pages, we also write on disk tiny things that says, Hey, here's
‫a journal on this date.
‫I on this date write Dear Diary, On this date I updated the salary from 10,000 to 10,050 cent.
‫It's a bad year.
‫What do you want me to say?
‫Right.
‫So.
‫And this on this date?
‫I wrote this on this date.
‫You just write the changes.
‫So then in case of a crash, we're going to lose the dirty pages on the on memory.
‫But if I came back, I have all the wall and I have the last checkpoint on the data file.
‫So I restore that and I redo the changes.
‫I apply the wall.
‫To the data files.
‫And now in memory, I have the final representation as it was when I crashed.
‫Brilliant design.
‫Anyway, I'm going all over the place, but the storage engine front end, that is the main pieces.
‫So we talked about what the front end is.
‫We talked about what the storage engine is.
‫The difference between the SQL and NoSQL mainly.
‫Is this puppy the front end?
‫The NoSQL guys came in and says, Are you really restricted me with this tables and columns and the
‫sequel?
‫I hate SQL.
‫I don't like SQL at all.
‫Right?
‫I don't like it.
‫And it just just didn't fit our application.
‫I want to be like, I want just to give you a document, just store it.
‫And that's where they redesigned.
‫I think someone came in one day and shop was like, No SQL.
‫They start a movement.
‫They said No SQL, no SQL, no SQL, no more SQL.
‫And they created their own technical storage engine.
‫And they I believe, if I'm not mistaken, they didn't even have indexes.
‫So, see, you guys are so because when databases Oracle and SQL Server created, they were just so
‫wired to be, you know, to have tables and rows and so everything was glued together and sticky can't
‫change it.
‫So they created everything from scratch.
‫A storage engine.
‫I'm storing just documents, for example.
‫That's the first use case.
‫I have a document that JSON document just store it.
‫It's just a bunch of bytes.
‫Later they added transactions, later they added wall, later they have the indexes, they slowly added
‫it and then the API just took it and set.
‫The user will get a document and we're going to store it on the storage engine.
‫That's it.
‫It's just a bunch of bytes.
‫We're going to convert the JSON into basin bison binary JSON, and then we persist it.
‫That's the only difference.
‫That's the only difference in SQL.
‫NoSQL, right?
‫The data format, which we changed from tables and rows into documents, and then the API, which is
‫the get and set instead of just SQL and clear separation.
‫And then of course there are out of the box storage engines such as Leveldb or Myrocks write Rocksdb.
‫Sorry, Rocksdb is a very popular storage engine that does exactly that takes a bunch of bytes, doesn't
‫care what you have in your bytes, it doesn't care, just gives you the beauty of indexes and storage
‫engine, all this stuff, right?
‫But then in the front end you can build your database the way you want.
‫That's why you can build a graph database.
‫So graph will prioritize not rows or columns per se or even documents, but the traversability like.
‫So if this is not connected to this node to connect to this node, I want to store them next to each
‫other right in this way.
‫And so that the whole goal between the API and the storage and the front end is that when I do an I
‫O and I give me a page.
‫You want as much as possible this page to be to have everything you need.
‫You don't want to go back to read more pages and I can go for ages about this.
‫You know, this just the efficiency of the I think this is the most important thing, but we still didn't
‫get to the main part, which is the MongoDB databases.
‫So now we talked about NoSQL versus SQL.
‫What's the difference?
‫Right now, what we want to discuss is the first version ish of MongoDB.
‫Yeah, this is prior to 4.2.
‫MongoDB first storage engine was called memory map version one, which is literally just a bunch of
‫data files, Right.
‫And the data file, right.
‫Uh.
‫The data file are stored document after one, one document after the other.
‫Now, I don't know if there is one data file have per collection.
‫Maybe when you have a collection you will have a data file.
‫Maybe.
‫Maybe it's different.
‫But what?
‫The brilliant design behind the first version was an offset based.
‫That means, Hey, I want this document.
‫Document.
‫This particular document with an ID.
‫So what Mongo has is as a unique identifier, right?
‫If you know about that and this, your unique identifier will tell you exactly what about this document?
‫Right.
‫And there is an index attached to it.
‫Right?
‫And this index.
‫This is a B3 index.
‫When you traverse the B3 index, you find the IDs.
‫Okay.
‫It's in this page.
‫And then you find it.
‫The pointer of this unique identifier is something called a disk location.
‫I think it's a 32 byte.
‫It's actually 64 bit.
‫Sorry.
‫It's a 64 bit pointer.
‫32.
‫The 32 bits, the first 32 bits tells you the file name, which file and the second 32 bit tells you
‫the offset because now you know which file.
‫But then the file is, is one gig, right.
‫Where exactly is the document in this file is the offset.
‫So with one single read, you can go.
‫Exactly.
‫Because this is how the OS read write the OS will read, will give you the file name, says hey, go
‫exactly to that location.
‫You can absolutely do that in the file system.
‫Allow it to say read that portion and read for X amount of bytes.
‫Right.
‫So I suppose the another property is the file is the document size.
‫So you need to store also the document size, right?
‫So say read this part and then you're going to read that, right?
‫And then you get a bunch of pages probably.
‫And then if you're lucky, you're going to get one document or more.
‫Right?
‫That's why the document also has a fixed size.
‫You can't go beyond certain size because of these limitations, right?
‫So now you got it.
‫So you do one B3 scan from the ID, right?
‫To find exactly which document to pull.
‫Right.
‫Again, you're going to get a bunch of bytes and then the front end is responsible to pass.
‫The bytes to actually find documents per se.
‫And of course, if this was like a relational database, then going to be columns and rows, right?
‫If there was a graph, you're going to pass it such that, you know the beginning and the end.
‫Right?
‫And it's not really rocket science at the end of the day.
‫So we're getting a big O of log N, right?
‫So it's just A1IO or multiple i os to traverse the nodes.
‫That's why it's important that the b-tree is small enough to fit in memory such that because the bit
‫the index is just not a data structure which is persisted on disk.
‫You read it from disk and you put it in memory.
‫Hopefully it fits in memory.
‫That's why this scored Actually one of one problem that this car faced was they moved from MongoDB because
‫their indexes were so large they couldn't even fit in memory.
‫And if your index doesn't fit in memory, that means as you traverse right, the operating system will
‫will do these paging and swap files and will swap things to disk if it's not used right.
‫And this scanning is going to become slower just to find the disk lock.
‫But that was the original thing.
‫The problem, the clear problem with this is anything you touch, you change the document size, you
‫update it to a longer string.
‫The entire file is now scrambled.
‫Right?
‫Because the offset you change the physical offset of the disk.
‫Right.
‫I suppose you can play with games with this, but this became very, very problematic, right?
‫Because the documents are based on offset.
‫The moment you change the document size, you push it a little bit.
‫The whole offsets are now off.
‫Right?
‫That was the original design, I suppose, if I'm not mistaken.
‫And my Isam Isom in MySQL, which is no longer used because of the same reason.
‫Yeah, it's nice for read only.
‫It's beautiful.
‫All right.
‫If I not changing it.
‫Yeah, it's very fast.
‫You know exactly what it is and you pull it, but as you change it, it's just.
‫It becomes really a mess.
‫I suppose you can play tricks.
‫Of course you can update the offsets.
‫Offsets, right?
‫You can update the offsets.
‫But.
‫That was a problem.
‫And plus, another problem with the map is the locking model, right?
‫That's another thing.
‫That is a responsibility of the storage engine really locking.
‫Right.
‫How do you prevent two people from editing the same document at the same time?
‫You shouldn't really do that, right?
‫Database is No.
‫Two database will allow you to update the same.
‫Unit of work, if you will.
‫If it's a row.
‫If it's a table, If it's a collection.
‫Right.
‫In EMAP.
‫It was very strict, right?
‫Imagine this like.
‫The first version of map was even they didn't bother.
‫Imagine, because these are people who are rebuilding a database from scratch.
‫So they didn't think about all this stuff that the databases.
‫People have been doing it for years.
‫Right?
‫For decades, actually.
‫So the first problem they run into is like, oh, two people can change the same doc, different documents.
‫Oh, the offsets are all base.
‫Oh, you know what?
‫Let's just create a lock, a global lock.
‫So the first version was a global lock per database.
‫So No.
‫Two.
‫Transactions can actually change documents in different collections at all.
‫So if you have collection one collection two, you can even change collection one and collection two
‫documents.
‫Concurrently, they are serialized.
‫There is one global lock.
‫Again, that was the first first version because it's a single database lock.
‫So you say read this data files.
‫This tells me that the data files are actually.
‫Collapse.
‫So multiple data files, I mean, multiple collections can live in the same data files.
‫That's one reason you have to acquire a lock so that No.
‫Two, No.
‫Two, transaction can change it.
‫But then they improve this in three.
‫Three in the version three that wasn't the version.
‫Two of Mongo and Mongo two, They made it a collection level lock, which is still not good.
‫Right.
‫It's it's for the, for, for the SQL people.
‫It's like saying a table lock.
‫Imagine you have a table of a million row and you want to insert a row in the table and then you want
‫to update another row in the in the same table has nothing to do with each other.
‫Right.
‫Imagine these are blocked.
‫Yes, it was blocked.
‫And if you're using a map, this is still the case.
‫One collection, which is deprecated, by the way, V1 is deprecated now.
‫One collection is a is a pair collection lock.
‫So now sure, you can do a concurrent write on two different collections, right?
‫Without blocking.
‫But now if you're updating the same document, that's a problem, right?
‫So then it became very challenging to manage the storage engine.
‫So what MongoDB did is, as you know what, let's just acquire this Wiredtiger storage engine, very,
‫very popular, very efficient storage engine.
‫So what they did is they.
‫MongoDB just described this and they built a storage engine out of the box.
‫This has become what we call a wiredtiger wiredtiger write.
‫And the front end didn't change.
‫So your application code doesn't change.
‫The storage engine in the back end changed, right?
‫So now this is Wiredtiger.
‫They gave Wiredtiger the ability.
‫Now, here's the thing with white Tiger, the ability of document level locking has become popular.
‫Now you can update two documents on the same collection.
‫I'm not saying these things.
‫And you might say this is all exist.
‫I know.
‫But I'm telling you the history of things because building databases is not really a trivial thing.
‫The the brilliant engineers went through this and they are you know, they ran into a lot of challenges
‫and this is one of them.
‫So the Wiredtiger storage engine allowed you to update two different documents on the same collection
‫concurrently, which is now a beautiful thing.
‫Right?
‫Now we can and this is now made it equivalent to basically all databases because the databases have
‫row level locks, like at least MySQL and Postgres.
‫You cannot you can definitely update two rows on the same table, but you cannot update the same row
‫on the same table.
‫Right.
‫We acquire a lock and then the second transaction tries to update the same row.
‫That will basically pause the second transaction with row level locking.
‫Now there is like I think what's the database called?
‫Yoga?
‫Yoga.
‫DB If I'm not mistaken.
‫They even introduced column level locking, which is another thing, like if I if I have a row and I'm
‫updating field one in that row, but you're updating field two.
‫Technically, I'm not.
‫We're not changing the same thing.
‫Postgres will lock you even if you're updating different thing MySQL, if I'm not mistaken, they will
‫also lock it because it's a row level lock.
‫But now.
‫You can also include column level locking which says, hey, if you.
‫Yeah, you touched this row but different fields from this row.
‫Same thing with the document.
‫Right?
‫I am really just updating this field in the document and JSON document and someone is inserting a new
‫field or updating another or re locking.
‫Do we really need to lock it?
‫Well, at the end of the day, this is what we do.
‫We lock it.
‫So yeah, if you have if you happen to have to transaction updating the same row, even different columns,
‫you can't do that unless you have column level locking or key level locking, if you will, in non which
‫I don't think it exists.
‫And believe me when you when you when I'm talking about these things, this is not cheap.
‫Right.
‫The moment you introduce column level locking, that's another expense because now you have to keep
‫track of what you're locking and locks are.
‫Guess what?
‫In memory and row locks are more expensive than page locks or table locks or collection locks because
‫you just need one versus if you have million and you updated a million rows and transactions are in
‫progress.
‫That's a million lock, right?
‫Imagine adding column locks to that.
‫So million times, whatever columns you're updating, it becomes really challenges, right?
‫Yeah.
‫Database building database is not trivial.
‫All right.
‫Go back to Wiredtiger.
‫We talked about that, right?
‫Mongo Wiredtiger introduced compression, which didn't exist, by the way, in MVP, right?
‫It didn't exist here.
‫Wiredtiger introduced compression.
‫Now, when you actually take the document, Wiredtiger compresses the json document, so that's really
‫brilliant.
‫Now you're because especially JSON documents have these fields repeated all the time, right?
‫The field repeats, so you need to compress it.
‫So MongoDB Wiredtiger actually compresses that.
‫So that's tiny.
‫Why is it tiny?
‫Because now if I'm compressing it, the page will fit more document 1IO will give me more documents.
‫Then it was A1IO in uncompressed.
‫If 1IO uncompressed give me three documents.
‫1IO compressed in a single page will give me 20 documents.
‫This is really powerful because now I don't really need to go if I'm fetching 20 documents in in, in
‫the older models, I have to do multiple iOS.
‫I have to hit the disk multiple times versus in the Wiredtiger Tiger just one pulled all this stuff
‫compressed.
‫Do a little decompression in the client side in memory and you get a beautiful 20 documents.
‫The major thing you have to think about here, how do I save iOS?
‫That is the number one job of a DBA, of a developer of a database saving iOS.
‫The list, the IO, the faster the database, nothing else matter.
‫That is exactly what it is.
‫All right, so now what?
‫The way Wiredtiger stored the database is completely changed.
‫It's no longer using this disk lock thing, right.
‫Where it's just a bunch of data file, and then you have offset because offsets are really terrible,
‫right, for changing like the offset changes and you have to update everything like one changing one
‫documents will will screw up all your offsets.
‫Right?
‫So what they did instead, they stored it as a cluster b-tree index.
‫And I talked about this in another video.
‫I'm not going to go in details, but in a in a nutshell, right, They have something called the record
‫ID here.
‫And you can basically create anything.
‫This is a hidden index cluster index into wiredtiger and.
‫Work based on the key you can search.
‫And when you get here, the value is actually the entire document.
‫And not only a document, right.
‫But physically all the documents.
‫Right or ordered next to each other.
‫So the page that you land on here in the leaf pages are the data is the data.
‫This is the data.
‫The entire data is the index.
‫That's what a clustered index is.
‫It's it's by default what you get for one MySQL, not in Postgres, but in MySQL.
‫Everything is a cluster index.
‫Every table has a cluster index and that's how your data is organized around the index.
‫So your table is organized around this index where the leaf pages is the data.
‫So now if you land here, you get the document and guess what?
‫Because it's on the one page, you get any document before it and you get any documents after it.
‫And because it's compressed, you're going to get a lot of tight documents as well.
‫So you you read this page and you get all the documents nearby because it's ordered.
‫Not only that.
‫Each leaf page in B+ tree is actually linked to the next page and to the next page and to the next page.
‫It's a linked list of pages, so the entire data is right here.
‫So if you find this, if you want to do a range query and say, find me all all record IDs between X
‫and Y and we're going to talk about what record is because this is not the ID of the document.
‫And that's the problem that Wiredtiger and Mongo introduced in a way.
‫Now, if we have this, you do a if you do a range scan, you're going to get all the documents that
‫are next to each other.
‫So a range scan is really powerful in B plus three, especially if it's clustered, because now you're
‫going to get all the nice documents tucked in together, right?
‫So you can find your.
‫Document using a B+ tree search in Wiredtiger using the required.
‫But guess what?
‫What is this record ID?
‫It doesn't mean anything to the user.
‫This is an internal thing.
‫But where did this disk lock happen?
‫This used to be called the disk lock, but they changed it.
‫And that's what they had.
‫They had this as disk lock and their indexes, the ID, the actual user facing ID document index has
‫been mapped always to the disk lock because that's what we had, Right?
‫Disk lock.
‫That's that's exactly this.
‫This.
‫Is this.
‫This used to be this this look they later changed it to recorded.
‫So it's like it doesn't make sense to call it this log.
‫But then this record ID now is just a pointer to where not to disk.
‫It is a pointer to this B plus three, which is the hidden index.
‫So now if you're actually searching for the ID, the primary key, you're doing two lock ups in Wiredtiger,
‫not one.
‫So actually I'd look up in Wiredtiger were slower than the older one because now you have to you have
‫to search two indexes, you have to load two indexes in memory, double the space, double the searches,
‫double the IO.
‫You have to write, you have to write to multiple indexes because you have to sync those two guys together.
‫Secondary index is not so much because the secondary indexes, right, If you think about it, really
‫secondary indexes.
‫Secondary indexes.
‫Now just point directly to the record ID, So yeah, in this particular case, all of these indexes
‫always point to the record ID, whether it's a primary index or a secondary index, they all point to
‫the primary key and that's what's causing us the double search effectively, right?
‫So very similar to MySQL.
‫Not quite because MySQL primary key is actually this thing, right?
‫But the primary key in the first version, at least from 4.2, 4.2 to 5.2 very recent.
‫This change, by the way.
‫Right.
‫Until very recently, 5.24.225.2 is like this.
‫When you search for ID, you find this and then you do another search, another search.
‫This is not a big o of one, right?
‫This is a big O of log n plus big O of log n two searches.
‫Right.
‫Whereas this guy you do big O of log n and then big O of one.
‫So now we have this beautiful design.
‫The problems we understood now the IDs, the problem we have, we have to we have to kind of duplicate
‫style, right?
‫The record is a 64 bit same thing here, but secondary indexes all point to the record already.
‫That's the state of art as of 5.2, right.
‫And the ID index is just another secondary index at this point.
‫It's not really a true primary index because the primary index, by definition at least, is the clustered
‫index.
‫Right?
‫It is this one.
‫But we have now doubled.
‫Now let's go to the final stage where 5.3, I think is July of 2022.
‫Really very, very brand new feature.
‫It's called clustered collections, where you can create a collection and you can make it a clustered
‫collection.
‫That means the wire tiger hidden key disappears and instead this becomes your hidden index.
‫Effectively, this becomes your clustered index.
‫And the ID field is the main focus for this.
‫Now, if you're searching by ID right, you will immediately search by ID, do a little bit lookup and
‫then find the document because the cluster document is right here.
‫All the leaf pages have were full documents right here.
‫Pretty neat.
‫You don't really need to do these two lookups anymore if you're searching for ID, right?
‫Again, this is not this is an option.
‫It's not you don't have to do it right.
‫So if you still want this design, for some reason, we're going to talk about why in a minute.
‫You can still have it.
‫But now in this guy.
‫You can do this.
‫What's the problem with this now?
‫We talked about the good thing.
‫The good thing.
‫Single search.
‫If you're using the ID for MongoDB.
‫Right.
‫If you're looking up a document by its ID, it's a single, beautiful search.
‫Immediately find the document based in document.
‫Right.
‫And you're going to if you're lucky, you're going to find anything in next to it.
‫Right.
‫It's not just one document.
‫This is a collection of documents in a single page.
‫I got a I got to find out what's the page size and why Tiger.
‫Right.
‫But this is what you get.
‫You're going to get this and it's going to be cached in memory temporarily.
‫Right?
‫So if you're lucky, the next the previous ID next to it is also you're going to get that as well,
‫right, If the sequence really makes sense here.
‫The problem, though.
‫The problem, my friends, is now let's go back to the secondary indexes.
‫The moment you introduce it, now this becomes identical to MySQL.
‫MongoDB after 5.3, if you choose to be a cluster connection, it's almost identical to MySQL now.
‫It became identical to MySQL.
‫The ID field, which is the primary key, is the cluster index.
‫The secondary indexes point to what now?
‫They have to point to the ID, right?
‫There is no record that you moved where the data is.
‫Right.
‫Previously the second index.
‫I should have drawn this, but sorry, I did not.
‫The second index is used to point to this thing, The Hidden, which is a very tiny value recorded 64
‫bit.
‫That's it.
‫You know, how large is the field?
‫And did I did I actually mention that it's called the object ID?
‫I actually mentioned that someone highlighted it.
‫12 bytes.
‫Dude, this is bytes, not bits.
‫This thing is a 12 bytes by default and it has like the first four bytes is the timestamp.
‫The second three bytes is.
‫I don't know what this is because Mongo decided to scale first, right?
‫So they wanted their IDs to be unique across machines.
‫So even the second four bytes is a combination between the process ID and the machine name.
‫And so the idea is truly universally identified across machines.
‫So that's why it's so big.
‫12 bytes is so large.
‫Yeah.
‫So it's 12 bytes compared to eight bytes.
‫Right.
‫Because 64 bit is, is eight bytes and 12 bytes is 12 bytes.
‫Right.
‫So four bytes extra you might say.
‫I'll say who cares.
‫Four bytes extra.
‫But here's the thing.
‫Here's the thing.
‫I didn't know MongoDB actually allow you in.
‫Those who who use MongoDB more might know.
‫You can actually set anything in the ID field.
‫So it's a user controlled field.
‫If you don't set an ID, it's going to generate one for you.
‫But if you do set it, you can have it to be a very large people can have crazy ID values and guess
‫what?
‫The secondary index is now has to point to the ID because that's where the data is.
‫And that's where all the problems of MySQL arise, where if the ID is a poorly chosen value, if the
‫primary key is a poorly chosen like a good right, again, there's a lot of, of course discussion about
‫having a UUID as a primary key, but we know it's very large.
‫If you use it as a as a primary key, then those primary keys are stored in the secondary indexes as
‫values and that's what blows everything up.
‫So now the secondary indexes just blow up.
‫Right.
‫And that's the basically the evolution of MongoDB, you guys, right.
‫As a summary.
‫We started with a map.
‫Right.
‫Move to Wiredtiger.
‫Gain a little bit of new features, but introduce the new problems for sync 5.3 and six zero.
‫You can actually do clustered indexes.
‫You're going to see in the next one.
‫You hope you enjoyed this video.
‫Goodbye.