‫What is going on, guys?
‫My name is Hossein, and this video I want to discuss, I caution that some of you asked me when do
‫I shahade my database?
‫And short answer for those who are impatient.
‫Last thing you want to do is the Shard, your database, there are so many other things that you can
‫do before complicating your life with.
‫So with that said, let's just jump into it, guys.
‫Sharding the idea of starting a database is I know it sounds that the that term sounds very sexy.
‫Arrives as I.
‫Oh, I'm sure magnetic resonance makes you sound intelligent, but if you think about what you're doing
‫with Sharding, it is a very complex architecture and complex operation to implement.
‫So.
‫In order to talk about Shardey, as you have so many other things you can optimises, like why are you
‫doing it to begin with?
‫Right.
‫So.
‫Start over, you have a beautiful database, and I'm now talking about mostly relational databases,
‫because that's my specialty.
‫I very rarely work with no more SQL databases, but they work similarly.
‫But here's the thing.
‫And in relational databases, you have a single instance database, right, and you're probably building
‫a Web application, so you need a Web server or an API framework such as Narges, Django, all that
‫jazz go application, your custom application that essentially speaks HTTP protocol that talks to the
‫browsers or some rest clients or.
‫Any job, easy job, easy, but what is the thank you.
‫Thank you.
‫Yeah.
‫So if you're anything that's basically SDB and you make requests and that request goes to the Web server
‫and the Web server makes queries to the database and response gets her back responses and kind of changed
‫the representation to something that the client understand could be JSON, XML, anything, and then
‫ship it back to the client.
‫That's their best representation.
‫Right.
‫Change the representation.
‫All right.
‫So that looks good.
‫What's the problem with this?
‫You get a lot of requests.
‫A lot of people are reading.
‫I say I was like, oh, I'm getting give me all these employees table or give me I'm reading this profile.
‫I'm reading this for I'm reading this tweet.
‫I'm reading this thing.
‫So you're hitting that database a lot and you're reading a lot.
‫Is it OK that you're reading from a table right at that table is large.
‫It's getting large because you're inserting a lot of stuff in it has one million, two million, three
‫million, four million twenty fourteen hundred million rows in it.
‫And you might say Hosain doesn't matter because I have an index on it and that index is going to be
‫smaller.
‫Right.
‫And I'm going to head to index because I'm smart.
‫I am going to index the fields and I'm only creating based on those fields that are indexed.
‫So my queries are fast, but then I can get you only so far because the larger the table gets, the
‫larger the index gets and almost you're doing and full index can, which can also be slow.
‫So the solution is not to just split everything into multiple servers.
‫You're doing it, you're overengineering too early.
‫And that's what the problem with most people are just overengineering too early.
‫You can still do so much stuff.
‫I hope you have this this is there a problem now, right?
‫You have to identify with the problem first in order to solve it.
‫You cannot just solve something that you don't know.
‫The problem is right.
‫Identify the problem.
‫So let's say this is my problem.
‫My reads are slow.
‫I didn't say anything about right.
‫Saslaw just reads because writing is always fast.
‫You're in writing to the end of the table and even if you're using a B3 data structure index.
‫Yeah.
‫If you're using and it's this D that could be.
‫Harmful for this is because you're rebalancing that tree sometimes if you're riding a lot, and that
‫could cause some shifts in the desk at Stulz.
‫Now, that is bad as Reed and LSM is optimized for Reed, but.
‫Right.
‫But that's another story.
‫Right.
‫So I'm reading a lot.
‫Meile is getting slower and slower as long as it used to take one milliseconds now is like 20 and now
‫it's 100 milliseconds.
‫What do you do?
‫Tables large, what do you do with it?
‫There's something called most database's supports called partitioning.
‫And I'm going to talk about just the horizontal partition.
‫So imagine this.
‫This is your table and I talk about horizontal and vertical partition.
‫I'm going to reference the video here.
‫Go check it out.
‫But in a nutshell, you have a huge table, hundreds of millions of rows.
‫You just slice it in half horizontally.
‫That's that's the horizontally slice it in the middle.
‫And you say this piece from this.
‫I.D. to this ID go to this stable, right, and we're going to call this a partition and this idea to
‫this idea goes to this table I'm going to go partition to and this this idea, this idea got to partition.
‫So now you instead of having one table, you have three different tables that represent the whole logical
‫table.
‫But when you query, we can find out based on something called a partition key, which usually is that
‫as the primary key of the table doesn't have to be there.
‫But based on that, you can know which table to hit.
‫Now you have smaller tables, you have 10 smaller tables.
‫And guess what?
‫With a little better knowledge from the client, we can know exactly which small table or partition
‫to hit.
‫And now I'm working with a smaller index and my queries will be fast instead of having this huge index.
‫Now I have a smaller index and I know exactly where.
‫Quite so.
‫My queries are fast, just like that.
‫By just enabling partitioning and most databases.
‫Postgres, for example, does that blindly for you if you enable it.
‫And the client doesn't have the even need the knowledge to know that, oh, I need to have this table
‫table partition.
‫Underscore what?
‫No, it's just happened for you previously and all the releases you had to have this knowledge.
‫Oh, this is the stable in the stable mistake.
‫Remember, guys, I'm still in the same server as one instance, one database instance, but with little
‫bit of a tweak of a feature of my database, I, I fix the problem.
‫Right, you don't need charting when I talk about sharding in a minute.
‫What does it really mean?
‫But.
‫You come back, you say, Hussein, no, that's not good, because, yes, I am, my singular reads
‫are fast selecting staff from the stable and yet very fast now because I partition all that stuff.
‫But my dad, a base server, cannot handle because all these TCP connection to a single server just
‫read stuff.
‫It's overwhelming my server and it's the file descriptors are blowing out of proportion and I cannot
‫handle that anymore.
‫Right.
‫So what do you do?
‫Well, I'm not even introducing the idea of caching.
‫You can obviously introduce a layer of readiest cache and all that stuff, but.
‫It comes with its own problems because you'll end up with inconsistency and you have to invalidate the
‫cash and all that stuff, but you can't play with that.
‫And without introducing that extra layer, I have some tricks up my sleeve to fix that problem.
‫Right.
‫And most of the stuff you guys know, it probably but application, why is nobody talking about this?
‫You have a master backup.
‫So and you some some databases have this by default.
‫This this features where you have one master and create another server and make this server a backup
‫of that master and what it will do as periodically the master will push the edits.
‫Right.
‫The rights to that.
‫To the replica, to the back up, so you can create as many backup as you want.
‫And multiple servers with partitions.
‫And.
‫Do all that stuff now, you segregated multiple servers, but you have one right to do a ride, you
‫go to that dang server, you don't want to write in multiple servers.
‫That will confuse a lot of things.
‫And we're going to talk about it in a minute because reading is easier than writing.
‫Reading is doesn't really cause conflicts or writing.
‫If I'm writing to the same rule you.
‫Yeah.
‫In the seven different servers.
‫Bad things happen.
‫How do I reconcile these changes.
‫Right.
‫It just you can't solve it, but it becomes a very challenging problem to solve.
‫So you want to avoid complex problems and why would you do it?
‫Well, I'm gonna write to myself because my rights are not as bad.
‫I don't I often barely write anything, but I read a lot.
‫So you can just multiple servers, you have one master server and just push all that stuff to that.
‫Replicas of the backup's and now your clients or the observers, but a reverse proxy, actually a proxy
‫in Haertsch, anything you want and then start FUNNYE or you want to read, go to this database or,
‫you know, read you get as the backup goes back up and now you're balancing the rights, but the rights
‫still goes to this database that is the master and some people who will come says or know who.
‫And you don't know me because I write a lot and I need to write a lot.
‫And my single server that does the right no longer is sufficient is going to die because I write so
‫much.
‫Do it.
‫Really doubt it first, because we're writing a lot, you will be writing a lot to saturate a single
‫server.
‫I'm assuming you have it big server that does that stuff right.
‫But let's say you do your writing a lot.
‫You still have some tracks before the final thing, which is Shati, you can do multiple replication,
‫a two way replication, but have two masters on different regions in in in the US.
‫This is very common.
‫This is a U.S. E and this is a U.S. W.
‫People from Florida writes to this server anything that has to do, customers with the with the E just
‫deal with this, anything it has to do with custom of the West, deal with this and most of the time
‫they don't overlap.
‫Right.
‫Really.
‫So if you have this segregation, you use it to your advantage and then synchronization of this can
‫happen very easily because you're barely going to have any conflict.
‫And if you do, you're going to resolve it.
‫And if this is so complex, you finally go and do sharding, which is the following sharding is the
‫idea of having a single key C after all of these are exhausted.
‫Right, come down and Lusardi, right, if and if all of this stuff is also done, goodo sharding,
‫if none of them, then probably you don't have to do shonen because it's shorting is the most complicating
‫thing to accomplish.
‫And let's talk about what it's become easier these days, but let's talk about Sharding.
‫So what is shocking?
‫Sharding the idea of splitting like partition, like horizontal partitioning, but pull the tables that
‫are partitioned, not in a single in a different table.
‫In the same instance, same server.
‫No, both those partitions or shards in a different server altogether.
‫Wuk.
‫Yeah.
‫With partitioning, you can still do transaction, by the way, guys, the database supports transactions.
‫Well, short thing for for what Sharding was sharding my, my, my tongue has been swallowed.
‫Will sharding.
‫You can all do transactions.
‫Forget about transactions.
‫Bye bye.
‫Roll back all this beautiful comet stuff, as I said, at Tomasetti.
‫Bye bye.
‫No sharding for you.
‫You cannot do all this because what do you do if you want to try to multiple shots?
‫This is different servers, right.
‫And you cannot maintain the asset transactions.
‫I know some people don't like as a transaction and the favor the others with eventual consistency.
‫But the old school kind of a guy.
‫All right.
‫So so what do you do is I am about to update customer number hundred and does the phone number 100,
‫you say, OK, the client, you are the client, which is the world.
‫So observer in this case is OK.
‫Customer number one is between hundred, between one and ten thousand.
‫So yeah, as server number one, let me establish the ECB connection with our number one and execute
‫that right or the right.
‫Right.
‫And and you received another request.
‫But this request has to do with customer number ten thousand and seventeen or seventeen is between 10000
‫and 20000.
‫And based on my sharding, arrange this a server to subject this connection to the server to and executed
‫and come back and result.
‫And yep, you might say, yeah, I'm going to use Pawling because I don't want to establish this connection
‫every time Endris and I, I absolutely agree with you have to use pooling.
‫You don't have to do this cold start of chickpeas.
‫Hopefully one database platform will get to use quick and all this problems will go away and they could
‫start and all that stuff, but.
‫Yeah.
‫Sharding is very complex, that logic that we just talked about is in your client application.
‫That's hard, right?
‫It's very hard because I made a video about Sharding guys and actually show you how it's done.
‫Check it out here.
‫But what do you do, guys?
‫Right.
‫This is do you do you really want to do this with your life and sacrifice rollbacks and comments and
‫transactions and all this stuff just to do a stinking sharding?
‫Just to do what?
‫Scaling.
‫You want to scale the rights?
‫I agree it's beneficial, but sometimes it's an overkill.
‫You get a really neat thing.
‫Do you really need to, Shahd?
‫Right.
‫All right.
‫That being said, YouTube ran into this problem, as you might as you might think, because YouTube
‫is a.
‫Millions, millions of videos which would billions of watch hours and hours uploaded, right, so thousands
‫and thousands of hours are uploaded to this platform and comments and and I watched listen to a podcast
‫with the lead of YouTube talking about how they solved it.
‫They have a MySQL database on the back.
‫And by the way, they manage all that stuff.
‫And the videos are, I believe in I forgot where it is.
‫Maybe it's three.
‫I might be wrong there.
‫Before that was before Google acquired it and all.
‫That's not right.
‫So they moved everything to YouTube, to Google infrastucture and all that stuff.
‫But they for a long time they had a single MySQL server and they managed to do fine until the rights
‫became absolutely disastrous.
‫So they started doing what we talked about and until they couldn't handle.
‫So they had to do Chardy, which is called.
‫What we talk about is application level.
‫Schoening, the application is aware of the Shahd, which is this piece of the partition, OK?
‫And when you know, when you are aware that's a problem because you create a coupling and coupling is
‫the worst thing you want in software because you want software to be as isolated as possible.
‫You want software to have social anxiety, social distancing in these days through social distancing
‫for software like Gabrielsson software apart, they shouldn't know stuff about each other.
‫That's just a general idea about separate.
‫Coupling is bad about software.
‫Come back now, what they did, it's like they had this application level story for a long time and
‫says, look, this is ridiculous.
‫We cannot maintain this because if they want to change some logic and everything breaks, right, they
‫want to forget about regarding the idea of rationing is like, oh, okay, I want to change the range
‫from ten thousand because ten thousand is really low.
‫Let's make it hundred thousand.
‫Right.
‫Each card has 100000 rolls.
‫Oh, forget about that.
‫Nobody can do that because I will break every single client.
‫You've got to go to that client and yeah.
‫So it's complicated.
‫So they created this technology called Vitez and I'm going to reference it here.
‫And the test is essentially a middleware that you send your query on top of my school and it's I believe
‫it's open source and it's Cuban and Israelis and all that jazz.
‫And you put it on your MySQL architecture and is going to do the magic sharding for you.
‫And that is awesome, because you send a beautiful school query and they pass that query and they understand
‫what charge to hit.
‫And you don't need anything better than that if you absolutely need, because nothing's for free.
‫If you want if you have the maintenance cost of implementing the test, go ahead and do it.
‫Any technology you implement there is a cost for it.
‫If you're ready, go ahead.
‫But ask yourself, do you really need it?
‫Sometimes you do not need it.
‫I was an awful British accent impression.
‫All right, guys, that's it for me today.
‫We're all learning we're going to learn a lot and keep learning, guys.
‫I see in the next one by.