‫What is going on, guys?
‫My name is Hussein, and this is the third episode of Suffragette.
‫Today's topic is distributor transaction.
‫Some of you ask me why this what is this thing distributed?
‫And to be honest, when I heard that when I heard that question, what is distributing a distribution
‫transaction?
‫What is the saga thing?
‫And I didn't know anything about it.
‫I read some articles talking about distribution transaction with within the context of Google Atomic
‫Clock, and I know I exist and this is then, but I didn't know why.
‫In the vernacular, why does this technology exist and why do we need distribution transaction?
‫What other solutions?
‫So I spent some time doing some research.
‫I'm going to reference all the videos I wash all the articles I read to kind of summarize this for you.
‫So and this video, I'm going to talk about distribution transactions.
‫Why does it exist?
‫Because I like all of us to start that any technology in software engineering exist for a reason.
‫And if it exists, it solves the problem.
‫What problems does solve?
‫Right.
‫And maybe.
‫That problem could happen to certain sector of engineers, not everybody.
‫That's why I'd like to address this problem.
‫Like, do you really need distributer transaction?
‫Well, what's your problem?
‫Like, that's that's why I'd like to do it.
‫And then I'm going to talk about the different solutions to distribution transactions, because there's
‫that's the beauty of social engineering.
‫There is no one way to solve a problem.
‫There are many, many, many ways.
‫That's what makes us really, really a creative endeavor field.
‫If you're a software engineer, you're and you come up with solutions, you're a create.
‫You're like an artist, man.
‫Just, yeah, you're an artist because coming up with creative solutions to solve our engineering problems,
‫it's it's an art, whether you like it or not.
‫So I respect anyone whether the solution is bad or good.
‫It's an art by itself.
‫You don't look at a painting and say, oh, that's that's not that's that's not the right painting or
‫that's a wrong painting.
‫You know, you can say that, right?
‫It's just it's an expression.
‫And I like it.
‫That being said, let's just jump into it.
‫All right.
‫So why does distribution distributor transaction exist?
‫Right.
‫So what is this thing?
‫Before we jump to distribute the transaction, let's briefly take some time to talk about transactions
‫in general.
‫I made a video about transactions and acid and all that stuff.
‫I'm going to freeze it right here.
‫Go check it out.
‫I really recommend you understand these basics.
‫I think every software engineer need to understand what acid is a tomasetti, consistency, isolation,
‫durability.
‫So that's why I made a full course, almost video one hour talking about these things.
‫It's at a very critical topic to discuss as a versing.
‫Right.
‫A transaction is.
‫And I'm going to take the example that I'm seeing all the time is this order payment kind of thing.
‫Let's say you're you're building Amazon or building some application that does ordering and payment,
‫OK?
‫And it's a Web application.
‫So we have one Web server or WAP framework, maybe a Norges application or a B application that does
‫the whole thing for you.
‫Right.
‫You submit a request, arrest maybe or or graft you up and you say, hey, I am I want to I have these
‫in my cart.
‫I want to buy the stuff.
‫Right.
‫So what is by this stuff mean?
‫It means that submit an order, charge my credit card and update the order and email me with the order
‫details and when is going to be shipped and all that jazz.
‫All right.
‫How do we do it today?
‫Very simple.
‫Very elegant.
‫We figured it out back in the 70s.
‫How to do this transactions right.
‫In an atomic manner.
‫That means all of these updates have to happen at the same time or none of them happen otherwise.
‫We lead into these inconsistent state, which aren't that OK?
‫Cotton gone bad, not bad for overapplication, but bad in this case.
‫All right, so how do we build it?
‫Well.
‫I am going to receive the request, this is something I want to buy these items, so I'm going to insert
‫these items and I'm going to create an order.
‫I'm going to create a bunch of items.
‫Right.
‫And insert these items and do the orders or there's a row.
‫And there was another maybe another table or other items, whatever.
‫And then item one item to item three.
‫And I'm going to update the status of them of the order saying, oh, this is initialized now and then
‫I'm going to go ahead and charge my the payment method.
‫You're using MasterCard.
‫OK, charge it.
‫If I succeed, I am going to move in and maybe estimate the delivery, say, OK, it's going to take
‫a week to deliver and then I'm going to update the order status to be paid because I another update
‫statements or the first insert and insert, maybe insert the payment method because I submitted it,
‫I just inserted it, not committed yet.
‫And then I update the order and then finally because the payment, all of these statements are now true
‫and correct, I'm going to commit.
‫Right.
‫And this is a whole bunch of atomic operations now.
‫This is one atomic operation, one transaction, OK, work, one unit of work.
‫Well, and that's what happened.
‫If the payments failed, well, the payments failed.
‫I want to roll back the whole thing, like the order didn't exist.
‫And just emailed that he was the hey failed.
‫I'm not going to even submit the order because your payment didn't charge a while.
‫Why would I create an order?
‫Just useless.
‫Right?
‫So that's that's a technical operation.
‫Right.
‫And you can do all of that stuff because all of this.
‫Work or methods or this is the same process that is running, it's either a Web server running or maybe
‫even a veteran, a client server back when he was a client server application, you said you have one
‫process running and it's doing this thing.
‫And the TCP connection between you and the database is one TCP connection dedicated for this transaction.
‫Easy peasy, right?
‫We out how to do that, meet micro services, micro services came in and says, no, these big applications
‫that does too many things are bad because they don't not scale.
‫Because we cannot we can't we don't have the luxury of picking up my own languages as team.
‫All the teams are forced to use the same programming language.
‫And these hip kids these days like to use JavaScript and Rust and what other languages these days go.
‫Yeah, everybody used to live in new languages.
‫Right?
‫So no.
‫So meet Microsoft says let's break things up.
‫We have problems with that.
‫Obviously, I discussed most of these problems.
‫The side Microsoft vs. whites good, whites bad.
‫But regardless, it's there it's here to stay microdevices.
‫It's painful, but it gives you some advantages.
‫So let's discuss what problems arise with Microsoft as will come back to this.
‫My ordering courting Ramezani kind of like system.
‫Well, the order code should live on its own server as well.
‫That's been up and OJEA servers and the spin up by order service.
‫And this is just for creating orders.
‫And the payment service is a completely different team, right?
‫Yeah.
‫Maybe you outsource to, I don't know, square or stripe or something like that.
‫Right.
‫So that's another service itself.
‫And then which is going to use for I you didn't see the benefits of this and then I don't know, the
‫the shipment is another service and and you can see now these are now not one process.
‫They are multiple processes.
‫Stop the order service.
‫Even if all of them talk to the same, that you still have the same problem.
‫But most of them now with micro services have their own flavor of a database.
‫So the order could use Postgres and and and the the payment could use Mongo DB and the shipment could
‫use.
‫OK, Sandra, who cares.
‫Right.
‫And good luck making one transaction between all these three databases.
‫It's impossible.
‫Right.
‫So yeah, even if it's one database, it's impossible to make a transaction, a single transaction that
‫you can roll back if something failed.
‫So now let's run through our scenario.
‫I supplement a beautiful order.
‫I go to the order service.
‫Right.
‫Assuming there is some sort of an ingress or reverse proxy that receives that thing and funnel and fan
‫out request accordingly.
‫That depends how it's done.
‫There are many ways to do it.
‫But I go ahead and submit that order so that orders say, OK, order created succeeded.
‫Go back now, submit the payment.
‫I'm the reverse proxy by the way.
‫Order succeed.
‫Go do the payment.
‫Succeed.
‫Good.
‫Come back.
‫Do that.
‫Go ahead and do the shipment succeed.
‫Email the customer.
‫So that's that's three transactions, right.
‫It's not one transaction but logically in the vascular it is one transaction to the user who gives the
‫user doesn't know what you are using and doesn't know.
‫He doesn't know or she doesn't know that you're using a transactional microdevices back end.
‫Who cares.
‫Right.
‫So you're doing that.
‫And what happened is the payment now fail.
‫All payments fail, rollback back.
‫Tough luck.
‫You can either roll back because that was a commit that order actually committed to the database.
‫Right.
‫And now you have to tell, oh, by the way, the payments failed, so, hey, order, can you just.
‫Rollback does not rollback.
‫So you have to do these ugly things that's compensating it, its it's like, oh, order, delete the
‫order, actually do a physical delete to go ahead and delete the order, which is kind of expensive
‫because now you have rows in the database that you're doing.
‫You're making the database do multiple work.
‫Obviously it's not so much, but still it's work.
‫Right.
‫And if your database actually having it's in a cemetery.
‫So all of these actually whereas entries that in the in your database so you will see them in your SSD
‫drive or wherever.
‫So it's expensive nevertheless.
‫Right.
‫Go back and do all the same.
‫So that's the problem.
‫That's why.
‫Distributed transactions was invented, my god, 11 minutes to just get to the point, right?
‫That's why I cannot do this stuff.
‫Individual stories, guys.
‫It's very hard.
‫And you guys told me that you want this content pulled from the YouTube story and actually made into
‫a video.
‫That's why I'm doing it.
‫Me know if this is a good idea, if you like it or not, if the content is boring or you are trying
‫to make it as entertaining as possible.
‫Software engineering is not attractive for everybody, but that's why we have this transaction.
‫Now let's jump into the solutions.
‫What can we do to fix daing this thing?
‫What what problems can we sell?
‫Well, we talked about one of them.
‫Let's talk about the best one, which is Google have invented.
‫Right.
‫Which is called atomic clocks.
‫Right.
‫Google decided that all of their database using they have a database called Spanner.
‫And all of these databases, have they they they have the same exact time.
‫OK, and if you make a transaction.
‫They know exactly what order of these transaction that happened, so they know because the time is so
‫accurate, they nailed the time to the nearly microsecond maybe I don't know the details about that.
‫And they were using atomic clock to actually synchronize all the databases so that you can actually
‫physically do a transaction and roll it back because you know the steps, right?
‫So they figured it out.
‫I don't know the details about that, but that's how they do distributed transactions there.
‫Another method, obviously not everybody has Google and nobody for the atomic clocks.
‫So what the other solution is, do this compensating edits.
‫Right, which is we talked about it created order, do a payment of the famous failed, come back and
‫go actually delete the order.
‫So just for each operation that you do, reverse it, reverse it.
‫You have to try to reverse a lot of work for you.
‫Very complicated logic to write, but it could happen, but very prone to bugs, I think.
‫So that's one way to do it the other way.
‫Instead of just doing this, having one process, do this stuff for you, which is also prone to failure,
‫because if this process dies, you lost your you you will be in an inconsistent state, which is the
‫second one, which I like a lot, the eventing model.
‫So the eventing model.
‫Coupled up with the what the what what's called with the compensating edit model SILLIAC.
‫Use COFCO, go in.
‫Q And when you publish when you create an order, the order service actually writes to that.
‫Q Or the public or a Kafka topic.
‫And by the way, I talked about cough, cough.
‫Check it out here.
‫Intellichoice Topinka says order has been created and people, people, services listen to this topic
‫and says, oh, look, created, it's my job as a payment service to pick up this order that I created
‫and start to pay it.
‫And if it paid it, what it does is says, OK, if it successful, it writes another topic, says payment
‫order paid successfully.
‫So the shipment service will listen to that and we'll start listening to that and pick it up the order
‫and if the payments failed.
‫It will write another topic, it will always write the topic and says, hey, payments failed and the
‫older service actually listens to the payment and to actually delete that, I think this is more elegant,
‫event driven architecture.
‫A lot of people talk about it.
‫I think it's a beautiful design, to be honest, and it's a really nice, elegant design to do this
‫thing.
‫And Kafka esque Kafka with distribution transaction people.
‫A lot more people are moving towards this architecture a lot because it's very elegant and nice.
‫Kind of has it has its own decoupling kind of a thing.
‫Right.
‫You still have the Kafka queue or the broker that does all that stuff.
‫But Robert complicated your stuff because you have to manage it out of Kafka is the one database.
‫But and the final solution is, believe it or not.
‫Go back to the old days, merge those dang services that are supposed to talk to the same database and
‫they are supposed to be homogeneous transactions in the same database, in the same service.
‫So go back to a mini monolith or a macro service.
‫It's a macro service in general.
‫Right.
‫Or a mini monolith, whatever you want to call it.
‫So the order and the shipment and the payment becomes the one service again.
‫Right.
‫But still, it's not does include every other things like shopping and cart service and product search
‫and doesn't include all of that stuff.
‫Right.
‫Includes all the stuff that everything kind of does the touching the database in a transactional manner.
‫So they use all the three methods that people are using to ensure distributed a transaction.
‫And when you say distributed transaction is is a transaction that spans multiple databases or spans
‫multiple services, multiple processes, not just A.D.A processes, doesn't have to be multiple services
‫on different computers or different containers.
‫They could be multiple processes in the same machine.
‫So that's kind of distributed in the main way.
‫It's not really horizontally distributed, but well, you can still you cannot achieve multiple transactions
‫with from different processes anyway.
‫So that kind of fall on the same track.
‫Let me know if that was useful.
‫Let me know if I said anything wrong or correct me if I said anything.
‫That doesn't really sound right.
‫I'm not really sure about the Google atomic clock.
‫I need to read more about it, to be honest.
‫But that was a summary of the transaction.
‫What do you think, guys?
‫Would you rather use a monolith and simplify your life or.
‫Would you rather.
‫Sacrifice and go to micro transactions and figure out how to distribute the transaction and figure out
‫how to minimize your latency, because now you spurt things down, you have network calls, you have
‫to figure that out and you have to also figure out how to debug your applications because debugging
‫is hard as well.
‫Right?
‫Because now the bug is no longer but a break.
‫But it's not it's not the same thing.
‫Right.
‫It's you have to put a attach a break debugger to this service and you need to know which exactly which
‫process to attach it to and also go to the other servers and attach the debugger.
‫And God knows that you hopefully you have one service of each one instance of each server so you can
‫nail debugging is hard.
‫So Data Dog and other providers and vendors are coming up with solutions like tracing and all that stuff.
‫Try to minimize, simplify these kind of things, but still micro services.
‫Communities, you have to do Bernardes, you have to manage all these daing containers, right?
‫So, yeah, yeah, I'm just very interested to see what where will micro services lead us.
‫Right.
‫So great architecture.
‫It's good to talk about it.
‫It's a really beautiful thing for nerds like us to nerd on.
‫And I just talk about Microsoft in general, but I don't know if it's feasible for every single one
‫of us or one to build.
‫Like Jenny, you want to build a donut shop.
‫Does she really need to spend up like Copernicus's Kluster to host her Statik website?
‫Probably not.
‫Yeah, I just like I'm just interested.
‫But the Googles and the Twitterers of the ward and the Facebook of the world.
‫Yeah.
‫They probably need this kind of micro services distributed so they can scale to the billions right off
‫request.
‫Yeah, I'm just very interested to know like how much would a monolith.
‫I don't.
‫Twitter reached a stage where they had to break down their micro services, their monolith they call
‫a monorail because it was built on rails.
‫Right.
‫Because they couldn't scale it out.
‫They that monolith was so big, spinning it up was so slow and just slow.
‫It doesn't it couldn't keep up with the requests, especially back in the World Cup.
‫2010 was it was year 2010.
‫I don't remember every time one of the Germany will score a goal or any any team will score a goal.
‫People will tweet Hegel.
‫And the moment do they do that, Twitter goes down immediately because this flux of just comes off all
‫over the world and the same exact second millisecond.
‫Right.
‫So, OK, let's let's just spin a more pvm there's been a more, but they couldn't keep up spinning
‫up more VIMS just spinning a single VM to hose this monolith just a couple of seconds.
‫And by the time it's spin up, it already filled up with requests, so.
‫They needed to break it down into micro services so they can scale the part that actually is busy and
‫instead of scanning the whole application, that is not like I mean, nobody's updating their profiles
‫in the World Cup.
‫Right?
‫Right.
‫The profile service just remains like in the single instance, whereas the actual rights service, whatever
‫it's called, the tweet service will will receive more instances.
‫Right.
‫So that's essentially where the how they did it.
‫And then it is very smartly.
‫All right, guys, that's it for me today.
‫Hope you enjoy this video and I'm going to see you in the next one.
‫What should I talk about next in the comments section below?
‫Give it a like subscribe if you didn't.
‫And.
‫Soon, the next one.
‫You guys stay safe, stay awesome.