Reserved seating vs general admission and system design
That makes no sense. It’s still a finite number of tickets.
— Caryn Rose (@carynrose) November 17, 2022
In retrospect, this explanation doesn’t make a lot of sense unless you’ve built high-scale systems like this before, so I thought I’d explain how these systems work, and why general admission sales might behave differently from reserved seating.
To start, let’s imagine a the Ticketmaster website as a black box, and a customer trying to buy a ticket for a concert that does reserved seating. The first thing that happens is the customer requests to buy the seat - let’s say they ask for section 113, row A, seat 21.
Ticketmaster looks to see if that seat is still available - lets say it is - and lets the customer know they got that seat. Then the customer decides whether they really want it, and, if so, gives Ticketmaster their payment.
This problem of having to get two separate systems (here Ticketmaster and the customer) to agree on a decision involving information that each holds privately is very common problem is distributed systems, and this particular style of solution is called two-phase commit (2PC).
It’s called two phase commit because first there’s a prepare phase, where all but one system is asked to prepare for some decision. If the system agrees to prepare the decision, it’s promising that it will follow through on the decision in the future no matter what.
In our example, Ticketmaster prepared the decision to sell the customer 113-A-21 and let the customer know that the decision was prepared. The customer then knows that if they decide to buy, they will definitely get that seat.
If another customer comes along and tries to buy 113-A-21 in the meantime, Ticketmaster will tell them “I’m sorry - that seat is not available.” If they prepared a decision to sell the other customer 113-A-21 then it has made a promise it can’t follow through on.
In 2PC, if the final party (the one that didn’t prepare) decides to follow through with the decision, we say that the decision was “committed” - e.g. the customer committed to buying the ticket. If the customer clicks “No thanks,” we say the decision was aborted.
Now in our example, Ticketmaster doesn’t actually want to wait indefinitely before the customer makes the decision - what if the customer decides to go have dinner and think about it - so it tells the customer that they have the seat as long as they buy it within ten minutes.
Above I told you that if a system prepares a decision, they are agreeing to follow through no matter what. But that’s only in the ideal world of two-phase commit. In practice, things can break, so prepare promises have timeouts guarding against a decision that never happens.
That’s the basic scenario - now lets look inside the black box. First, Ticketmaster is going to have many individual servers handling customer requests - I’ll draw a little box for the one each of our customers connect to.
Since our two customers each talk to different web servers, something has to tell the light blue server on the right not to let customer B buy seat 113-A-21. In general, that thing is going to be a database - likely a relational database such as Postgres.
Side note: why are databases drawn as these weird stacks of tuna cans? Well, here’s a picture of the first computer with a disk drive, the IBM RAMAC. It was an accounting database. Those two big cabinets hold the disk drives.
Here’s a closeup of the cylinder in each cabinet. It looks like a stack of vinyl records, which is pretty close. Each of those is a magnetic platter that stores information. Those disk towers became the symbol for databases.
Back to our flow: when customer A asks for seat 113-A-21, the web server looks that seat up in a database to see if it’s sold. If it’s not, it updates that seat to show that it’s promised to customer A, and how long that promise lasts.
When customer B comes along, its web server looks up 113-A-21 in the database, sees it’s promised to customer A, and tells B “So sorry.”
For reserved seat sales, there’s going to be an entry in the database for each unique seat. When somebody tries to buy one, the database is updated to show that promise, and then updated again when the customer pays.
What about general admission? In general admission, all the seats are fungible - the ticket I buy is exactly the same as the one you buy. How does this change the way the system works?
Instead of promising a customer that the particular seat they want won’t be sold to anybody else, for a GA event, the system is just promising that there are enough tickets left that the customer can get one.
You could imagine the system working in a very similar way, however. The system could keep a list of all the tickets. When a customer asked for one, it could note in the database that ticket “5327” was promised to customer A.
When customer B asked for a ticket, it would pick a different ticket, say “12895” and promise it to B. Customers wouldn’t explicitly ask for a particular ticket, but behind the scenes there would be a separate entry for each ticket, sold or not.
The problem with this approach is updating database entries consumes resources on the database, and databases are generally relatively expensive and harder to scale than web servers. So developers would like to avoid database transactions where possible.
Instead of keeping an entry for each ticket, the system could keep a “promised” count and a “sold” count. This simplifies the number of entries, but doesn’t reduce the number of updates to the database: you still have to record when a ticket is promised and again when it’s sold.
One way to cut the number of database updates in half is only to record the sales. When a customer asks for a ticket, you check and see that the total number of tickets sold is less than the maximum, and if it is, you promise it to them.
Then when they commit to purchasing, you update the count of tickets sold. Obviously, there’s a problem here. What if at that point, there are no tickets left?
Obviously, getting to this point is pretty foreseeable, and Ticketmaster is certainly more sophisticated than this. One way you could imagine getting most of the benefits of saving database transactions is for the system to have two modes.
When a customer asks for a ticket, the web server checks how many tickets are left. if there are more than, say 1,000, the web server goes ahead and promises a ticket to the customer and goes on with life. If the customer buys the ticket, the database is updated.
If there are fewer than 1,000 tickets left, the web server enters a different phase, where it writes the promise to the database. In this phase, the web server checks sold tickets and promised tickets and can be sure never to over promise.
The question is where to set this number? If you set it too high, you don’t get as much system benefit and you end up having to scale your database unnecessarily. If you set it too low, you can overshoot.
So generally you look at how many customers you might have in flight and set it to something bigger than the biggest number you have seen before. Or maybe you set it to a number that would only be exceeded once a year. Or whatever.
But however you set the number, it’s possible to get too many people in the system. So let’s say that you set it to 1,000. Then when there are 1,001 tickets left, you get a big rush of customers: 20,000 people all come at once to buy a ticket.
Each web server looks in the database, sees that there are more than 1,000 tickets left, and responds that they can have a ticket. Because none of those purchases have gone through yet, even after 5,000 people have been promised a ticket, the database still shows 1,001 available.
If the customers come fast enough, all 20,000 could be promised a ticket before the first few actually complete their purchase so that it’s recorded in the database that there are fewer than 1,000 tickets left.
My point in the OP was not that this is what happened, but only that a bug that oversells seats might be at least plausible for GA seating, but very unlikely with reserved seating - I was responding to the the question of if it could have been an accident.
I’m not asserting that this is how Ticketmaster’s systems work for GA. It was more of a “I can’t think of how it would be accidental” offhand remark! Now if you were trying to order a Playstation 2 on Amazon in 2000, this might describe what happened!