Reserved seating vs general admission and system design

7 minute read

In retrospect, this explanation doesn’t make a lot of sense unless you’ve built high-scale systems like this before, so I thought I’d explain how these systems work, and why general admission sales might behave differently from reserved seating.

View on Twitter

To start, let’s imagine a the Ticketmaster website as a black box, and a customer trying to buy a ticket for a concert that does reserved seating. The first thing that happens is the customer requests to buy the seat - let’s say they ask for section 113, row A, seat 21.

A box representing Ticketmaster. A stick figure with an arrow to the box labeled "113-A-21 ?"

Ticketmaster looks to see if that seat is still available - lets say it is - and lets the customer know they got that seat. Then the customer decides whether they really want it, and, if so, gives Ticketmaster their payment.

Same box and stick figure as previous image. This time with an arrow from the box labeled "113-A-21" and an arrow to the box with a dollar sign.

This problem of having to get two separate systems (here Ticketmaster and the customer) to agree on a decision involving information that each holds privately is very common problem is distributed systems, and this particular style of solution is called two-phase commit (2PC).

It’s called two phase commit because first there’s a prepare phase, where all but one system is asked to prepare for some decision. If the system agrees to prepare the decision, it’s promising that it will follow through on the decision in the future no matter what.

In our example, Ticketmaster prepared the decision to sell the customer 113-A-21 and let the customer know that the decision was prepared. The customer then knows that if they decide to buy, they will definitely get that seat.

If another customer comes along and tries to buy 113-A-21 in the meantime, Ticketmaster will tell them “I’m sorry - that seat is not available.” If they prepared a decision to sell the other customer 113-A-21 then it has made a promise it can’t follow through on.

Same image as previous one. This time with another stick figure to the right of the box. Arrow from that figure labeled "113-A-21 ?" and an arrow from the box labeled with "Nope!"

In 2PC, if the final party (the one that didn’t prepare) decides to follow through with the decision, we say that the decision was “committed” - e.g. the customer committed to buying the ticket. If the customer clicks “No thanks,” we say the decision was aborted.

Now in our example, Ticketmaster doesn’t actually want to wait indefinitely before the customer makes the decision - what if the customer decides to go have dinner and think about it - so it tells the customer that they have the seat as long as they buy it within ten minutes.

Image with two stick figures and box. Box is replying to left figure with the reservation for 113-A-21, but no purchase yet. Picture of stopwatch with 10 minutes shaded above the left figure. Right figure still has "113-A-21 ?" arrow and "Nope!" response.

Above I told you that if a system prepares a decision, they are agreeing to follow through no matter what. But that’s only in the ideal world of two-phase commit. In practice, things can break, so prepare promises have timeouts guarding against a decision that never happens.

That’s the basic scenario - now lets look inside the black box. First, Ticketmaster is going to have many individual servers handling customer requests - I’ll draw a little box for the one each of our customers connect to.

Image with a box and two stick figures, one to each side of the box. The box now has two smaller boxes on the inside. The inner box on the left is red and the inner box on the right is blue.

Since our two customers each talk to different web servers, something has to tell the light blue server on the right not to let customer B buy seat 113-A-21. In general, that thing is going to be a database - likely a relational database such as Postgres.

Same image as previous tweet. This time the big box now also has the icon for database inside.

Side note: why are databases drawn as these weird stacks of tuna cans? Well, here’s a picture of the first computer with a disk drive, the IBM RAMAC. It was an accounting database. Those two big cabinets hold the disk drives.

The IBM 305 RAMAC. In the fore- and midground of the photo are two big cabinets, about 1m wide by 3m deep by 2m tall. In the background is a woman sitting at the console, operating the computer.

Here’s a closeup of the cylinder in each cabinet. It looks like a stack of vinyl records, which is pretty close. Each of those is a magnetic platter that stores information. Those disk towers became the symbol for databases.

A photo of the first disk drive, the IBM 350. From the side, you can see a stack of round platters.

Back to our flow: when customer A asks for seat 113-A-21, the web server looks that seat up in a database to see if it’s sold. If it’s not, it updates that seat to show that it’s promised to customer A, and how long that promise lasts.

Same image as previous schematic. Arrow from left customer to its web server with the label "113-A-21 ?" and from the web server to the database. There is now a little squiggle in the database.

When customer B comes along, its web server looks up 113-A-21 in the database, sees it’s promised to customer A, and tells B “So sorry.”

Same image as previous tweet. In addition there's an arrow from the right customer to its web server with the label "113-A-21 ?", an arrow from that web server to the database, and an arrow from the web server to the customer labeled "Nope!"

For reserved seat sales, there’s going to be an entry in the database for each unique seat. When somebody tries to buy one, the database is updated to show that promise, and then updated again when the customer pays.

What about general admission? In general admission, all the seats are fungible - the ticket I buy is exactly the same as the one you buy. How does this change the way the system works?

Instead of promising a customer that the particular seat they want won’t be sold to anybody else, for a GA event, the system is just promising that there are enough tickets left that the customer can get one.

You could imagine the system working in a very similar way, however. The system could keep a list of all the tickets. When a customer asked for one, it could note in the database that ticket “5327” was promised to customer A.

When customer B asked for a ticket, it would pick a different ticket, say “12895” and promise it to B. Customers wouldn’t explicitly ask for a particular ticket, but behind the scenes there would be a separate entry for each ticket, sold or not.

The problem with this approach is updating database entries consumes resources on the database, and databases are generally relatively expensive and harder to scale than web servers. So developers would like to avoid database transactions where possible.

Instead of keeping an entry for each ticket, the system could keep a “promised” count and a “sold” count. This simplifies the number of entries, but doesn’t reduce the number of updates to the database: you still have to record when a ticket is promised and again when it’s sold.

One way to cut the number of database updates in half is only to record the sales. When a customer asks for a ticket, you check and see that the total number of tickets sold is less than the maximum, and if it is, you promise it to them.

Then when they commit to purchasing, you update the count of tickets sold. Obviously, there’s a problem here. What if at that point, there are no tickets left?

Obviously, getting to this point is pretty foreseeable, and Ticketmaster is certainly more sophisticated than this. One way you could imagine getting most of the benefits of saving database transactions is for the system to have two modes.

When a customer asks for a ticket, the web server checks how many tickets are left. if there are more than, say 1,000, the web server goes ahead and promises a ticket to the customer and goes on with life. If the customer buys the ticket, the database is updated.

If there are fewer than 1,000 tickets left, the web server enters a different phase, where it writes the promise to the database. In this phase, the web server checks sold tickets and promised tickets and can be sure never to over promise.

The question is where to set this number? If you set it too high, you don’t get as much system benefit and you end up having to scale your database unnecessarily. If you set it too low, you can overshoot.

So generally you look at how many customers you might have in flight and set it to something bigger than the biggest number you have seen before. Or maybe you set it to a number that would only be exceeded once a year. Or whatever.

But however you set the number, it’s possible to get too many people in the system. So let’s say that you set it to 1,000. Then when there are 1,001 tickets left, you get a big rush of customers: 20,000 people all come at once to buy a ticket.

Each web server looks in the database, sees that there are more than 1,000 tickets left, and responds that they can have a ticket. Because none of those purchases have gone through yet, even after 5,000 people have been promised a ticket, the database still shows 1,001 available.

If the customers come fast enough, all 20,000 could be promised a ticket before the first few actually complete their purchase so that it’s recorded in the database that there are fewer than 1,000 tickets left.

My point in the OP was not that this is what happened, but only that a bug that oversells seats might be at least plausible for GA seating, but very unlikely with reserved seating - I was responding to the the question of if it could have been an accident.

I’m not asserting that this is how Ticketmaster’s systems work for GA. It was more of a “I can’t think of how it would be accidental” offhand remark! Now if you were trying to order a Playstation 2 on Amazon in 2000, this might describe what happened!

View on Twitter