View on Twitter

The problem had to do with semantic coupling, which is a particularly insidious form of coupling. It’s insidious because it was probably good when it was put in, isn’t explicitly in any API, and is very hard to take out.

The goal of FBA was to allow 3ʳᵈ-party merchants to use the Amazon fulfillment network. At the time (2005), you could buy products on the Amazon website where the seller was not Amazon, but if you did, the seller had to ship the product to you from their facilities.

We wanted to allow 3ʳᵈ parties to buy inventory and ship it to Amazon fulfillment centers. Then when a customer ordered that product, Amazon would box the item up (maybe with other Amazon-fulfilled orders) and get it to the customer.

As we started looking at what we’d need to do to offer this service, we quickly found a big problem: the ASIN. The ASIN (Amazon Standard Identification Number) is the unique identifier for every product on the Amazon website. So what was the problem?

The ASIN uniquely identifies what a shopper thinks of as a particular “thing.” For example, What if? in paperback has one ASIN (1848549563), while the hardcover version has a different one (0544272994).

At the time we were starting FBA in 2005, those ASINs were also used for fulfillment. So if I ordered three different items, a list of those three ASINs would be sent to the fulfillment network, where pieces of inventory matching each ASIN would be boxed up and shipped.

This worked great when all the inventory in our fulfillment centers was owned by Amazon, but if we have a paperback What If? owned by Amazon in our FC and another owned by Bob’s Books, when the customer buys from Bob, the ASIN isn’t enough information to know which one to ship.

Now maybe you can make an argument that the two physical books are “the same” and it doesn’t matter which one you ship (though google “FBA comingled inventory” for more on that), a primary goal of FBA was to attract used book sellers.

Obviously, in some sense, all used copies of What If are the “same” - one of the strengths of Amazon is presenting a single web page for What If that shows all the offers. But after I buy one with a particular wear description, Amazon needs to ship that one.

We’ve gotten into the heart of the problem now. When Amazon launched in 1995, the ASIN¹ represented a thing for sale on the web site, implicitly new and sold by Amazon². The FC software took that identifier and used it with all the implications left unstated.

Once 2005 rolled around, the implications of seller and condition had been realized and unrolled on the website, but the FC software hadn’t needed to. The website software and the FC software were coupled by the semantics of the ASIN, but those semantics had started to drift.

In Code Complete the author writes about the dangers of semantic coupling, and describes it as one module using knowledge of another’s inner workings. That’s true, but in my experience, easy to avoid. Our example is one that is much harder to avoid.

Our example is one where we are modeling in code something in the real world. The ASIN represented a “book for sale” on the website, and a “book to be fulfilled” in the FC software. These two concepts seemed to be the same when Amazon launched, and practically they were.

The challenge with modeling the real world in code is figuring out which parts of the world to model and which parts to elide. In general, you want the model to be as simple as possible but no simpler. The problem is that the definition of “no simpler” changes over time.

Over time, as the system evolved, it became clear that these were two separate concepts and needed to be modeled as such. The FC software needed an ID for a “fungible physical thing for order fulfillment” as opposed to a “thing that customers think of as a unique product.”

These two IDs don’t really need to have anything in common with each other. If you were building a company that only did fulfillment, you’d have an ID for that concept, and it would have attributes such as weight, dimensions, shipping restrictions, etc.

But if you had those two concepts as separate, you’d need some way to tie them together. An inventory system would have to keep track of the correspondence between, for example, the ID for What If that the website uses and the one the FC uses for that physical book.

Also, much of the information that the FC systems need, the website needs as well. So either you’d have to duplicate that information and keep it updated, or the inventory system would have to keep track of it all.

Obviously Amazon has that system now, because the two IDs have been broken apart. But in 1995, it was much faster to tie all that information together with this one ID. So ASINs modeled both the thing for sale and the physical item.

It would be easy in 2005 to look back and say, “They should have thought harder about this coupling and modeled it better.” I’ve heard some version of that refrain dozens if not hundreds of times in my career.

But I dunno. Maybe. It’s easy in retrospect to see the axes of flexibility that are important and the places where you need independence. But at the time, you might be able to come up with a hundred places to add flexibility, of which only twenty would add value.

While I do think it’s very important to think clearly about the concepts you are modeling (and writing things in prose helps with this a lot), I also think it’s important to move fast and ship software. Hopefully the value that senior engineers brings is discerning that balance.

One way I’ve gotten engineers to think about where they have coupling between internal systems is to think about what would have to change if you replaced one part of that system with something sold by a 3ʳᵈ party.

In a sense, that was what we were doing with FBA, but in reverse. We needed to convert the Amazon fulfillment network from internal software (where semantic coupling is possible and sometimes good) to an externalized service (where it’s not possible).

Whenever I’ve been involved in an effort to externalize software that was written for internal use, uncovering and fixing these places with semantic coupling have been the hardest parts of the job.

It’s why in AWS we have only rarely tried to directly externalize internal software. DynamoDB was based on what we learned building the system described in the Dynamo paper, but it was written from scratch to be an external service.

CloudWatch was based on what we learned building our internal metrics system, PMET, but it was written from scratch to be an external service.

We couldn’t do that with FBA, because the scope of all the fulfillment software was just too big. So we powered through, but it was a lot of work figuring out how to separate the concepts and retrofitting the software in place.

FBA has obviously been a huge success (though Jeff said it took too long to launch), so I wouldn’t say never try to externalize an internal service. But be very aware that the hard parts are likely things that you aren’t thinking of.

As I wrote at the top of the tweet, this thread was inspired by some consulting work I’ve been doing. If you are interested in learning more, see my services page.

View on Twitter

Want to be promoted?

1. Tell your manager
2. Ask them for a bullet point list of things they’d need to see from you to feel comfortable promoting you
3. Continually check in w/ them to see how you’re progressing against that checklist

This will increase your chances of success.

This is good advice, and I want to expand on it a little more for senior (like Principal plus) tech IC promotions, because one failure mode I often see is that engineers are looking for the bullets to be checklist items, but they won’t be, here’s how I think about them:

View on Twitter

Amazon has extensive and detailed job level guidelines, and each of them contain a “moving to” section. This section is literally bullet points on what you need to demonstrate to be promoted to the next level.

However, these items are not “checklist” items - they are general areas to demonstrate proficiency at a particular level. For example, one of the items reads something like “you are seen as an expert in your area and your opinion is regularly sought out by other teams.”

The items listed on this site are not exactly as I remember, but they are representative, e.g. “Your delivery of code and architecture sets the standard for the organization with a high focus on engineering excellence and innovation.”

So you have a bullet list of items such as these; what do you do with that?

A friend of mine used a “stoplight” system to help engineers understand how to fulfill these criteria. For each item in the “moving to” list, he and the engineer would independently rate the engineer on a red/yellow/green scale.

Green would mean “sufficiently demonstrating at the next level,” yellow would mean “inconsistently demonstrating at the next level,” and red would mean “not demonstrating at the next level.”

It’s really important to do the exercise independently, because some of the most valuable information is where the ratings differ.

Often, the difference is that the employee would have a higher self-rating than the manager (or mentor). It’s critical to suss these differences out, because they are going to be the source of frustration in the promotion process.

When there is a difference, it could be because the manager is unaware of examples where the employee is demonstrating that skill. However, it’s often because the employee is mis-calibrated in the expectations for the job level.

For example, in the “opinion is regularly sought out” item, if the employee’s opinion was being sought out by SDE3s on peer teams, that would be green for promotion to SDE3. For promotion to Senior Principal, however, that level of visibility would be red.

The more senior you get, the more abstract the list items are going to be, and the more subjective the assessment. You need to make sure that you are at first agreeing on the facts of what you are demonstrating, and then are agreeing on their impact relative to expectations.

I noted that you could could do this exercise with a mentor. If you are struggling to understand promotion expectations, a mentor’s judgment can be invaluable. They should have a very good understanding of the criteria, and their assessment might feel more collaborative.

If you try this process out, asking somebody to assess you, the most important part is to really listen to what you are being told about your yellow and red areas. If you really don’t agree, ask a second person to do it and compare notes.

And if you are already in one of these roles and mentoring people, I encourage you to do the exercise with them. It’ll help you know where to help them, but also if you find that you would rate your mentee mostly green, go challenge their manager to put them up for promotion.

View on Twitter

That makes no sense. It’s still a finite number of tickets.

In retrospect, this explanation doesn’t make a lot of sense unless you’ve built high-scale systems like this before, so I thought I’d explain how these systems work, and why general admission sales might behave differently from reserved seating.

View on Twitter

To start, let’s imagine a the Ticketmaster website as a black box, and a customer trying to buy a ticket for a concert that does reserved seating. The first thing that happens is the customer requests to buy the seat - let’s say they ask for section 113, row A, seat 21.

Ticketmaster looks to see if that seat is still available - lets say it is - and lets the customer know they got that seat. Then the customer decides whether they really want it, and, if so, gives Ticketmaster their payment.

This problem of having to get two separate systems (here Ticketmaster and the customer) to agree on a decision involving information that each holds privately is very common problem is distributed systems, and this particular style of solution is called two-phase commit (2PC).

It’s called two phase commit because first there’s a prepare phase, where all but one system is asked to prepare for some decision. If the system agrees to prepare the decision, it’s promising that it will follow through on the decision in the future no matter what.

In our example, Ticketmaster prepared the decision to sell the customer 113-A-21 and let the customer know that the decision was prepared. The customer then knows that if they decide to buy, they will definitely get that seat.

If another customer comes along and tries to buy 113-A-21 in the meantime, Ticketmaster will tell them “I’m sorry - that seat is not available.” If they prepared a decision to sell the other customer 113-A-21 then it has made a promise it can’t follow through on.

In 2PC, if the final party (the one that didn’t prepare) decides to follow through with the decision, we say that the decision was “committed” - e.g. the customer committed to buying the ticket. If the customer clicks “No thanks,” we say the decision was aborted.

Now in our example, Ticketmaster doesn’t actually want to wait indefinitely before the customer makes the decision - what if the customer decides to go have dinner and think about it - so it tells the customer that they have the seat as long as they buy it within ten minutes.

Above I told you that if a system prepares a decision, they are agreeing to follow through no matter what. But that’s only in the ideal world of two-phase commit. In practice, things can break, so prepare promises have timeouts guarding against a decision that never happens.

That’s the basic scenario - now lets look inside the black box. First, Ticketmaster is going to have many individual servers handling customer requests - I’ll draw a little box for the one each of our customers connect to.

Since our two customers each talk to different web servers, something has to tell the light blue server on the right not to let customer B buy seat 113-A-21. In general, that thing is going to be a database - likely a relational database such as Postgres.

Side note: why are databases drawn as these weird stacks of tuna cans? Well, here’s a picture of the first computer with a disk drive, the IBM RAMAC. It was an accounting database. Those two big cabinets hold the disk drives.

Here’s a closeup of the cylinder in each cabinet. It looks like a stack of vinyl records, which is pretty close. Each of those is a magnetic platter that stores information. Those disk towers became the symbol for databases.

Back to our flow: when customer A asks for seat 113-A-21, the web server looks that seat up in a database to see if it’s sold. If it’s not, it updates that seat to show that it’s promised to customer A, and how long that promise lasts.

When customer B comes along, its web server looks up 113-A-21 in the database, sees it’s promised to customer A, and tells B “So sorry.”

For reserved seat sales, there’s going to be an entry in the database for each unique seat. When somebody tries to buy one, the database is updated to show that promise, and then updated again when the customer pays.

What about general admission? In general admission, all the seats are fungible - the ticket I buy is exactly the same as the one you buy. How does this change the way the system works?

Instead of promising a customer that the particular seat they want won’t be sold to anybody else, for a GA event, the system is just promising that there are enough tickets left that the customer can get one.

You could imagine the system working in a very similar way, however. The system could keep a list of all the tickets. When a customer asked for one, it could note in the database that ticket “5327” was promised to customer A.

When customer B asked for a ticket, it would pick a different ticket, say “12895” and promise it to B. Customers wouldn’t explicitly ask for a particular ticket, but behind the scenes there would be a separate entry for each ticket, sold or not.

The problem with this approach is updating database entries consumes resources on the database, and databases are generally relatively expensive and harder to scale than web servers. So developers would like to avoid database transactions where possible.

Instead of keeping an entry for each ticket, the system could keep a “promised” count and a “sold” count. This simplifies the number of entries, but doesn’t reduce the number of updates to the database: you still have to record when a ticket is promised and again when it’s sold.

One way to cut the number of database updates in half is only to record the sales. When a customer asks for a ticket, you check and see that the total number of tickets sold is less than the maximum, and if it is, you promise it to them.

Then when they commit to purchasing, you update the count of tickets sold. Obviously, there’s a problem here. What if at that point, there are no tickets left?

Obviously, getting to this point is pretty foreseeable, and Ticketmaster is certainly more sophisticated than this. One way you could imagine getting most of the benefits of saving database transactions is for the system to have two modes.

When a customer asks for a ticket, the web server checks how many tickets are left. if there are more than, say 1,000, the web server goes ahead and promises a ticket to the customer and goes on with life. If the customer buys the ticket, the database is updated.

If there are fewer than 1,000 tickets left, the web server enters a different phase, where it writes the promise to the database. In this phase, the web server checks sold tickets and promised tickets and can be sure never to over promise.

The question is where to set this number? If you set it too high, you don’t get as much system benefit and you end up having to scale your database unnecessarily. If you set it too low, you can overshoot.

So generally you look at how many customers you might have in flight and set it to something bigger than the biggest number you have seen before. Or maybe you set it to a number that would only be exceeded once a year. Or whatever.

But however you set the number, it’s possible to get too many people in the system. So let’s say that you set it to 1,000. Then when there are 1,001 tickets left, you get a big rush of customers: 20,000 people all come at once to buy a ticket.

Each web server looks in the database, sees that there are more than 1,000 tickets left, and responds that they can have a ticket. Because none of those purchases have gone through yet, even after 5,000 people have been promised a ticket, the database still shows 1,001 available.

If the customers come fast enough, all 20,000 could be promised a ticket before the first few actually complete their purchase so that it’s recorded in the database that there are fewer than 1,000 tickets left.

My point in the OP was not that this is what happened, but only that a bug that oversells seats might be at least plausible for GA seating, but very unlikely with reserved seating - I was responding to the the question of if it could have been an accident.

I’m not asserting that this is how Ticketmaster’s systems work for GA. It was more of a “I can’t think of how it would be accidental” offhand remark! Now if you were trying to order a Playstation 2 on Amazon in 2000, this might describe what happened!

View on Twitter

View thread on Twitter

I left Amazon/AWS in June, 2022, and it’s time to start working again. I’m not looking for the same kind of role that I held at Amazon/AWS, and this thread attempts to explain what I’m looking for. DMs open/RTs appreciated.

My ideal role likely doesn’t exist. I’ve repeatedly led small teams to build ambitious, deeply technical products that had great impact. But I know that such a role often requires massive time and emotional commitment, and my life circumstances don’t allow that right now.

So that’s the first thing I’ll say: I’m not looking for a full-time role. I have too many other things going on in my life right now. I would like to find something meaningful and impactful to do, but it will need to be limited in hours or duration.

OK, so what am I looking for? I would like to do technical work that has meaning. I love helping teams deliver, either leading initial projects or improving teams and systems that need to scale to the next level.

Most importantly, I want to work with nice people on hard problems. I love collaboration and the rush of a team coming together to do something special. I enjoy working with a mix of experience levels. I also value diversity of backgrounds in my coworkers.

As a side note, I was the DE sponsor for adding “Lead with Empathy” to our PE tenets. I won’t consider working for a company that doesn’t take a strong stand on being thoughtful leaders.

What do I bring to the table? Working so long at Amazon has given me lots of experience analyzing people, projects, products, and technology. I’ve done hundreds of design reviews, over a thousand interviews, and several due-diligence analyses.

Importantly, I also delivered. In AWS, I launched three major new initiatives, EBS, the software that runs on our datacenter switches, and QLDB. I also led DynamoDB development, turning around a team that was struggling to deliver.

Working at Amazon trained me to focus on finding the path to delivery. All of my initial launches were less than a year from start to production. The turn-arounds took longer, but always with a strong focus on tangible delivery.

I would love to work on something hard and deeply technical that leverages my skills and experience, and where people are willing to take limited hours from me (e.g. 10-20 hrs/wk). So I’m throwing this out there into the world to see what comes back.

If any of this sounds like a possibility, please get in touch. You can DM me on Twitter, email jobs@thecertains.com, or use LinkedIn. For more words on my background, see my about page.

PS. One thing that won’t be a match: if you need an expert in building on AWS. I am an expert in building some of AWS, but I have only moderate experience using the services. AWS has amazing solutions architects that are experts in building on AWS.

View thread on Twitter

Now that many are talking about the ZX Spectrum due to the passing of Clive Sinclair. It would be remiss of me to add that Mike Singleton’s “The Lords of Midnight” was the most innovative sandbox adventure/strategy games of its age and is not as celebrated as it should be. pic.twitter.com/NS1PHqLdUW

View thread on Twitter

This tweet got me thinking: what are those watershed games that changed the industry’s perception of what was possible, and how did they do it? I’ll start with one: Pitfall!

To understand how amazing Pitfall was, you have to understand a little about the Atari 2600. It didn’t have a frame buffer - instead it had a video chip (the TIA) that could draw a scan line based on 6 elements: a background, plus 5 sprites: 2 players, 2 missiles, and a ball.

The background had a 2 color registers plus 20 bits indicating which color should be drawn for the left 80 pixels of the screen (each bit controlled 4 adjacent pixels). For the right half, the same 20 bits were used, either in the same order or reflected.

The player and missile sprites are paired and each pair gets a single color. The player sprites are 8 bits of on/off plus position and the missiles are 1 bit plus position. Finally, there’s the independent ball sprite which is 1 bit as well, and uses the background color.

The player/missile pairs can be replicated up to 3 times on a scan line, and the 1-bit elements can be stretched to cover 1, 2, 4, or 8 pixels.

Combat uses the chip as intended. Each row in the game is 80 pixels on the left, reflected on the right, with pixels colored in 4-pixel chunks. Each player has a tank or plane (or three) of a single color, and can shoot missiles at their opponent’s tanks.

Note that all these elements are one-dimensional. So when drawing a tank, the CPU has to wait until the first scan line where the tank is, then load the TIA with that slice, wait until the end of that scan line, load the TIA with the next slice, and do on.

The TIA is why the 2600 version of Space Invaders has 6 columns of aliens: it’s using the two player sprites each replicated three times. Now there is a trick going on here, because aliens get shot down independently.

The programmer set the two player sprites to “wide replication,” which spread them out enough that they could be interleaved “A B A B A B.” Then there was just enough time (two CPU cycles to be exact) between the drawing of two copies to change the bitmap.

This is part of the fun of direct hardware - it’s all just electrical signals. So when it’s clocking out another pixel, it just takes what’s in the register at that point in time. You better get your timing down exactly!

Adventure added a couple of other innovations that preceded Pitfall. First, the “world” is bigger than a screen. So when the player moves off the screen in one direction, they find themselves on a different screen.

Adventure also has more moving objects than there are sprites, so when there were too many things on the screen - bats, dragons, the sword, etc. - the components started flashing. The code uses a sprite for one item one frame and a different one on the next.

Now to Pitfall. Compared to its predecessors, the look and feel is totally different. It has a rich background, with vines, pits that open and close, and lots of moving obstacles with no flashing. How did they do it?

At some level it’s the same tricks mentioned above: you reuse the sprites for different things on different scan lines, or even on the same scan line. But they took it to a qualitatively different level. In large part, the advances were mostly in the play field.

First, both colors in the background element are used, and they vary as the scan lines move down, in eight regions. At the top is solid dark green for the tree tops, then dark and light green to form the canopy edge, and so on.

You can tell when the background element is being used because it’s almost always reflected across the middle. So the tree trunks vary, but they always mirror across the middle. Also, the colors always change at 4-pixel boundaries.

Second, they use player sprites for fine details in the background: the 1-pixel tree branches, the brick walls, the timer and score. See how much nicer the timer and score numbers look relative to those in Combat. The ones in Combat use the background element.

Third, the animation of the background element to make the pits that open and close.

Because they use one player sprite for the game character, they only have one more to use for any scan line where the player could be. This is why you never see a scorpion on a screen with a brick wall, or a fire combined with rolling logs.

However, because they were careful about ensuring that they never needed more than a single non-character sprite per scan line, they never get the flashing that detracted from Adventure.

The play field is what really sets the look and feel for the game, but I’ll note a few other tricks that the game employed.

First, the vines use the ball sprite, which is why if you look closely, the color of the vine changes when it gets up near the canopy. The TIA had collision detection built in, so it could tell the CPU if the player had grabbed the vine - the CPU didn’t have to figure that out.

Second, they changed the color for the character sprite on each scan line, to make the head, shirt, and body. All of these tricks rely on the fact that it’s a side-scroller, so movement is very restricted: the character can’t rotate and most of the movement is lateral.

Finally, they didn’t turn on the TIA to draw a scan line right away - this is why the game has a black region on the left, and the background mirroring isn’t complete.

They didn’t draw those 8 pixels to give them a few more cycles to compute each position on that scan line. Literally three more CPU cycles. The game was that tight that they needed those extra cycles to get everything done.

When Pitfall! was released it felt so much closer to the arcade games of the time. As noted, the restrictions from side-scrolling were a big help. Another game that used restricted movement to deliver its breakthrough experience is Doom. But that’s a story for another time!

View thread on Twitter

Why school is so important, distilled: Each year of schooling increases IQ by 1-5 points, and these effects persist for a lifetime. This meta-analysis concludes "Education appears to be the most consistent, robust and durable method yet to be identified for raising intelligence." pic.twitter.com/vH4wI6VIyo

View thread on Twitter

I wrote a tweet about why I was skeptical about these claims, but before sending, I decided to actually read the paper (H/T @hillelogram). I discovered that it seems to be a well-done study that does not conclude that “each year of schooling increases IQ.” Here’s what I found.

The reason I was skeptical because the vast majority of people in the US have between 12 and 18 years of schooling. If each year of schooling increased IQ between 1 and 5 points, it would be very hard to find that signal.

The problem is that the within-group variation would be so large. Somebody with just a HS education would have between 12 and 60 points of benefit, while somebody with a graduate degree would have between 18 and 90 points.

It is possible to find these differences between groups even if groups have such high within-group variation, but it’s hard. So I was skeptical. (Also, obviously, it’s insane to think that any schooling could have 60 points of IQ benefit.)

But before hitting send I had my second thoughts and found the paper (which wasn’t linked in the original tweet, but it’s here). The first thing I found was that the paper goes to great pains to control for the problems plaguing social science statistical studies.

If you’re not up on these problems (p-hacking, file-drawer effect, the garden of forking paths), you could do worse than to read https://statmodeling.stat.columbia.edu/, or just about anything by Andrew Gelman.

I’m not an expert in these topics by any means, but they are at least aware of these problems and seem to take reasonable measures to counteract them.

OK, so what does the paper conclude? It’s right there in the abstract: “we found consistent evidence for beneficial effects of education on cognitive abilities of approximately 1 to 5 IQ points for an additional year of education.”

But, wait, you say, “Isn’t that what the quoted tweet said?” No. And it illuminates how these games of telephone distort good research. The paper is looking at the impact of the marginal impact of a single additional year of education.

That means you look at people who are going to get 12 years and instead get 13. Or you look at people who were going to get 8 years and instead get 9. It does not mean that they are measuring the cumulative impact of multiple years of schooling.

The paper even addresses this point directly: “The finding of educational effects on intelligence raises a number of important questions that we could not fully address with our data. First, are the effects on intelligence additive across multiple years of education?”

So the paper says “one extra year of schooling is worth 1-to-5 IQ points.” Then the tweet changes that to “every year of schooling is worth 1-to-5 IQ points.” A small change, but it distorts the meaning in a way that (inadvertently) sensationalizes the results.

Maybe this isn’t the biggest deal, but it was interesting to me to see how really good work gets summarized in ways that subtly change its meaning.

Getting back to my original skepticism, it was from the large spread on effect size combined with the “every year” misstatement. I’m now less skeptical about the study, per se, but I think that it’s only a start.

I think “a year of education” is at best a weak proxy for the true cause. There are surely some types of education that improve reasoning ability, but maybe the variance in the estimate hints that the kind of education matters (and not just the quality).

And, what do you know, the paper authors agree: “Finally, what are the underlying psychological mechanisms of the educational effect on intelligence?” It’s their final area for more study, and, I think, the most important!

Also, obviously the biggest problem with all of this is that the whole concept of IQ testing is fundamentally flawed. See Radiolab’s excellent “G” for a good primer.

View thread on Twitter

From quiet homes and first beginnings,
out to the undiscovered ends.
There’s nothing worth the wear of winning,
save laughter and the love of friends.

— Dedicatory Ode, Hilaire Belloc, 1945

When my mom told my sisters and me that Dad had requested a poem be read at his service, our first guess was that he had picked Casey at the Bat. You see, in grade school, Dad was given an assignment to memorize and recite a poem, and he picked Casey at the Bat. But after he recited it (flawlessly), the teacher failed him, saying that “real poetry” couldn’t be about baseball.

In any event, this poem – really just a stanza from a longer ode – makes more sense for his memorial service. Anybody who has experienced my parents’ hospitality knows that Dad cherished both laughter and the love of friends. And I know that this stanza was important to him, because I carry a hand-written copy in my wallet. He gave it to me when my wife Margaret and I moved out to Seattle together – a reminder of what matters.

People mattered to my father. Since his death, I’ve heard many new stories about the way he touched people’s lives. As Margaret put it, “He had a way of being generous, without making it about himself at all.”

From my sister, I heard about a time in high school when a late-night ad for McDonald’s came on TV. She mused out loud how she could really go for some French fries, and the next thing she knew, Dad had gone into the kitchen and cooked some home-made fries for her. I’m sure anybody who knew him is not surprised at this story.

My cousin Brian recounted a story from when he was in high school and starting to take up running. My dad saw him with his running shoes and asked if Brian would like company on his run. After tiring him out running around the neighborhood, my dad challenged Brian to a sprint back to the house. In Brian’s telling, my dad was back in the house and showered before he made it up the driveway. He may be exaggerating a bit (he is a Southerner after all), but my dad’s running made a lasting impression. My cousin told this story to share with me his experience of how avid a runner my dad was. However, as far as my family and I remember, my father was athletic, but not particularly a runner. I think he just saw an opportunity to quietly support somebody he cared about by simply showing up.

Though these are little examples, he was quietly generous in big ways too. When my mother got a job teaching at Lawrence University in Appleton, WI, she would be gone every week from Monday morning to Wednesday night and all day Friday. My father quietly took on these extra duties - making our meals and making sure we made it to sports, choir practices, and whatnot – so that my mother could do something she loved.

But none of this is to say that my father didn’t also care about his career. He cared a lot about it. In fact, the stanza that I carry is copied on the back of the business card for his last job – that of Dean of the College of Letters and Science at Wisconsin. It was his dream job. It allowed him to support the people in the college as they fulfilled a mission he cherished.

Although I didn’t get to see him go about his job the same way I got to see him go about being a father, grandfather, husband, and friend, I know that his quiet generosity came through there as well. A few months ago, I was talking to a former University of Wisconsin faculty member who was considering an Amazon Scholar position. These positions are for distinguished members in technical fields, and this person was no exception – I felt honored to talk with him. When he found out who my father was, he told me how much my father meant to him and to his career, getting choked up in the process. The son of another former faculty member told me a similar story about what my father did for his mom.

You can read lots more stories like these, both of him as a family member and as a UW faculty member, on the tribute wall at his obituary. I feel lucky to have been raised by him, and I try to emulate his generosity in my own life.

And now to close, the final stanza of Casey at the Bat:

Oh, somewhere in this favored land the sun is shining bright;
the band is playing somewhere, and somewhere hearts are light,
and somewhere men are laughing, and somewhere children shout;
but there is no joy in Mudville — mighty Casey has struck out.

— Casey at the Bat, Ernest Thayer, 1888

Phillip Robinson Certain, September 21, 1943 - August 11, 2020

Got a recipe handy? I would love to try that!

— Than (@mynockspit) March 29, 2020

View thread on Twitter

I’ve been cooking cornbread for a long time and eating it for much longer. I grew up in Wisconsin, but both my parents are from Georgia. Also, I lived for seven years in North Carolina, so I’ve been exposed to lots of cornbread.

The wikipedia article on cornbread is pretty good, explaining the differences between northern and southern cornbread (though I take some issue with describing the northern version as “slightly different”).

The cornbread I make is squarely in the southern camp. I enjoy a good corn muffin, but it’s not cornbread - I’m 100% Team No Sugar! (The linked Serious Eats article - from back when SE generated great content - is also worth reading.)

The truth is that cornbread is very forgiving. I’ve experimented with lots of variations (here are two pages of notes from my cooking notebook, e.g.).

At this point in my life, I often just wing it, but I’ll offer a basic recipe to start you off.

Start by putting a 9” cast iron skillet in oven set to 400 degrees. While preheating, mix the dry ingredients together:

• 1½ c medium grind cornmeal (e.g. Bob’s Red Mill)
• 1 c flour
• 1 t baking powder
• ½ t baking soda
• 1 t table salt

Now you want to wait until the oven is preheated. Maybe longer. You want the skillet to be completely up to temp (use an IR thermometer if you are that kind of cook). Then remove the skillet from the oven and add 3T bacon grease and put back in the oven.

Next, add the wet ingredients:

• 4T melted butter
• 2 eggs
• the right amount of whatever liquid you want

Here’s where it gets tricky, for two reasons. First, I never measure the liquid any longer. Second, the amount of liquid is highly dependent on the grind of the cornmeal.

You need to add enough liquid so that it’s a batter, not a dough. I usually aim for a viscosity that’s between pancake batter and cake batter. If your liquid is cold, you probably want to add it to the dry before the butter. But, again, it doesn’t matter that much.

Now take the skillet out of the oven and carefully pour the batter into the skillet. It should start sizzling and bubbling! The bacon grease should come up slightly over the top of the batter around the edges. Put it back in the oven and turn down to 350 w/convection.

Cook until it’s done. When’s that? Somewhere between 30 and 60 minutes, probably. It should be >180 degrees in the top center. A toothpick should come out clean (or with a few crumbs). It’s best eaten immediately, with butter!

Some baked products are very sensitive to following the recipe. Cornbread is not one of them. Once you have the basics down, you can do a lot of substitution. When I was home for Xmas 2019, we didn’t have enough cornmeal, so I used polenta. Worked fine.

If you look at my notes, you can see I experimented. Parts of my family are egg-, dairy-, and gluten-free, so I tried using masa harina instead of the flour for binding. Obviously, you can also use xanthan gum. But overall I found I preferred crumbly to gummy/dense.

When I make it now, I either use corn flour, or Authentic Foods GF Classical Blend from Gluten-Free Baking Classics by Annalise Roberts. However, if using it for stuffing, you will need the gum. Also, you need to accept that it’s still going to crumble!

Without eggs, wheat, or gum, it’s going to be very crumbly. We often eat it with chile verde, and just mush the cornbread into the chili! If you don’t use milk, you might not get as much browning. The more carbs in your non-dairy milk (i.e. almond over hemp), the more browning you’ll get.

I hope you’ve enjoyed this little excursion into skillet-cooked, no-sugar cornbread (also known as “true cornbread” or, simply, “cornbread”). If you have any questions about other substitutions or whatever, let me know!

View thread on Twitter

How about a pecan pie recipe?

— davechen (@davechen) March 29, 2020

View on Twitter

I last made a pecan pie was a couple of months ago. I was on a guys weekend when the subject came up. Somehow, it became clear that (similar to cornbread), non-southerners don’t understand that pecan pie refers to something very specific!

It’s fine if you want to put chocolate or bourbon or whatever into your pie, but even though you might call it “chocolate bourbon pecan pie,” it is not a type of “pecan pie” any more than “rice milk” is a type of “milk.”

Pecan pie has only the ingredients as listed on the bottle of Karo syrup. The other thing that my fellow weekenders were confused about was pronunciation, of Karo and pecan. First of all, the syrup. It’s KAY-rho. There is no alternative.

Pecan, however, has two accepted pronunciations, PEE-can and puh-KAHN. NC, where I went to school, prefers PEE-can. Georgia, where my parents are from, prefers puh-KAHN (it’s more subtle than that). But don’t ever, ever say puh-CAN or PEE-kahn, Northerner!

Our host, like me, has Georgia roots and went to school in NC (but at Duke, ugh), and even grew up in Georgia, so had all the ingredients for the pie laying around his vacation home, and I got to work!

Now, for all my big talk about the “one true way,” like most people, I only consider sacred what is important to me. As you saw, I make non-dairy, egg-free cornbread! And I don’t use the recipe on the side of the Karo bottle! My host was dumb-struck!

Instead, I use a recipe from Treasured Southern Family Recipes (note that I probably would go for the $2 used copies rather than the $1000 new one). But like me, my mom likes to tinker with recipes, so I use her modifications.

Wow. I just noticed the “6 to 8 servings.” This pie has about 3600 calories. I would suggest cutting smaller slices!

Even better than pecan pie, though, is chocolate chess pie. If you’re not from the south, you may have never heard of a “chess pie,” though technically pecan pie is a type of chess pie. A chess pie is a pie principally made from sugar, eggs, and butter.

In other words, a custard pie. Why is it called a “chess pie?” Nobody is really sure, but my favorite origin story is that some Yankee was offered pie and asked what kind it was. Well, it was a custard pie, but without any additions: just pie, or “jess pie” with the accent!

I got my chocolate chess pie recipe from Christ Church Cooks (again, not sure I would spring for the new version), which is a collection of recipes from the women of the church where my wife grew up (Christ Church Episcopal in Charlotte, NC).

Side bar: ahh yes, the South. You can see the women who contributed are cited with their husbands names. When we were first married my wife’s parents would send her mail addressed to Mrs. Andrew Certain. I miss a lot about the South, but not that.

As you can see, I’ve made a few notes. The problem with all custard pies is that the shell needs to be baked at a fairly high temperature so that it crisps up, but custard needs to be cooked at a fairly low temperature so that it doesn’t separate.

So I usually blind bake my crust for 25 minutes at 375 with weights and another 5 w/o to crisp it up. Then I finish in a combi-steam oven at 250 w/60% humidity. If you don’t have a combi-steam, you’ll probably need a higher temperature.

How long? Until it’s done: it should be 190 degrees in the top center, but mostly it should be completely set. So if you jiggle the pan, it should all move as one semi-solid mass. It shouldn’t seem more liquidy in the center. Probably 45 minutes or so?

Funny story - at Xmas last year I forgot to put aluminum foil over the crust before putting down the weights. These were little ceramic balls the size of peas. The crust came out looking like a golf ball! And the weights were very tedious to remove. Don’t do that!

Also, you’ll see that my past self encourages my future self to mix using the hand mixer. I’ve definitely not mixed well enough and ended up with a layer of unsweetened chocolate on top and a layer of sugary-eggy stuff on the bottom!

This pie is amazing. Like deep chocolate fudge. Also, it doesn’t really need to be a pie. You can just bake in ramekins, or whatever. I mean, the crust is nice and all, but really it’s just a vehicle for the amazing chocolaty goodness!

One thing that I didn’t write is that if you double the recipe, it’ll fit into a deep-dish 9-inch shell. It does expand while cooking, though, so don’t fill it all the way to the top.

You might be wondering why if these are both chess/custard pies, do I not blind bake the crust for the pecan pie? Mostly laziness. I’m sure it would be better, but for whatever reason (insulation from the pecans?), the soggy-crust problem isn’t quite as pronounced.

Also, you can see I’ve reduced the sugar. My general rule for desserts is to reduce sugar by ⅓ to ½. Why not with the pecan pie? Pecan pie is supposed to be almost overpoweringly sweet, like a praline. With chocolate chess pie, you want the chocolate to be front and center.

Next up, the last stop in our journey through Andrew’s southern favorites, barbecue.

View on Twitter

View on Twitter

You’ll get a lot more out of this post if you read the article first. Really: go read it. Even if it means you forget about this post and never come back.

What did I see of myself in that article? I saw both sides. Obviously, I’m a man, and I’ve reached the highest level in my industry; there’s literally no higher rung for me to attain. But it’s not quite that simple.

As I wrote here, I am the first Amazonian hired as an SDE2 to be promoted to DE. That’s four promotions, from level 5 to level 10 (there is no level 9 🤷). So when were those promotions?

• 5➝6: 2000
• 6➝7: 2003
• 7➝8: 2015
• 8➝10: 2020

Three years between the first two, five years between the last two, and twelve years in between. What happened?

Obviously, in 12 years, lots happens, but what resonated with the article were that I did the following:

• four months paternity leave for each of my kids
• over a year off to be at home when my wife went back to school
• a year of working ¾ time
• four months off when my wife had cancer (she’s fine)

I also coached my kids’ Ultimate teams six seasons and their football teams four.

So it’s not surprising that my career slowed down. I made a conscious decision to prioritize my family. I don’t regret that decision for a moment. I have relationships with my sons that I wouldn’t have had without that focus.

All that, and I was still able to make DE. So it’s a happy story, right? Yes and no. I feel very lucky to work for a company where I could walk this path and make choices aligned with my values. And I think it shows that it is possible to prioritize family and still rise.

But, and it’s a big but, I think I’m unusual. I was a white male whose talents and personality closely matched what Amazon values. And I really value what Amazon offers. So I was strongly committed to making it work, and I was very lucky to be who I am.

I think Amazon is better off for me being here. I just got a pretty strong signal that I’m right. But I easily could have left. Internally, I’m trying to use my influence to help keep the 10-year-younger versions of me engaged so that they can be VPs in 2030.

I don’t know how to impact the internal struggle women feel about trading off work and family. But I do know that anything we can do to make it easier to make difference choices at different points in time will make those choices easier.

If you’re in a position of influence (at Amazon or anywhere), I encourage you to think about the signals you’re sending to people. Are you thinking about the long-run, or just how much you can get out of them this month?

When people take time off for family, do you question their commitment? When they leave early for a child’s event, do you mentally write them off? As I keep telling my team: it’s a marathon, not a sprint.

Many of the most-impactful people at Amazon have been there for 20+ years. If as a leader you give up on people because they aren’t as committed to work as you are 100% of the time, you are going to lose those people long before they hit 20 years.

PS. If you work at Amazon and are struggling with these issues, please send me an internal email.

View on Twitter