2016          2024
Population > 18               29,027,447    32,600,713
Units available               15,507,364    16,786,020
Units/person                       0.534         0.515
Units needed to get to
2016 levels                           -       630,000

I.e. we need 630,000 houses or apartments just to get back to 2016 levels of housing availability. A bit over half of this number comes from Ontario; repeating the math for that province gives us a shortage of 333,000 units compared to 2016.

To be clear, this number misses a lot of nuances: we don’t know which of these “units” are 8-bedroom houses and which are micro-condos, and we don’t know which of these adults are single and which have families.

On the surface, this shortage of 630,000 units doesn’t seem too horrifying: in 2023, the construction industry has started building 240,267 new units (CMHC housing starts table) — down from the peak of 271,198 in 2021 — so it might look like it should take a bit less than three years to catch up.

But remember: Canada is a growing country, so as we’re building these housing units, new people over 18 are appearing. So let’s update our math. Suppose Canada’s growth follows StatCan’s moderate growth projections (M2 scenario in this table), and we continue building houses at the same rate as we did in 2021 (the best year we’ve seen so far). How long will it take to catch up to the 2016 housing level?

Well, then the projection looks something like this:

Year    Population > 18    Units short
2025        33,283,800         577,130
2026        33,548,800         495,079
2027        33,822,800         418,454
2028        34,312,900         457,893
2029        34,800,100         496,398
2030        35,280,800         532,044
2031        35,761,400         568,248
2032        36,241,000         604,527
2033        36,715,500         638,690
2034        37,187,400         672,071
2035        37,652,100         702,210

Uh-oh, we never catch up.

So what will it take?

I’ll spare you the math and go straight to the answer: if we want to get back to 2016 housing stock levels by 2035, then over the next five years we need to increase the housing starts by 26% over the 2021 levels to about 380,000 starts/year, with the bulk of this increase happening in Ontario, which needs to raise its housing starts by ~45%.

It’s not a small amount, but not impossible either. For comparison, in the years between 2016 and 2021, housing starts have increased 37% Canada-wide ans 33% in Ontario.

But there are a few problems in the way

A lot of the discourse online and among activists has been focused on onerous regulations, slow approvals, and unhelpful building codes — and rightfully so. That said, these seem to be improving as the public mood shifts towards building more housing, and, looking at the industry reports, it seems like at least some of the insiders no longer see the public policy as the biggest long-term drag on construction.

There are a couple of other major problems though.

First, we need the new housing starts to go up, but instead since 2021, they’ve been going down. The explanation for this is fairly straightforward: less money has been spent on building new houses because the interest rates have gone up and house purchases have slowed down.

This problem will partly fix itself as the Bank of Canada lowers its policy interest rate. However, it won’t cut the rate back down to 0.25%, and the total investment into new housing isn’t projected to go back up to the 2021 levels either (source: BuildForce National Summary for 2024):

But we need the investment to go much higher: if the average historic costs per housing unit stay the same, we’ll need to be investing as much as $54 billion per year more than we’re forecast to catch up to our 2016 housing availability target. If that money doesn’t materialize from somewhere, no amount of regulation reform will help plug the gap.

The second problem is the shortage of skilled labor in the construction industry, which is projected to worsen due to retirement. In the short-term, construction is limited by lack of people with a few specific skill sets like crane operators, elevator mechanics, or ironworkers, especially in major cities like Toronto (source: BuildForce Ontario report 2024):

In the long-term, from the same report, based on the current hiring levels and trades education pipeline, Ontario alone is expected to be short 35,500 construction workers by 2033 — and that’s assuming BuildForce’s own not-quite-adequate construction investment forecast seen in the earlier graph. To get back to 2016 affordability levels in a reasonable time, we’d need to find more skilled construction workers somewhere, either by attracting more young people to trades locally, or through immigration.

Or… what if we lower the population growth?

While we work on addressing the supply-side issues of housing, there’s one lever in the hands of the federal government that can easily control the demand side: immigration levels.

How will the changes to the immigration policy affect the outcomes?

Let’s tweak some numbers and redo the math:

• StatCan’s M2 growth projection scenario already assumes a net departure of around 212k non-permanent residents per year between 2025 and 2027. That’s roughly consistent with reducing foreign students to about 2019 numbers and losing about half of the temporary foreign worker permit holders. Let’s keep this as-is.

• The current permanent resident admission target is 500k/year. Let’s lower it to 350k until 2035, which will still keep it mostly above the historic levels and at around 0.8% of the current population.

Let’s keep the construction industry output at their 2021 levels and see what the situation looks like:

Year    Population > 18    Units short
2025        33,283,800         668,723
2026        33,548,800         521,036
2027        33,822,800         382,890
2028        34,312,900         250,171
2029        34,800,100         233,516
2030        35,280,800         215,927
2031        35,761,400         195,478
2032        36,241,000         175,588
2033        36,715,500         155,773
2034        37,187,400         111,129
2035        37,652,100          85,173

We gradually catch up! Not quickly, but the effects should start being felt after the first year, and should improve every year after that. And if the construction industry raises its output by just 3%/year (for a total of 12% in 5 years), we get to 2016 housing availability levels by 2031.

Conclusions

It took a while to get into the current housing crisis, and it will likely take a while to get out. Whoever ends up taking on this will have their work cut out for them.

We’ll need:

• The federal government doing its part in controlling the demand by managing the immigration levels.

• Cities and provinces, especially Ontario and BC, to keep pushing for regulatory reforms;

• Provinces continuing to expand capacity for trades education and making it more attractive for young people;

• The construction industry finding ways to build better houses and apartments for less;

• Figure out how to keep funding flowing into the construction industry;

Whatever happens, I will keep following this with great interest.

But I can’t play like that. First, I’m lazy, and second, my slow human brain can’t do information theory calculations or depth-first searches.

Instead, every time I play Wordle, I enter the same sequence of words until it feels like there aren’t many solutions left, at which point I try to make more tailored guesses.

Here are a couple examples of my recent games:

Until today, my go-to words were STALE, CHIRP, MOUND, and GAWKY, picked mostly by gut feel while trying to check as many frequently used letters as possible. So far I’ve been doing alright with them.

But can we do better?

Yes! Let’s use math and programming to do better.

Though to do computer science, we’ll need a more precise algorithm than “play these words until it feels like there aren’t many solutions left”. Let’s go with this:

1. Pick four words.

2. In every game, keep guessing these words in order.

3. If there are 3 or fewer possible solutions left, make tailored guesses that minimize the number of turns to victory.

Oh, and let’s use only the list of 2309 valid solution words (heplfully provided in Wordle’s JavaScript source code), and ignore the larger list of valid guesses. In part, that’s to make the calculations simpler, and in part because you’ll forget that THEWY is an acceptable guess by the time you’ll finish reading this sentence.

Pick words that produce fewest remaining possible solutions?

That was the first thing I tried.

For the first word, I picked one that resulted in the lowest remaining information entropy (basically, fewest remaining options) on average across any of the 2309 valid solutions. That turned out to be RAISE.

Then, I picked the next word that gave me (on average, across all solutions) the most information after guessing RAISE. That turned out to be CLOTH. And so on, getting DUMPY and BEGAN as the best third and fourth words.

But how does it compare to the original STALE>CHIRP>MOUND>GAWKY? If we simulate all 2309 possible games we see that:

• RAISE/CLOTH/DUMPY/BEGAN wins all games and uses 3.89 guesses on average.

• My original STALE/CHIRP/MOUND/GAWKY wins all games and uses 3.84 (!) guesses on average.

😲 ⁉️ My original gut-feel word choices did better?

Because each word being optimal ≠ the full set is optimal

So it turns out it’s not enough to optimize each word; instead, we have to jointly optimize the entire sequence WORD1 WORD2 WORD3 WORD4 across all possible word sequences.

But that’s a lot of sequences! I’m not sure how much exactly 2309⁴ is, but it’s definitely more than my computer can handle. Even doing some intelligent pruning (e.g. ‘a letter must not repeat in the first 3 words’) still leaves a lot of options to evaluate. What else can we do to cut it down?

What I ended up doing was ranking all words by the number of bits of information that they provide, picking the top 10-100 words at each position for a total of ~160,000 four-word combinations, and then simulating all possible games for each sequence.

After two days at 100% CPU, my computer gave me the answer:

• SLANT PRICE DOUGH BALMY: 3.79991 average guesses to win.

But there were a lot of other sequences that produced similar results. For example #2 sequence was CRANE SPILT DOUGH BALMY that took on average 3.8008 guesses to win.

But what if I’m a smart and tailor my guesses sooner?

Or, to be less flippant, you can often get into situations where there are more than three guesses left, but you can easily enumerate them. The simplest example is if your first guess SHARE results in something like

There are lots of great videos and articles on the optimal Wordle strategy. I particularly enjoyed this YouTube video by 3Blue1Brown on using the information theory to come up with the best guesses.

But I can’t play like that. First, I’m lazy, and second, my slow human brain can’t do information theory calculations or depth-first searches.

Instead, every time I play Wordle, I enter the same sequence of words until it feels like there aren’t many solutions left, at which point I try to make more tailored guesses.

Here are a couple examples of my recent games:

Until today, my go-to words were STALE, CHIRP, MOUND, and GAWKY, picked mostly by gut feel while trying to check as many frequently used letters as possible. So far I’ve been doing alright with them.

But can we do better?

Yes! Let’s use math and programming to do better.

Though to do computer science, we’ll need a more precise algorithm than “play these words until it feels like there aren’t many solutions left”. Let’s go with this:

1. Pick four words.

2. In every game, keep guessing these words in order.

3. If there are 3 or fewer possible solutions left, make tailored guesses that minimize the number of turns to victory.

Oh, and let’s use only the list of 2309 valid solution words (heplfully provided in Wordle’s JavaScript source code), and ignore the larger list of valid guesses. In part, that’s to make the calculations simpler, and in part because you’ll forget that THEWY is an acceptable guess by the time you’ll finish reading this sentence.

Pick words that produce fewest remaining possible solutions?

That was the first thing I tried.

For the first word, I picked one that resulted in the lowest remaining information entropy (basically, fewest remaining options) on average across any of the 2309 valid solutions. That turned out to be RAISE.

Then, I picked the next word that gave me (on average, across all solutions) the most information after guessing RAISE. That turned out to be CLOTH. And so on, getting DUMPY and BEGAN as the best third and fourth words.

(Along the way, I had to keep in mind that I had to switch to “smart” guesses if I had 3 or fewer valid solutions left.)

Here’s what each word contributed:

Guess Ave. remaining entropy (bits) Ave. options remaining Worst case options remaining RAISE 5.2 17.5 167 CLOTH 1.8 2.38 30 DUMPY 0.8 1.53 6 BEGAN 0.4 1.28 4

But how does it compare to the original STALE>CHIRP>MOUND>GAWKY? If we simulate all 2309 possible games we see that:

• RAISE CLOTH DUMPY BEGAN wins all games and uses 3.89 guesses on average.

• My original STALE CHIRP MOUND GAWKY wins all games and uses 3.84 (!) guesses on average.

😲 ⁉️ My original gut-feel word choices did better?

Because each word being optimal ≠ the full set is optimal

So it turns out it’s not enough to optimize each word; instead, we have to jointly optimize the entire sequence WORD1 WORD2 WORD3 WORD4 across all possible word sequences.

But that’s a lot of sequences! I’m not sure how much exactly 2309⁴ is, but it’s definitely more than my computer can handle. Even doing some intelligent pruning (e.g. ‘a letter must not repeat in the first 3 words’) still leaves a lot of options to evaluate. What else can we do to cut it down?

What I ended up doing was ranking all words by the number of bits of information that they provide, picking the top 10-100 words at each position for a total of ~160,000 four-word combinations, and then simulating all possible games for each sequence.

After two days at 100% CPU, my computer gave me the answer:

• SLANT PRICE DOUGH BALMY: 3.79991 average guesses to win.

But there were a lot of other sequences that produced similar results. For example #2 sequence was CRANE SPILT DOUGH BALMY that took on average 3.8008 guesses to win.

But what if I’m a smart and tailor my guesses sooner?

Or, to be less flippant, you can often get into situations where there are more than three guesses left, but you can easily enumerate them. The simplest example is if your first guess SHARE results in something like

So let’s say that we start making tailored guesses when there are five possible solutions left instead of three. How does this change our optimal sequence?

After another day at 100% CPU, I ended up with TRICE SALON DUMPY GLOBE, which solved the puzzle in 3.72 steps on average.

But SLANT PRICE DOUGH BALMY wasn’t far behind, at 3.74 average guesses, and even the original STALE CHIRP MOUND GAWKY clocked in at respectable 3.79 guesses.

But what if the word list changes?

That’s a good question. It wouldn’t do to have a set of words that’s hyper-optimized for Wordle but not other similar puzzles.

So I took the list of words ranked by their occurrence in Google Books, and selected top 3000 five-letter words.

Redoing the same analysis here gives us a new winner: SAINT BORED CLUMP FIGHT at 3.91 guesses to win, though again, SLANT PRICE DOUGH BALMY is not far behind with 3.96 guesses (athough with 99.93% win rate), and even my original STALE CHIRP MOUND GAWKY comes in at a not-terrible 3.98 average guesses and 99.90% win rate.

So what’s the lesson?

So it turns out that the most optimal sequence really depends on how you play the game and which Worlde-like game you play.

But if you want a good-enough sequence, it looks like you can pick any three words that cover the most frequently used letters, and then a fourth word that complements the other three well.

Personally, I think I’ll try SLANT PRICE DOUGH BALMY for the next while.

So let’s say that we start making tailored guesses when there are five possible solutions left instead of three. How does this change our optimal sequence?

After another day at 100% CPU, I ended up with TRICE SALON DUMPY GLOBE, which solved the puzzle in 3.72 steps on average.

But SLANT PRICE DOUGH BALMY wasn’t far behind, at 3.74 average guesses, and even the original STALE CHIRP MOUND GAWKY clocked in at respectable 3.79 guesses.

But what if the word list changes?

That’s a good question. It wouldn’t do to have a set of words that’s hyper-optimized for Wordle but not other similar puzzles.

So I took the list of words ranked by their occurrence in Google Books, and selected top 3000 five-letter words.

Redoing the same analysis here gives us a new winner: SAINT BORED CLUMP FIGHT at 3.91 guesses to win, though again, SLANT PRICE DOUGH BALMY is not far behind with 3.96 guesses (athough with 99.93% win rate), and even my original STALE CHIRP MOUND GAWKY comes in at a not-terrible 3.98 average guesses and 99.90% win rate.

So what’s the lesson?

So it turns out that the most optimal sequence really depends on how you play the game and which Worlde-like game you play.

But if you want a good-enough sequence, it looks like you can pick any three words that cover the most frequently used letters, and then a fourth word that complements the other three well.

Personally, I think I’ll try SLANT PRICE DOUGH BALMY for the next while.

So here are some of the less obvious REST API design lessons that I’ve learned over the years.

1. Do not nest objects in URL paths

Suppose you have task objects that always belong to project objects. What would should be the URL to get information about a task? Should it be this:

GET /api/projects/{projectId}/tasks/{taskId}

or this?

GET /api/tasks/{taskId}

Having gone both ways, I now confidently think that the bottom choice is the best one. If you have a taskId, then projectId is redundant, and by including projectId in the URL you’re making the caller look up a piece of information that they may not have or need otherwise. And what happens when the caller provides the wrong projectId? Should you return 404 Not Found even though you can unambiguously locate the task? You’ll also have to add extra validation code to check that the task is really in the right project—something that you wouldn’t need to do with the shorter version of the URL.

You may ask: “But how do you add a task to a project then?”

Well, how about this?

POST /api/tasks
...
{
    “projectId”: 1234567,
    ...
}

You may then ask, “But how do you look up tasks for a project if you can’t do GET /api/project/{id}/tasks?”

Like this:

GET /api/tasks?projectId=<projectId>

As a bonus, you get an endpoint to list all the tasks irrespective of which project they’re in. And this approach treats projectId as one of multiple optional filters that may be applied on /api/tasks, alongside maybe status, assignedTo, or any others.

2. Of the 2xx HTTP status codes, you only need 200 OK

This one wasn’t obvious to me until a coworker recently pointed out that he’s never seen a case where the caller had to make a decision based on the type of 2xx return code they got back.

And indeed, although RFC 9116 — the current HTTP spec — defines seven different 2xx success codes, in practice, the client-side code almost never needs to make a decision based on whether it receives e.g. 200 OK or 201 Created. Alright, in very rare cases it’s convenient when the server returns 204 No Content when you’re debugging and trying whether the request is really supposed to be missing a body. But you can accomplish the same thing with Content-Length header, or by some other means.

I have, however, seen developers spend hours going back and forth on which 2xx code should be used for a specific endpoint. So to save everyone’s time, let’s just use 200 OK and forget about the rest.

3. Encode timestamps in a human-readable format

If you’re sending or receiving timestamps, it may be tempting to use something like the Unix epoch time in milliseconds (the number of milliseconds since 00:00 UTC on Jan 1, 1970). It’s not a bad choice, and it’s quite convenient to work with.

Except it’s not human readable. Something like ISO 8601 (e.g. 2023‐07‐12T23:57:23Z) is easier to read for a human, and you can even drop the colons (2023‐07‐12T235723Z) to make it URL-compatible.

It may take a bit extra work to configure your framework to emit times in the right format, but it will make using (and debugging) the API a lot easier.

By the way, don’t forget to include time zone in your timestamps, even if it’s just a Z at the end.

4. Pagination: avoid offsets and provide a link to the next page

There are broadly two ways to do pagination:

• Page number and page size: e.g. https://.../things?page=10&pageSize=20. This will work alright up to maybe page 10,000 or so, at which point, if the server is fetching the results from a database, you’ll start running into a quirk with OFFSET that can substantially degrade performance. So while this pagination approach is convenient when the results go into a paginated UI, it’s a bad idea to use it for an endpoint that will be called by export jobs or by anything else that will ask for a large number of pages with high offsets. Instead, it’s much better to use…

• WHERE-based pagination: where the caller provides something akin to a where clause, like https://.../things?fromId=12345678. If this argument corresponds to a DB index, the response times should stay constant-ish as fromId argument increases, unlike OFFSET whose response times tend to increase linearly.

Oh, and if anyone will be iterating over multiple pages of responses, it really helps to include the URL of the next page in the response body:

{
    "data": [ ... ],
    "metadata": {
        "nextPage": "https://..."
    }
}

That’s something that I’ve been curious about for a while now, so I’ve read a bunch of papers on the subject.

Metal electrodes

Surprisingly, off-the-shelf 44 AWG insulated copper wire is not a completely terrible choice. Its 51μm diameter (plus insulation) is a bit on the high end for a typical intracortical electrode, but it’s about the right order of magnitude. If you stick it in, for a short time it has a decent chance of picking up electric spikes that nearby neurons send when they get excited.

But it’s still a terrible choice. First, the copper will quickly oxidize and corrode in the brain’s electrolyte-rich fluid. And second, copper is toxic to the neural tissue. So don’t put copper electrodes into anyone you love.

Okay, let’s try something else. How about an off-the-shelf stainless steel 40μm wire? These are actually a legit choice, once you figure out how to insulate and sharpen them. In 1958, at UCLA, Felix Strumwasser sharpened 80μm stainless steel wires with hydrochloric acid and used them to record spikes in neurons of ground squirrels for over a week. The wires were likely insulated by dipping them into lacquer, as described by Green, also from UCLA, later the same year.

However, the most popular electrode metal these days seems to be tungsten or platinum. Though noticeably pricier, it seems that these give you a better signal-to-noise ratio without messing with the brain chemistry.

So let’s say we get some platinum or wire, coat it with tissue-friendly parylene C for insulation, and then expose and sharpen its tip. Are we good?

Not quite.

Multiple electrodes

According to Techniques for Large-Scale Multiunit Recording, inserting one electrode by itself is often not that useful. First, in addition to neural signals, the electrode will also pick up the background noise, including the AC power lines in the walls. Second, we often want to collect spikes from multiple sites to observe the behavior of more neurons.

The simplest way to increase the number of electrodes is to twist a few of them together, insert them, and measure the difference in voltage between some of the pairs. Taking the difference in voltage between the electrodes helps not only remove the background noise but also isolate individual neurons because the electrodes closer to the spiking neuron will register higher voltage swings.

The next step up is electrode arrays, which help measure signals over larger areas or a range of depths. Here, silicon probes became popular, like the widely used Utah array with as many as 100 needle-like electrodes, or a Neuropixels probe, which has a single shank with 960 electrodes.

Utah array in particular featured in a number of really cool experiments, like this BrainGate experiment where researchers implanted Utah arrays into a tetraplegic individual and connected them to electrodes implanted in the hand muscles, allowing the subject to move his hands.

However, it turns out that silicon probes have their own problems, and aren’t the best thing to insert into the brain either.

Biocompatibility

So what’s the problem with the silicon probes?

Well, a number of studies have shown that over the months after the implantation, the signal from the electrodes weakens, and some of the previously working electrodes go silent.

And metal electrodes aren’t better either. For example, this study tracked the performance of 50μm tungsten electrodes over 9 months, and found that within the first few months, many electrodes could no longer detect spikes and the few that could, had much weaker or unstable signals. In some individual rats (who got the arrays implanted), by the end of the experiment, zero out of sixteen electrodes could detect useful signals

To be clear, all of these studies showed that in almost all cases it was still possible to get useful signals at the end of the experiment, just that over time the signal quality got noticeably worse.

So can we do better?

A lot of studies have dug into why the signal quality decreases. This recent review (section 2) summarizes the causes: (1) the electrodes create small stab wounds on the way in, damaging the tissue and the blood vessels; (2) electrode bodies cause chronic inflammation that results in a scar that surrounds the probe and reduces the signal quality; (3) all of the above cause nearby neurons to die or migrate away from the insertion site.

This can be made worse by the rigid bodies of the probes. Mammalian brains naturally move inside the skull, swelling with each pulse of blood from the heart, and rigid metal and silicon electrodes cannot deform at the same rate as the brain tissue.

To top it off, poorly chosen insulation can crack or detach from the probe, letting carefully collected signals leak back into the surrounding tissue.

Smaller, softer electrodes

Various recent reviews, such as this one and this one, suggest that the best electrodes would be:

• Made of very thin wires, comparable in size to neuron’s appendages (axons and dendrites) - that’s <10μm across.

• Soft, bending as easily as cortical tissue.

• Covered in well-attached insulation that won’t agitate neurons or other cells in the brain.

• “Floating”, that is, loosely attached to the skull or signal processing circuitry, allowing the electrodes to move as the brain moves.

Over the last decade, a number of electrodes satisfying these conditions started appearing:

• NeuroRoots, made of 100nm-thick platinum wires encapsulated in parylene-C insulation; total cross-section: 1.5μm x 7μm.

• Elon Musk-sponsored Neuralink, which uses insulated gold “threads” that are 4-6µm-thick and 5-50µm wide, depending on the number of electrodes per thread.

• Neurotassels uses 100nm-thick gold electrodes wrapped in polyimide insulation; total cross-section: 3µm x 12µm.

• Ultra-soft conducting polymer microelectrodes, though at 125µm these were much thicker than others on this list.

• Carbon fiber electrodes around 9-10µm in diameter.

• And a variety of other wires, meshes, and probes.

Most of these came with a study showing how much they outperform the previous generation of electrodes, and how little damage they do in a rat brain. In the case of something like NeuroRoots, it looks pretty impressive on paper: after 7 weeks, all 32 of the 32 implanted electrodes could pick up signals nearly as well as on the day they were implanted. In the case of the conducting polymer above, the study compared it to much thicker tungsten electrodes, making the comparison somewhat unfair.

Conclusions

So basically, yes, you would need a brain-grade wire to collect signals from neurons, and we’re still working out the specifics of what makes the wire “brain-grade”.

With the progress above made in the 2010s, I’m looking forward to seeing what the 2020s will bring.

The FPGAs are certainly… different from any other kind of programming, and it took a bit of effort and troubleshooting to get my FPGA calculator working. If you’re interested in getting into FPGAs, here are a few tips below.

1. The basics

Programming FPGAs is different from programming in JavaScript, C#, or C. There are no levels of abstraction between you and the hardware, so it’s essential to learn the foundations of digital logic.

For me, The Art of Electronics worked very well — specifically, chapters 10-14.

The book first introduces the basic logic circuit elements, and then builds up to other components and concepts: flip-flops, clocking, different logic families, logic interfacing, three-state outputs, busses, etc.

Almost all of it was useful when I got to programming the FPGA.

If textbooks are not your thing, there are lots of YouTube videos covering the same subjects.

2. Verilog

There are three main hardware definition languages: Verilog, VHDL, and SystemVerilog. Either one is a valid choice, though I found that for Digilent and Xilinx (my choice of FPGA boards and chips) there were more Verilog examples, so I went with that.

Unfortunately, Verilog just doesn’t have as much great documentation as, say, TypeScript or C#. The best I could find were these two websites:

• Chipverify Verilog Tutorial

• This Verilog guide on readthedocs.io.

I’ve read these before I started programming, and referred back to them a number of times afterwards.

3. The board

You can try buying an FPGA chip and soldering it to a board yourself, but it’s much easier to get a starter board that already includes all the necessary components such as configuration flash memory, USB interface, and voltage regulator.

For starter boards, I liked Digilent boards the most and went with Cmod A7 that had a Xilinx Artix-7 chip on it. It was powerful enough to fit the projects I wanted to build with room to spare, and it could plug into a breadboard.

Alternatively, if you’re looking to spend less, Cmod S7 seems like a decent choice, and if you’re looking for more power, there’s something like Arty A7.

4. Making the LED blink

Now we have everything we need to take FPGA for a test drive.

If you’ve gone with a Digilent board, the next step that worked for me was to go through the Getting started with Vivado guide from Digilent.

Be aware that there are a few quirks that you may run into:

• The LED port name in the module definition must exactly match the port name in the constraints file. For example, it can’t be led[0] in the constraints file and led in the module definitions - you’ll get an error when generating the bitstream.

• Xilinx wants you to create an account before using Vivado, and they want you to enter something that looks like a real address when doing so. They’ll check the address, but not too thoroughly. You’ll need to log into the account when starting Vivado.

5. Now build your first project.

From here, build whatever you were thinking of.

If you’re looking for an idea for a simple project before moving on to something more interesting, I found a calculator to have just the right amount of complexity:

From here on, the things that helped me the most were:

• Carefully reading the documentation and the guides, as they often contain the answers you’re looking for. For example, each Digilent board has a detailed reference manual that explains how to work with each component that’s on the board.

• The chip manufacturer has detailed guides as well. For example, if you’re looking to change the clock frequency on your FPGA chip, or want to work with multiple clock frequencies, Xilinx has a very thorough guide on working with Mixed Mode Clock Managers.

• If you run into a problem, keep simplifying your logic until it starts working. For example, when I was having trouble sending data over the USB/UART link, I ended up removing all of my logic and just tying the RXD (send) wire to TXD (receive) wire to make sure that I could echo the characters through it using the COM terminal. Turns out I couldn’t because I mixed up which wire was the input and which was the output. The rest of my logic was correct.

• If you get stuck, like everywhere else, use Google. Someone has run into a similar problem. If you don’t find the exact solution, at least you may get an idea that you can try next.

6. This will save you days

I was going to lump this with the tips above, but this deserves its own section.

Put

`default_nettype none

at the beginning of every Verilog file (yes, including the backtick), and

`default_nettype wire

at the end.

This is because by default, if you make a typo in a variable name, Verilog will assume that you’ve defined a new variable of type wire. Because of this, I literally spent a day debugging a problem that was caused by a typo.

If you put `default_type none, Verilog will complain that it doesn’t know the variable type and will bring your attention to the typo.

The closing `default_nettype wire is necessary because apparently some of Xilinx’s own modules (called “IP” in FPGA terminology) rely on implicitly defined variable types.

So when we see something in front of us, like a pencil, how exactly does the brain understand that yep, that’s a pencil?

Like, what is the detailed sequence of steps that happens? The eyes presumably convert the light to neural signals (how?), then these get sent to the brain (where?), and then the brain does… what?

I’ve been reading through a bunch of neuroscience textbooks and papers for the past six months, and I think I’m at the point where I’ve now got the whole picture.

So here are the core pieces of this picture, with a lot of details omitted to fit it into a short blog post.

Let’s start with the input.

What exactly do the eyes do?

The main output of the eyes are the neurons called ganglion cells that live in the thin layer of tissue at the back of the eye. To grossly oversimplify, these ganglion cells transmit the brightness of each pixel of the picture that’s projected onto the back of the eye, though the reality is much more complex.

To transmit this information, the neurons use spikes of voltage – that’s when the voltage between the inside and the outside of the cell suddenly rises from somewhere around -60 mV to around 0 mV and falls back down over the course of about a millisecond.

In the eyes, these spikes are caused by light-sensitive proteins called opsins that live in cells of another type, the photoreceptors. When photons hit the photoreceptor cells, the opsins in these cells absorb the photons and slightly change their structure, which causes a rapid cascade of chemical reactions across a few cells. At the end of this series of reactions, the ganglion cells let in some positively charged ions, raise their internal voltage, and start a spike.

(I’ve written more about how the ions cause spikes in another post)

The whole system is set up to handle different volumes of light: the more light is shining on the photoreceptors, the more spikes are produced by the ganglion cells.

So, to grossly oversimplify even more, more light => more spikes in the ganglion cells.

And now what?

Where do the signals go?

The ganglion cells live at the back of the eyes, but they have long output extensions (axons) that run from the eye, through the optic nerve, and right into the center of the brain. That’s where the thalamus – the brain’s main switchboard – is located.

In an adult human, that’s about 10 cm or 4 in of length, and that’s far from the record for the length of a neuron. The spikes travel along the full length.

One of the interesting things that happen along the way is that the optic nerves cross at about the half-way mark, and the neurons from the left half of each eye end up traveling to the left half of the brain, while the neurons from the right half of each eye go to the right.

This way, each hemisphere of the brain will be able to construct a stereo picture of each half of the visual field.

From the thalamus, a new set of neurons picks up the same information and forwards it to the back of the brain - the occipital lobe - where the brain will do its magic.

And the brain does… what?

In the brain, visual processing is done through a sequence of visual cortex areas - V1, V2, V3, etc, each of which extracts progressively more complex features from the original signal.

The first one of these, V1, is located right at the back of the brain; it represents nearly pixel-for-pixel what the eye sees - that is, the neighboring neurons in V1 represent neighboring points on the image.

In V1, a series of neural connections detect basic features like edges, colors, directions, or motion in each patch of the visual field. If the neurons responsible for detecting a specific feature detect it, they fire a series of spikes, letting the downstream areas know what they found.

The downstream areas - V2, V3, etc. - extract progressively more and more complex features over larger and larger areas.

If you’re familiar with Convolutional Neural Networks (if you’re not, here’s a good explainer), that’s pretty much exactly what’s going on here.

Early on, the processing path splits into two branches: the bottom one that’s focused on recognizing objects, and the top one that’s focused on building a mental picture of space and motion.

At the end of the bottom branch are the areas that recognize faces, places, or objects, including the pencil that we started with.

Here, if the final collection of features extracted along the way add up to a specific known object, a collection of neurons that are associated with that object fires spikes to other areas of the brain, triggering appropriate emotions, thoughts, memories, and other associations that we may need at the moment.

What’s interesting about it are both its simple wiring (at first glance), and the extensive hidden support circuitry that can tell us more about how the brain works.

VOR is not the only circuit that helps us keep our eyes locked on target; there are others that work in parallel. I suspect that they probably use the same underlying principles.

(Note: some links here are behind a paywall. Remember that it’s illegal to go to Sci-Hub and search for the DOI link of the article to get free access to it.)

So what’s Vestibulo-Ocular Reflex?

Pick a point on the screen, and while looking at it, turn your head left to right. Your head will move, but your eyes will stay locked onto that point.

For that to happen, the muscles that control your eyeballs need to change their tension at exactly the right time and by exactly the right amount to compensate for the movement of the head.

While at first glance that may seem tricky, in reality, the brain deals with it the same way as the smartphone: it just uses our internal accelerometer, the vestibular apparatus, pictured below in the red box.

From accelerometer to muscles

From the vestibular organ, there’s a simple three-neuron-long connection that feeds the accelerometer signal to the muscles that move the eyes, first documented in the 1930s. We can follow it to see what it does.

The neurons coming out of the vestibular organ output a steady stream of electric spikes. When the head turns left, the left vestibular neuron starts producing more electric spikes, and the right one - less. When the head stops moving, both vestibular neurons go back to the equal rate of spikes. If the head turns right, the opposite happens.

The signals from the vestibular neurons end up in the brainstem, the more basic part of the brain that sits between the cerebral cortex and the spine. There, another set of neurons directly connect them to the pools of motor neurons that move the eye muscles.

The muscle neurons, in turn, work according to rules we won’t go into right now, but it’s safe to simplify and say that the more excited they get the more the eye muscles contract.

So basically, when the head turns left, the increased rate of spikes from the left vestibular neuron is directly fed into the motorneurons that turn the eyes right.

End-to-end, this circuit is very fast. Experiments show that once the head starts turning, it takes less than 10 ms for the muscles to engage.

(Btw, Wikipedia has a different wiring diagram; I’ve seen it drawn both ways in papers. In practice, the result is the same either way.)

Complication #1: the neural integrator

So, on the surface, it seems that we’re done. The acceleration output feeds directly into the motorneurons, the eyes move, and we’re happy.

But here’s the problem: when the head is done turning, the vestibular organs go back to signaling at the base rate. Then the tension in the eye muscles goes back to equal, pointing the eyes straight. That’s not what we were hoping to see.

To fix that, we need to add a neural integrator to the picture.

The neural integrator does this: (1) it collects the total velocity signal sent to the eye muscles, and (2) outputs the spikes at exactly the right rate to keep the eyes pointed in the right direction.

I.e. it literally integrates the angular velocity to come up with the final position.

Our best guess that the integrator works using some clever mutually inhibitory feedback. If the left vestibular neuron sends more spikes, the right half of the neural integrator gets more excited, and ends up inhibiting the left half more, which ends up inhibiting the right half less. This entry in the Encyclopedia of Neuroscience describes the math in more detail.

Some papers indicate that there may be a number of different neurons involved, and this may be a multi-stage process.

(On a tangent: there isn’t just one neural integrator. There seem to be a number of them in the brain; for example, if you close your eyes and move around the room, a part of the cerebral cortex will keep an estimate of your position from your motion, though it won’t be very accurate. One of the papers linked above even argues that accumulating the evidence for two alternatives in a decision process could also be considered an integrator.)

Complication #2: controlling the gain

Setting the neural integrator aside, here’s a different question: why is it that a 5° turn of the head causes a 5° turn of the eyes? What makes it not be 4°, or 10°? Are we just lucky that the neural connections work out to be exactly right?

Turns out, there’s a circuit that runs through cerebellum that keeps the gain (ratio of head turn to eye turn) at exactly the right value. It’s a pretty clever circuit that turns out to be so useful that the brain has copy-pasted it thousands of times (with a few changes) and reused it in nearly all motor functions, plus a number of non-motor functions.

The center of this circuit are the inhibitory Purkinje cells, which, in this case, output a low, constant stream of spikes that lightly inhibit the connections from the vestibular neurons to the motorneurons.

The vestibular signal comes in through excitatory parallel fibers that make connections with Purkinje cells. When the head turns, these fibers make the Purkinje cell a bit more excited, making it inhibit the VOR circuit a bit more, kinda like tapping on the break.

Most of the time, this inhibition is just right to make the 5° head turn result in 5° eye turn. But what if it’s wrong?

This is where another set of neurons comes in - the climbing fibers. When the brain detects that the image on the retina lags behind motion (i.e. the inhibition is too strong), these fire. This sets the Purkinje cell ringing like a bell for a bit, and it turns off some of the connections from the parallel fibers, reducing the inhibition, and letting the eyes move faster.

I haven’t seen a good description for how the opposite error is fixed (eyes move too fast), but from what I remember from other circuits, the parallel-to-Purkinje connections gradually turn back on when the parallel fibers keep firing, so the error may just fix itself over time.

That’s it

That’s basically all I know about the Vestibulo-Occular Reflex so far. As I mentioned, there are other mechanisms to keep the eyes steady, like the neck reflexes, or the slower and finer Opto-Kinetic Reflex that uses the information from the retina instead of the vestibular organ.

There’s also a YouTube version of this article.

The picture above shows some (far from everything!) of what we know about the cells and neural circuits in the human retina.

What’s interesting is - the retina is not just a light detection organ. It’s a complex electro-chemical calculator with three neural layers, and it uses these layers to extract a number of useful signals from the light that hits our eyes.

Below, we’ll step through some of the more interesting features of the retina, but if you’d like to learn more, the above picture (and this article) are mostly based on this review article published in Feb 2020 in the journal Progress in Retinal and Eye Research, and on chapter 17 of the latest edition of Ryan’s Retina.

Photoreceptors are always on - except when there’s light

The photoreceptor cells in the vertebrate retina - rods and cones - are at the very back of the eye, behind the other layers of neurons.

Cones work best in bright light and can detect colors. In humans, there are three types of cone cells, each specializing in detecting red, green, and blue light. Rods work best in the dark, but can’t detect colors - just the brightness level.

Since the photoreceptors are at the very back of the retina, light has to navigate a tangled maze of neuronal bodies and wires to get to them. It does so using something called Müller cells (not shown), that stretch across the entire thickness of the retina and act like fiberoptic cables.

When the light gets to the photoreceptors, it turns them down. In complete darkness, the photoreceptors continuously emit neurotransmitters (“fire”, or “are excited”) at their maximum rate. The light inhibits them.

Parallel channels for brightness and darkness levels

For red and green cones, right from the start, the signal splits into two separate channels: ON cells, which fire neurotransmitters when the light is shining on their cone, and OFF cells that fire when the light is not shining on their cone.

Since the cones fire most when it’s dark, we can say that they produce the OFF (darkness) signal. So how do the ON cells fire when there’s light? Well, the ON bipolar cells are intrinsically excited and want to fire neurotransmitters, but they get inhibited by the photoreceptors. So as long as the photoreceptors are firing, ON bipolar cells aren’t. When there’s light (and photoreceptors stop firing), ON bipolar cells are no longer inhibited, and produce more neurotransmitters.

Why are there two channels instead of just one? Good question. Maybe so that it’s easier to recognize dark objects on light backgrounds more easily?

Either way, as far as we can tell, this ON/OFF channel pair only occurs with the red and green cones. Blue cones and rods have only ON bipolar cells. Though with rods, the story is a bit more complicated.

Rod signals are carefully fed into cone pathways

Humans (and other primates) have a fantastic vision. One of its crazy feats is that it can see across 10 log units of light intensity by using different light detectors and different retinal circuits in different lighting conditions.

Rods can detect single photons (!) and become saturated when it gets brighter than dusk/dawn. Rod signals are further amplified by neurons that collect multiple rod signals, suppress the noise, and feed the signals into the cone signal pathways.

This is done with help of the so-called AII amacrine cells (that’s roman numeral “2”, so it’s really “A2”, but the literature calls it “AII” for historical reasons). These AII cells collect inputs from the rod bipolar cells (which come only in ON variety), and use that signal to inhibit the cone OFF bipolar cells, as well as directly feed voltage into the ON bipolar cells through the connections called “gap junctions”.

This way, even when it’s dark, the same parts of the brain can be used to process the incoming information even though it’s been collected by a different set of photoreceptors.

Motion gets detected starting in the retina

Some output neurons - Parasol cells (ON and OFF varieties) and Recursive cells - produce the biggest response when the stimulus moves across its visual field.

What kind of neuron circuits make that happen?

For parasol cells, this paper suggests that it’s a network of special diffuse bipolar cells that makes it happen.

When the signal moves left to right in the image below, the bipolar cells excite each other through gap junctions that let voltage jump from one cell to the next.

Due to the internal mechanisms of neurons, these signals add more than linearly and produce a big response in the last bipolar neuron - ultimately exciting the parasol cell a lot.

Horizontal cells sharpen the picture

Retina also has a number of circuits for sharpening the image and enhancing the contrast.

One of them involves the so-called “horizontal cells” (yellow “H1” and “H2” cell in the image at the start of this post). These live in the same layer as the bipolar cells and connect to a lot of nearby cones.

The way the horizontal cells work is pretty clever: they are excited by the cones, and then, in turn, inhibit the cones back.

To see how these sharpen the image, let’s carefully look at what happens when the light shines only on one photoreceptor in the picture above.

If only one cone receives light, then

1. It fires less, so

2. Horizontal cell is less excited, and so it fires less too, therefore

3. Horizontal cell doesn’t inhibit the other surrounding cones as much, and so

4. Other cones fire more, producing a stronger OFF (darkness) signal.

The result is that we see a bright dot surrounded by a darker ring, making the dot more prominent.

By the way, this type of center-surround contrast enhancement is a common feature across all sensory systems, including touch, hearing, and smell.

Like image sharpening in Photoshop, this process can leave artifacts. The most famous one is below - the Hermann grid. The dark spots at the intersections disappear when you look straight at them because the surround inhibition fields are larger at the periphery of vision than in the center.

Blue cones are lower-class citizens in the retina.

For various reasons, blue cones get treated differently by the retina. They sit a bit farther back from green and red cones, they don’t get OFF channels, and there aren’t output (ganglion) cells that produce a pure blue signal.

Instead, the ganglion cells responsible for blue color combine blue ON signal together with red+green (= yellow) OFF signal to produce a “relatively blue” signal.

Blue cones are also sparser and aren’t present in the very center of the retina. This is in part because the lens in the eye produces chromatic aberrations at shorter wavelengths, and tighter spacing between blue cones wouldn’t help us see better.

Dopamine acts as additional gain control

Retina also adapts to light through Dopaminergic Amacrine Cells (DAC - dark blue on the diagram above). They spread their tentacles throughout the retina and around the bodies of some neurons.

When the light becomes bright enough, they ramp up the dopamine production, which in turn lowers the volume in the retinal circuits through various mecanisms.

Over the years I’ve come across a few approaches that help coordinate work between threads without having to make anyone wait. Here are a couple of them.

WARNING: I stopped using these patterns because they are easy to misunderstand and misuse, which can lead to random hard-to-diagnose bugs. If you use these patterns, you’re not helping the next develper understand the code better or avoid mistakes. You have been warned.

Reference to an immutable object

We know that in a multithreaded environment, modifying a shared object without locks is a bad idea. But what about replacing it?

For example, consider this code. I’ll use C#, but the same applies in Java, C++, or another traditional language with threads.

public class FunWithThreads {
    private Thing thing;

    public void InThreadOne() {
        this.thing = new Thing()
    }

    public Result InThreadTwo() {
        var localThing = this.thing;
        var result1 = SomeCalculation1(localThing);
        var result2 = SomeCalculation2(localThing);
        var finalResult = DoMoreWork(localThing, result1, result2);
        return result;
    }
}

Suppose lots of threads are concurrently calling InThreadOne(), while lots of other threads are concurrently calling InThreadTwo(). Will we ever get a wrong result, a race condition, or a conflict?

Not really. InThreadTwo() always refers to a consistent version of this.thing because it places the reference to it into a local variable, and from then on uses only the local reference. If the calculations take too long, then maybe we get a result that’s a bit stale, but that result will always be consistent.

Where it’s useful: I found this approach to be useful whenever this.thing is a large, pre-computed data structure that gets infrequently updated but frequently read. Mostly I’ve used it to cache some data for internal capital markets applications to make the application feel snappier to the user.

The pitfalls: It’s really important to make sure that this.thing is referenced only once at the beginning of the method, or else we’re back to race conditions. It’s also really important that this.thing is never modified after it’s assigned. That means lots of descriptive comments, very careful code reviews, and making sure that every developer that ever looks at this code knows all the gotchas. In a larger codebase, someone (maybe even you!) might still find a way to make a mistake somewhere.

Complex atomic operations

The reason the above code works is because in C# (and in most other languages) assignment of references is an atomic operation, i.e. while it’s executing, another thread can’t sneak in mid-way and do something else.

Modern processors have a few other interesting atomic operations, like compare and swap, implemented in C# as Interlocked.CompareExchange(), and in Java as AtomicInteger.compareAndSet(). This function takes three arguments and does the following as an atomic operation:

1. Compares the first and the third arguments, then

2. If they’re equal, sets the first argument to be equal to the second argument, or

3. If they’re not equal, does nothing.

One case where this can be used is when trying to limit requests for an expensive operation on a limited resource. For example, this code will drop any requests if the expensive operation is already running:

// Let’s pretend this class is a singleton.
public class ExpensiveOperationLimiter {
    private int isProcessing = 0; 

    public void RequestExpensiveOperation() {
        // If isProcessing == 1, this does nothing and returns 1.
        // If isProcessing == 0, this sets isProcessing to 1 and returns 0. 
        var wasProcessing = Interlocked.CompareExchange(ref isProcessing, 1, 0);

        if (!wasProcessing) {
            // Only one invocation will be running at a time.
            DoExpensiveOperation()
        }

        isProcessing = 0;
    }

    private void DoExpensiveOperation() {
        // ...
    }
}

If you look at the above code carefully, you’ll see that you can’t replace Interlocked.CompareExchange() and still have thread safety - there will always be an opportunity for another thread to sneak in between the instructions.

With a bit of work, this code can also be expanded to queue up the next request.

Where it’s useful: I’ve used this kind of code in situations where events can randomly trigger an application server’s cache to refresh from a database or from another system.

The pitfalls: Generally, I’d consider this to be a micro-optimization and not worth the effort. Many developers understand or have heard of locks, and have an easier time following them than any code that involves compare-and-set. As before, by using this, you’re risking someone (including you!) eventually making a mistake.

The best way to avoid locks

Don’t write multithreaded code!

One of those gaps was about how exactly the information travels through a neuron. Up to this point, all I knew was “something involving electric pulses”.

The reality, it turns out, is more interesting: neurons use ion channels to create and strengthen electric signals, and propagate information along their lengths.

So here is more or less how I understand this so far.

The setup - the neuron

First, a quick recap of the neuron structure.

A neuron has the main cell body. From that body, it grows a large number of branched extensions called “dendrites”. These usually serve as the input “wires”, though they also do some signal processing.

Neuron usually also grows another long extension called an “axon”. This is normally the cell’s output, and it can be up to a meter long in humans, though in the brain it’s usually shorter.

Let’s zoom in to the axon. First, we’ll look at shorter axons, because longer ones have myelin sheath, which will complicate the picture.

Ions provide the electric charge

The fluid inside the neuron (cytoplasm) and the intercellular fluid between the neurons have a number of ions dissolved in them: calcium (Ca²⁺), potassium (K⁺), sodium (Na⁺), chlorine (Cl⁻), and others.

For various reasons, neurons actively work to pump out Na⁺, Ca²⁺, and Cl⁻, so these are much more concentrated outside the cell than inside the cell. If the membrane wasn’t keeping them out, they’d quickly rush into the cell. Some still leak back in.

Similarly, neurons actively work to concentrate K⁺ inside.

Each of the ions carries a charge, so the end result of this pumping is that the cell’s inside is negatively charged relative to the outside. This voltage difference is called the “membrane potential”. Resting membrane potential is typically between -75 and -40 mV (millivolts).

Ion channels create voltage spikes

Ion channels are small valves set in the cell’s surface that let ions through if certain conditions are met. Some are opened by specific chemicals binding to them (e.g. neurotransmitters), while others open when the voltage inside the cell rises high enough. The latter are called “voltage-gated ion channels”, and they’re the ones we’re interested in.

Voltage-gated ion channels exist on all parts of a neuron, but are most concentrated in the axon. Most let through a specific ion: there are Na⁺ channels, K⁺ channels, Ca²⁺ channels, etc.

Let’s take a look at how Na⁺ channels and K⁺ channels would work to produce a voltage spike (called an “action potential”).

If something increases the cell membrane voltage to -55 mV, Na⁺ channels quickly open and let in a lot of Na⁺ ions. These ions carry a positive charge, so they quickly raise the voltage across the membrane to +30 mV. Then the channels automatically close. This happens over the span of milliseconds.

At the same time, K⁺ channels open and start letting out some K⁺ ions, opposing the Na⁺ current and slowing the voltage increase. The channels first open slowly, but after a while, the K⁺ current strengthens.

To start, Na+ channels dominate, but after they close, K⁺ outflow quickly makes the membrane voltage drop. This drop overshoots the initial resting level before leveling off at the voltage where K⁺ concentrations stay stable.

Then K⁺ channels gradually close.

The total number of ions exchanged throughout the whole process is minuscule, and has little impact on the overall concentration of ions inside or outside the cell.

Voltage spike opens nearby channels

When the channels create a voltage spike, they then trigger other nearby channels, which then trigger other channels, and so on. This makes the voltage spike travel down the length of the neuron.

The ion channels usually take time to reset before the next firing, so a single voltage spike doesn’t usually set off the entire neuron ringing indefinitely. However, the voltage can propagate from the base of the axon back into the dendrites, where it can play an important role in influencing whether the next incoming signal will set off the ion channels there.

Membrane properties limit the signal speed

Neurons are somewhat conductive, but not very conductive - they’re not copper wires. They’re also leaky - a large portion of their current seeps out into the surrounding intercellular fluid, partly because the cell membrane is not 100% impermeable to ions, and partly because of the membrane capacitance.

That means that if a given ion channel generates some voltage locally, that voltage will quickly drop off as we move farther from that channel.

So the ion channels cannot be spaced too far apart, and the signal will travel only as fast as the channels can open and close between the start and the end of the axon.

Myelin speeds up the signal

Vertebrates (including humans) have a clever innovation to speed up the propagation of signals: they use myelin sheaths to insulate axons that travel over distances longer than millimeters.

Myelin sheaths surrounding an axon of the peripheral nervous system (i.e. outside the brain or spine). Source: “Anatomy and Physiology” by the US National Cancer Institute’s Surveillance, Epidemiology and End Results (SEER) Program.

Myelin sheaths form in 0.2- 1 mm segments over the neurons and are made from layers of cell membrane and protein made by the dedicated cells - oligodendrocytes in the brain and spinal cord, and by Schwann cells everywhere else.

Myelin significantly thickens the axon’s wall and reduces the amount of current leaked out through the cell membrane. This allows the myelinated neuron to space their ion channels farther apart, significantly speeding up the propagation of the voltage spikes.

Newcomen engine is one of the first viable steam engines originally made in 1712 in England to pump water out of mines.

Unlike later steam engines, it was an “atmospheric engine” in which the power stroke was produced by the steam condensing into water and thus creating a vacuum; the atmospheric pressure then pushed the cylinder down, moving the big beam.

It was extremely inefficient by modern standards, but that wasn’t a big deal if it was sitting next to a coal mine.

Some fun details

• This engine is shown pumping the water out of a mine. The bulk of this water was discarded, but some was pumped up to a container at the top that would feed the engine.

• If too much air got into the main cylinder, it would stall. That’s why you see the engine leaking a bit of steam through a pipe when it’s inflating - to make sure that there’s minimum air in the main cylinder. The valve that the steam is leaking from was called the “snifting valve”.

• The big vertical plank that moves up and down is called the “plug tree”.

• The little stream of water on top of the cylinder is there to create an airtight seal.

• Fire is really painful to animate by hand if you don’t have any animation experience.

Not shown in the animation is an avanced feature of some of the engines: a pressure-controlled valve that helped refill the water in the boiler.

The animation above is rather stylized. This sketch from a 1868 book shows what it actually looked like on the inside:

Though typically the boiler, the pipes, and the switches would be enclosed in a brick structure like the one below. You can see the same engine in action in this YouTube video.

Some of the root causes may be organizational, but technical choices also make a big difference.

After seeing plenty of projects on either end of the spectrum, here are the biggest differences between codebases that age well and the ones that don’t.

1. Have less stuff to maintain

The most maintainable systems I’ve seen had architects that pushed for:

• Fewer features by being choosy about which features to implement

• Fewer technologies rather than more - e.g. avoiding polyglot microservices

• Sticking to the same patterns over time, even if they’re imperfect (but good enough)

• Fewer services, technologies, 3rd party components, etc

• Fewer lines of code, relying more on third-party libraries and services

• Cleaning up old unused parts

• Fewer moving parts overall

Whatever is left would ideally be organized into groups of very similar-looking components so that whoever works on the system in five years would be able to understand it easily.

The inverse of this — a technology/feature/sub-system sprawl — is one of the scariest ways in which the system can become unmaintainable because it is difficult to undo. This can happen when generations of programmers come through an organization, decide that the old way of building things is bad, introduce new technologies/patterns/systems without replacing the old ones, and then are replaced by the next generation. Or by multiple teams working in parallel, independently picking their favorite tech stacks. To a lesser degree, this can happen when less experienced teams keep adding new tech to the mix without thinking through the complexity that comes out.

Either way, the end result is a system that’s hard to fully comprehend, easy to break, and probably has team members that can’t or don’t want to touch unfamiliar parts of the system.

2. Choose mature, popular technologies

If you want your codebase to be easy to update in five years, don’t use a technology that will be dead in five years. Or one that will change so much that upgrading becomes impractical. Anything built on Angular 1, Silverlight, or early milestone Spring Boot versions would be very uncomfortable to work with today.

For this reason, I prefer to pick popular technologies that are already on v2 or v3 and have largely stabilized.

As of early 2020, this means:

• React: Good. Popular, widely used, about six years old. Probably the best pick on the front end.

• F#: Bad. Mature, but not widely used. Avoid unless you already have F# experts, or it’s a solo side project.

• Svelte: Bad. Popular, high satisfaction, but new; might not survive or might change a lot in the next few years.

• C: Good. Will likely outlive all of us. Safe choice for its use cases.

There are a number of other, immediate advantages to picking popular, older technologies over newer or less popular ones:

• More and better tools + libraries

• Lots of Stackoverflow answers to frequent problems

• Easy to find developers who can work with it

There are several variations on this theme; one of my coworkers calls himself “Service Pack 2 guy” (service pack 1 is too early for him). Choose boring technology is another popular essay that expands on this.

3. Tests, tests, tests

At my new job, about a month into it, I needed to update a service written in Python that was rarely touched by others. It was written a few years ago and the original authors have left the company. After spending some time browsing around the source code, I discovered that not only did it have great comments, but also 98% unit test coverage! In addition, it was part of a massive integration test suite. A couple weeks later when I was done, I was extremely confident that I didn’t break any existing functionality, and that my new additions would probably work as expected.

While automated tests can slow you down in the short run, on systems that have a long life they’re totally worth the effort.

I find that the tests that give the most return for the effort are:

• Unit tests, because they’re easy to write and run, and

• End-to-end tests on the whole system, because they can uncover most integration issues and prove that the entire system works as expected.

In some cases, the latter type can also be run in production after deployments to catch problems early.

4. Help the next developer understand the system

Whether it’s a new developer joining a 100-person dev org, or a single dev looking at the source code that has not been touched in the past five years, they’ll have a hard time modifying a system when they don’t know what it does or why.

So help them out in advance. This means:

• Don’t surprise them with unexpected side-effects

• Leave comments that explain the “why”

• Have clear names

• High-level documentation

Who knows, the next developer could even be you!

Having done it in four different tech stacks over the past few years, I look forward to TDD a lot more in some of them than in others.

Let’s compare a few of the experiences to see what they get right and what they don’t, from best to worst:

1. .Net: xUnit and NCrunch

2. JS/TS: Jest and VS Code Jest extension

3. Python: pytest

4. Java: IntelliJ and Infinitest

What makes a good TDD experience

The main difference between test-driven development and regular development is:

• You write a lot more unit tests,

• You run unit tests more often,

• You debug the unit test failures more often.

So if the tech stack’s tools and libraries are better set up to support these, then the TDD process will be more pleasant.

The winner: .Net

NCrunch

The main star of the .Net TDD ecosystem is NCrunch add-in for Visual Studio. The best description of it is this animation from its landing page (unfortunately it looks best on desktops and tablets):

In it we see a few things:

• NCrunch runs continuously as you type.

• The colored dots on the left of each line lets you know whether the line is covered or not.

• The same dots go red when the tests covering the line fail.

• Dots turn to a red “X” when that line throws an exception. Can hover over the “X” to see the exception.

• Context menu on each line allows to run the test and break at that line.

As a result:

• Tests run continuously, with very fast feedback,

• Feedback is given in the context of the code you’re looking at,

• Debugging is very easy and is often unnecessary.

The overall experience is very smooth with most of the time spent on writing the tests and the code, and very little on debugging or troubleshooting the tools.

The downside is the price. Like with many other tools for the Microsoft stack, NCrunch is proprietary with a $159 one-time fee per named user, though in my experience it is totally worth it.

xUnit

The other outstanding element of the .Net ecosystem is xUnit test framework. Similar to other _Unit frameworks, it makes one subtly different choice that makes life easier: it makes a new instance of the test class for each test. Aside from saving a bit of typing (“before-each” functions are replaced with constructors), it ensures that the mess left by one test function does not impact another test function, which eliminates a class of test issues.

Finally, both major .Net test frameworks - NUnit and xUnit - are in a rare group of frameworks that provide an easy way to write in-line test cases like this:

public class SomeTests 
{
    [Theory]
    [InlineData(”“, “”)]
    [InlineData(”/somepath”, “/”)]
    [InlineData(”/some/deeper/path.png”, “/path.png”)]
    public void ShouldRewritePath(string input, string expectedOutput) {
        // ...
    }
}

Pytest does a good job with this as well. Jest doesn’t, but at least the flexibility of JavaScript lets you improvise something. Java’s JUnit 5 introduced something similar with parameterized tests, but its ergonomics is very poor due to JVM’s quirks.

Downsides

Moq is a nice mocking framework, but it can mock only what it can override, which in some cases is not a lot. For this reason, mocking many things in ASP.NET and the rest of the .Net Framework is not straightforward. This has been gradually getting better in .Net Core.

In addition, though C# syntax has evolved a lot over the years, its types sometimes force you to write pretty contrived expressions.

Putting it all together

The end result is that in .Net my experience writing tests has more “Oh, you can do that?” and less “WTF happened just now?” than in other tech stacks, which is one of the reasons I chose it my personal projects.

Runner-up: JS/TS, Jest, VS Code

So how does working with Jest compare to xUnit/NCrunch combo above? Let’s write something.

describe(”Submit button tests”, () => {
    test(”When user clicks the button, show an incomprehensible error.”, () => { 
        // Do something
    })
})

The awesome:

• Writing simple tests is super simple. Just write the test() function.

• Test names can be proper English sentences. Great for readability!

• describe() lets us group test suites and sub-suites. And these can also have nice, readable names!

• The power of JS lets us mock anything. Don’t like interfaces? Then don’t use them, mocking still works! (But I like them anyway)

The downsides:

• Have to improvise parametrized tests with a .forEach(() => {test(...)}). It works, but it requires a bit more thinking every time.

• Test functions can interfere with each other. You learn quickly to not do the stupid things that cause that, but again, it adds a bit of mental overhead.

Running and debugging the tests involves a bit of friction. You have two options for running the tests continuously: (1) jest --watch and (2) VS Code Jest extension.

The first option works, and it will give you regular feedback when you break or pass the tests. It’ll point you to the offending line too. But the debugging has to be done through console.log(), and the test runs can take a while even on modest codebases. It also occupies additional real estate as a separate terminal window, which is not terrible, but not fantastic either, especially on a laptop.

The second option, Jest VS Code extension, integrates the test results with the IDE and gives a handy “Debug” link above each test. Clicking that link launches VS Code debugger.

The debugger integration is quite solid, but the rest is not. After saving the file (and remember, in NCrunch we didn’t have to do that), you have to be a bit patient and let the extension catch up. After a few hours of work the extension inevitably crashes, and will not start up again until VS Code reloads the folder.

Alltogether, this leads to a continuous testing experience that’s workable, but not the best.

Python’s pytest: probably the second place too

My experience with Python is not as extensive as with the other three languages on the list, but using VS Code and Pytest seems to have a similar experience to jest --watch.

Pytest framework itself is a bit more pleasant, and mocking seems a bit easier. The downside is no static types, which means less help from the IDE. Also, there seems to be more limited IDE integration for Python unit testing in general.

The laggard: Java, IntelliJ

Finally, let’s write and run some tests using Java, JUnit 5, IntelliJ, and Infinitest.

Right off the bat, you can run into problems with @Test annotation and other JUnit API elements. They exist in both org.junit (JUnit 4) and org.junit.jupiter.api (JUnit 5) packages. If you’re using Spring Boot, you probably somehow have both, and it’s easy to accidentally import the wrong one and spend half an hour troubleshooting the resulting problems (and then helping every new coworker who accidentally does that). After two years with JUnit 5, I still repeat that mistake.

The second problem we run into is enabling continuous testing. In theory, Infinitest and IntelliJ should function seamlessly together: just install Infinitest, add it to the project, enable the continuous builds, and done. That’s the experience with NCrunch and Jest VS Code extension.

In practice though, something always goes wrong. Just now, for this blog post, I tried to set up a new project with Infinitest. After already doing this a couple times in the past few years, it took me about five minutes just to translate these instructions from the Infinitest website:

into these steps:

due to the different places where you can look at the facets and find a “+” button that has an “Infinitest” option. It still didn’t run.

In any case, in the past I had to stop running Infinitest because the continuous IntelliJ builds were giving me trouble; ultimately, it was easier to just run the tests manually.

There are also a bunch of other minor frictions, like the “command line too long” error on Windows, or an uncomfortable interface for parametrizing the unit tests.

To be fair, there are some bright spots. Running and debugging the tests is pretty smooth, and I can use the context menu to debug a specific test.

Or this one if you’re using Kotlin:

fun `That’s valid Kotlin function name`() {

}

Which is great for writing descriptive test names.

But overall, TDD is noticeably less pleasant in Java than in other tech stacks.

While we ultimately liked it, right from the first interaction it became apparent that Kubernetes was not a complete platform, just a good backbone for a platform. If we wanted a group of 90 developers, business analysts, and product owners to work comfortably with it day-to-day, we needed to deal with configuration/operation complexities, platform quirks, and some gaps in tooling.

To deal with that, we did what I’ve observed as a trend: we put our own platform layer on top of Kubernetes. This post goes into that layer.

There are three areas where we had to make additional improvements:

1. Interface between the developers and Kubernetes

2. Operations and maintenance

3. Auxiliary tooling, like logging, monitoring, and dashboards.

Still, we liked Kubernetes

The migration wasn’t cheap (for us). It took five months from “let’s try it out” to post-migration cleanup, with two to nine people working on it depending on the time. But ultimately it was worth it.

Compared to our last platform, AWS Elastic Beanstalk, on Kubernetes we got:

• Quick, painless deployments that needed less attention.

• Less distractions for the delivery teams, who could now iterate faster on their features.

• Stupidly easy high availability, including 100% uptime during deployments and self-healing after random crashes of service instances.

• Helm charts. These are third-party “packages” of functionality that you can install on your Kubernetes cluster with a single command. We discovered these after starting work on the migration, and they made our lives much simpler. For example, Splunk (Helm chart), ALB Ingress Controller (Helm instructions) and other components and tools were installed this way.

• Lower costs because we could pack more containers on individual EC2 instances.

• Easier disaster recovery. With declarative deployments and Helm charts, we’ll spend a lot less time redeploying a new, fully working production environment.

• Many convenience features that made the overall setup easier, including cluster-internal DNS for service discovery and simpler provisioning of AWS load balancers.

Overall, Kubernetes now feels like a solid, stable backbone of our infrastructure. Aside from some initial move-in pains, it is now giving us minimal trouble, and it feels like we won’t have to think much about for the next five years.

A side note: pension plan investment management is not the kind of business that requires scaling to a large number of instances. Our problem was making the infrastructure work effectively for a dozen or so teams, and making horizontal scalability not a concern in the few cases where we needed it. I’ve had people wonder whether Kubernetes was needlessly complex for this kind of organization. With hindsight, and having experienced other platforms and infrastructures, the answer is definitely “not too complex”.

…But Kubernetes wasn’t enough.

Though it was a good platform, but if we had made our delivery teams interact with Kubernetes directly, their lives would have become a lot more complicated.

1. Developer interactions with Kubernetes

The config files

After turning all the knobs and dials, our typical Kubernetes config files ended up being about 100 lines long. Each service needed its own config file, as there were some per-service differences between the files. If each team has 4-10 services, that’s a lot of config files. Easy to make a subtle mistake and spend days troubleshooting; hard to make wholesale changes to all of them.

Instead of large config files, wouldn’t it be nice if each service had a shorter manifest file that explained how it’s different from other services, like this:

name: my-service
service-type: java-8
instances: 2
environments:
  prod:
    instances: 4

We ended up building exactly that. Each service got a service.yml file in its root folder; our build+release tooling interpreted it, generated an appropriate dockerfile and Kubernetes config.yml, and used the right settings in various intermediate commands, like the command to create a container repository in AWS Elastic Container Registry.

The initial work to implement this was around two people working over four weeks, and included design/development, updating Azure Pipelines, educating the teams, and adding service.yml to each existing service. We then spent about as much time on updates and cleanup.

In case you’re familiar with Cloud Foundry, the service.yml file was heavily inspired by its manifest files. We didn’t go with Cloud Foundry because the providers we looked at were either too expensive (Pivotal) or were buggy and had questionable customer service (IBM Bluemix).

We did look at the options on the market to solve the problem. In each case, the time to evaluate, implement, and deal with the quirks would have been longer than to build our own, and that’s before adding time for the vendor assessment and contract negotiations.

Other interfaces to Kubernetes

From our previous experience, we found that:

Product Owners liked to control the time of deployment to UAT or production. We already had existing Azure Pipelines deployments, and the product owners were comfortable using them, so we made sure to keep the experience the same for them when switching over to Kubernetes.

Developers:
The developers’ preferences varied from team to team. Some teams liked continuous deployments to their test environments, and some did not. Some teams had several test environments, and some had only one. The solution to most of these preferences was Azure Pipelines as well.

We did, however, run into a problem when someone wanted to restart a service. One of the Kubernetes quirks is lack of an explicit rolling restart ability, i.e. the ability to restart all instances in batches, while keeping at some of them online. This is an explicit design decision that will not be fixed. There is still a way to force a rolling restart to happen, but it’s rather roundabout.

After some discussions, it turned out that our developers typicallty restarted services because of application configuration changes. When the config changed in the central configuration Git repository, they needed the service restarted. So we built an Azure Pipelines release that restarted all services affected by the configuration change and reported the event in Slack:

Sometimes the developers did need to interact directly with Kubernetes, and so everyone got kubectl installed on their machines, and got access to the Kubernetes dashboard.

Service dashboard

One of the gaps that we still have is a good service dashboard. Kubernetes does have a dashboard, but it presents a Kubernetes view of the system: separate lists of deployments, pods, namespaces, other Kubernetes objects. To see the complete state of your service, you’d have to browse across multiple objects on multiple screens.

The developers, on the other hand, thought of a service as a single line on the dashboard that combined all underlying Kubernetes information from the pods, deployments, and ingresses. It would have maybe the service name, a state indicator (healty/unhealthy/deploying), a few actions (e.g. “restart”), and links to relevant resources (e.g. logs).

This is still on our wish list, and we may build one if we don’t find one in the next few months.

2. Maintenance and operations

This is the area where Kubernetes made us do more work than Elastic Beanstalk, our previous platform.

Once you have yor cluster, you’ll have to take care of a few operational problems:

• Cluster auto-scaling, i.e. expanding or reducing the number of nodes in the cluster in proportion to the load. This did not come out of the box and we had to build it using AWS auto scaling groups.

• Cluster upgrades, including the Kubernetes version, the kubelet agents, and node EC2 images. Some components have to have versions that are in sync with some other Kubernetes components. Any upgrades will need to be done across all clusters you have, without disrupting production or integration testing. The good news is that Kubernetes will help you out with moving the replicas from one node to another and keeping some replicas alive. But we still had to orchestrate the whole upgrade process ourselves and fine-tune it to make sure that it causes minimum disruption.

• Container image upgrades. Sometimes you’d want to swap out the base image for the containers without changing the code that’s running on them. In cases of security issues, you’d want to do this for all containers on the cluster, ideally within a day. Again, without disrupting production or integration testing.

• Be prepared for service instances getting killed by any of the above processes at any time. If your teams are not used to programming for random disruptions, they now will have no choice because the service instances will get taken offline at inconvenient times.

None of the above were rare one-off processes; they occured regularly, and you have to be ready for it. For us, the work on this involved not just building CloudFormation templates and back-end processes, but also updating our service framework, working with the teams to educate them about the best architectural decisions for the platform, and helping implement some architectural changes.

Automation of some of these processes came in very handy when we were fine-tuning the EC2 instance types used as the cluster nodes. This allowed us to try out several instance types and eventually settle on the ones that had the right mix of CPU, memory, and cost.

We still have not automated container image swaps, in part because it would have require reworking parts of our build+release pipelines and retaining more build artifacts for longer time than we would like.

3. Additional tooling

The other major functionality that we wanted out of the platform was logs, monitoring, alerts, security scans. As features, these were either barely present in Kubernetes (log viewing), didn’t work well out of the box (the dashboard), or wasn’t there at all (the rest).

Fortunately, with Helm charts it was easy to set up the tools we already had in-house.

• Logging: we added Splunk, which we already using elsewhere. The Helm chart install was straightforward, except for a few minor surprises like incorrect splitting of individual events and incorrect timestamps. Some of it we fixed; some was not worth the trouble.

• Monitoring and alerting: added DataDog. Kubernetes and AWS feed some useful metrics into DataDog, though some weren’t immediately obvious.

• Kubernetes Dashboard: has nice features and is a good UI alternative to the kubectl command line tool. The biggest issue was that we wanted it to integrate with either AWS access management or with the corporate SAML. It turned out harder than expected; we eventually did it months after the Kubernetes migration. As mentioned before, the dashboard was great for Kubernetes operations, but not quite what the delivery teams needed.

• Container security scanning was not necessarily a Kubernetes problem, but it was something new that we had to deal with. After reviewing the options, we went with Anchore scan, which we also installed as using a Helm chart.

Final words

Kubernetes is a very nice tech, and a great piece of an overall solution. But for now it needs to be combined with custom in-house and third-party tools, otherwise it won’t help speed up the development or operations.

I would not be surprised if in five to ten years we will see emergence of a platform that will offer a complete end-to-end service management experience that would combine Kubernetes, Istio, and various other components, just like Kubernetes emerged as an end-to-end solution to the container orchestration problem.

This is not the first list of books to be published on the topic. There are many others like it, but this one is mine. I found these books to be useful after I was first promoted to the Technical Lead role, and even more helpful when I tried to improve the rest of the organization around me.

Lean Software Development

You are already familiar with Scrum, Kanban, etc. Now you can learn how to take the scalpel to them to get your team’s or organization’s productivity to the next level. Lean Software Development by Mary and Tom Poppendieck explains how to deliver software that solves the right problems, and how to do it efficiently while maintaining quality.

The key concepts that stayed with me were:

• Eliminating waste: questioning the value of all activities, and eliminating any that do not contribute to better or faster delivery. The book also highlights more subtle sources of waste, such as handoffs, or code sitting in a repository waiting for deployment. Avoiding handoffs (which can add weeks or months to delivery) was high on my mind when I started advocating for teams going into each others’ codebases instead of placing work items on each others’ backlogs.

• Delaying commitment until you really need to commit. In particular, for me this translated into avoiding having meetings to make decisions when there is zero chance of these decisions being implemented in the near future. Too often, either (1) the reality changes and the decision is no longer relevant, or (2) different people end up implementing the decision, and, since they have their own opinion, they start the decision-making process from the beginning again.

• Keeping the feedback cycles short. On the shortest scales, I could clearly see the improvement when we started using continuous deployments, and the business analysts could give feedback to the developers within minutes of the developers committing their partially completed changes. On larger scales, I’ve benefited several times from rolling out a partially completed feature to figure out how to develop that feature further.

Continuous Delivery

Written by Jex Humble and David Farley, Continuous Delivery is the technical counterpart to Lean Software Development above. It lays out specific techniques and practices that help improve delivery and lays out some arguments and case studies for why it’s a good idea to do so.

If you’re working for a company with modern development practices, most of this book should not be a surprise. Some suggestions may be controversial, like encouraging all developers to work on one central branch. If you find yourself shaking your head in disagreement, I suggest avoiding the knee-jerk reaction of slamming the book shut and instead reading to see where the authors are coming from, because they usually have solid reasoning.

The Leadership Pipeline

The Leadership Pipeline by Ram Charan is a simple explainer of what the expectations are from people at different levels in a corporate hierarchy, from an individual contributor to the CEO, with focus on internally building leaders who can move through these levels.

The book is focused on a hierarchical structure in a large enterprise, but is still relevant in smaller or flatter organizations, with some adjustments. For example, a manager ultimately responsible for work of several teams would ideally be making sure that (1) there is the right set of teams, (2) they have the right key individuals, and (3) that the teams are succeeding, generally by setting expectations, providing support/resources, and holding the teams or the key people accountable. But normally such manager should not be giving direct instructions to the team members or deciding the work at the team level.

One of the directly actionable takeaways for me (and a lesson after I left the first team where I was a tech lead) was always to be preparing someone to replace me.

The First 90 Days

The premise of the book is: “the President of the United States gets 100 days to prove himself; you get 90” when you get a new role. That may be a bit over-dramatic, but the book does present a good set of strategies that can help get started in a new position, whether at the same company or a different one.

What I liked more about it, however, was the emphasis on human interactions to get things done in an organization:

• Ensuring that you understand the lay of the land in your organization, what motivates your peers/superiors/reports.

• Building the right team, and making sure that the team has bought into what you’re trying to do.

• Setting the right expectations with the superiors.

These are useful even if you have not moved into a new role anytime recently. Come to think of it, I probably should pay more attention to these than I do now.

Death by Meeting

There are two books from Patrick M. Lencioni on this list, each one deserving its own mention.

Once you get past the tacky case study that comprises the majority of Death by Meeting, the general gist is this:

• Most meetings are done wrong. They’re boring, unproductive, and often don’t get to the real matter.

• This can be fixed by structuring the meetings right, and ensuring that in-depth, candid discussions occur before making a decision.

The suggested meeting structure is:

1. Daily status updates. Should be quick. Basically, the daily scrum.

2. Weekly tactical: about 1 hour long. Should have a short status update, plus in-depth discussion on 1-2 topics selected on the spot.

3. Monthly strategic (or ad-hoc): 2-4 hours. Discuss, decide on the 1-2 top critical issues. Should do some homework in advance.

4. Quarterly off-site: 1-2 days. Focus on the overall strategy, general trends, organizational design, etc.

Sales EQ

I mentioned the emphasis on human interactions to get things done in an organization, right? If you want to crank that up a notch, it’s worth reading books on sales. Out of the four that I liked, I picked Sales EQ by Jeb Blount because it serves as a good entry point, presents the general principles well, and is not overly focused on a specific aspect of the sales pipeline, like prospecting.

If I get the general gist of it right — and I’m not a salesperson, so I may be wrong — the three main points to making a large sale (e.g. your idea) are:

• Make the people feel like you understand them and can relate to them;

• Figure out what the people’s problems are and link those to what you’re selling, e.g. how your idea will help your boss(es) achieve their strategic goals;

• Just push through the fear of objections, because they won’t buy if you don’t ask.

There are more points in the book, but they’re a bit more sales-specific, like “don’t waste time on people who aren’t moving forward through the sales pipeline”.

Aside from Sales EQ, Rainmaking Conversations and SPIN Selling also have elements and techniques that can be useful in a non-sales leadership position.

The Five Dysfunctions of a Team

The last book on this list, The Five Dysfunctions of a Team, also by Patrick Lencioni, focuses on making a group of people work well as a team.

Similar to Death by Meeting above, most of the book is a single large case study (what he calls “a fable”). This case study presents a dysfunctional executive team and shows how to make them work together and perform well. The conclusions of the book are similar to Google’s five keys to a successful team (which is also a good read if you’re responsible for team performance), except this book makes it a bit more tangible.

• Gateways as libraries, for example Spring Cloud Gateway and Netflix Zuul 2 for JVM, Ocelot for .NET;

• From the cloud providers: AWS Application Load Balancer, AWS API Gateway, Azure Application Gateway, GCP Cloud Endpoints;

• Envoy, either by itself, or as Ambassador for Kubernetes, or as part of Istio;

• Platform solutions like Kubernetes Nginx ingress controller.

• Other open source gateways, like KrakenD or Kong (both of which come in open source and enterprise versions).

• Other commercial ones Cloverleaf.

It’s impossible to give a detailed feature comparison for all of them, but after some experience of selecting and running an API gateway in production, I feel that I’m at least slightly qualified to give advice on how to pick one.

Features - What the gateway needs to do

Before picking a gateway, let’s go over what’s expected from one.

Routing and service discovery

Routing is the most basic feature of an API gateway; without it, there is no point of having one. At a minimum, a gateway should be able to route via HTTP other URLs.

Many gateways also come with some form of service discovery, where back-end services register themselves with a registry, and the gateway gets the list of routable services from that registry. From my experience, unless you maintain hundreds or more distinct services, built-in service discovery is not that helpful because sometimes you may want to keep a service not accessible through the gateway but still discoverable by other services. (This assumes that gateway is only used by users outside your system, like human users or other external callers.) In that case, you’d want the services to opt into the routing explicitly.

On top of that, the platform on which you run the services may already have a built-in rough service discovery capability, e.g. DNS on Kubernetes. It’s also not that hard to roll your own crude “service discovery” with DNS naming conventions.

Authentication and authorization

Whether this is a requirement will depend on your specific authentication and authorization architecture. However, having a gateway that does not support auth in any form will definitely limit your architectural choices. Fortunately, almost all of them support auth in some way; as of writing this, Azure Application Gateway is the only major exception.

Here, the gateways split into three camps.

Gateways-as-libraries, such as Ocelot or Spring Cloud Gateway, can integrate with their respective frameworks — ASP.NET Core and Spring Framework. This makes it possible to let the gateway fully handle auth using any method supported by the framework (and usually there are a lot of choices) with minimal additional work.

You will also get the choice of letting the same framework handle user sessions in the gateway, which again can be set up with minimal effort.

If you choose not to use JWT (JSON Web Tokens), you will have to invent your own way of passing the user information from the gateway to the back-end services. This was the choice my team made a couple of years ago due to the capabilities of our OAuth 2 provider that we did not have a say in selecting. Today, I’d suggest sticking with JWT, which is widely supported. Depending on your web framework, you may also have to come up with a way of having the back-end services extract JWT/other user info and stick it into the service’s security context.

Delegating gateways: some gateways either require or give an option to authorize each request by making a sub-request to an authorization service, which you’re expected to provide. Kubernetes Nginx ingress controller, Envoy, and AWS API Gateway are in this category. Generally, the gateway would expect a simple pass/fail answer from such service and not anything fancy like a redirect.

Each of the gateways has their own requirements:

• AWS API Gateway can delegate authorization sub-requests only to an AWS Lambda or to AWS Cognito. That’s fine if you already have both feet in the AWS serverless world, but not great otherwise.

• Kubernetes Nginx ingress controller wraps Nginx auth functionality in Kubernetes annotations. In this setup, Nginx makes an HTTP sub-request to a service that’s expected to return 2xx or 401/403.

• Envoy expects a gRPC or HTTP service that handles a specific request/response schema. It passes the whole request context (location, headers, etc) to the delegated service. Envoy can also verify JWT tokens.

In all of these cases, the gateway’s primary role is routing, and you’re expected to bring your own components for authentication, sessions, and propagating user information to the back-end services.

Standalone gateways with auth provide opinionated authentication/authorization support, usually limited to JWT (JSON Web Tokens):

• KrakenD works exclusively with JWT. It can verify, sign, and revoke them, but it cannot issue them. You’d need to bring your own service that would log in users and issue tokens.

• Standalone Envoy can also verify JWT tokens, but it cannot handle the revocation.

• Istio, which comes with a built-in Envoy, supports only JWT and has streamlined integrations with several sources. Istio docs don’t mention support for token revocation. The price of Istio is that you have to install, manage, and use it; Istio includes a lot more than the gateway. This works well if you’re already planning to use Istio, but is not great otherwise.

• Kong provides the most authentication options among the standalone gateways; some of the options are available only in the enterprise version. Kong also provides its own user repository. The possible downside with Kong is that you’d have to set up and manage Postgres or Cassandra. This is not a big deal if you already use either of these, but not optimal if your organization has deep expertise in other database types.

• Google Cloud Endpoints supports JWT, and also has streamlined integrations with various sources. But, as with AWS, you’d need to be on GCP to take advantage of this gateway.

Overall by now you should start seeing a subtle theme: if you’re using a gateway that integrates very well with your tech stack, you’ll be able to set up more features faster, and it will be easier to maintain. Later in this post, I will make it a lot less subtle.

Routing to static websites

You may be interested in deploying website front-ends as static sites behind the gateway, e.g. in S3 buckets or in containers that have only Nginx and the website files. This has a few benefits:

• All API URLs in your front-end code will be in form of "/some/path" instead of envHostname + "/some/path".

• The gateway can block unauthorized users from loading a front-end page.

• Potentially simpler local development experience.

• Not having to deal with CORS.

If you do that, you’ll run into an interesting routing conflict. Many front-end frameworks such as React and Vue have their own routing that rewrites the contents of the browser URL bar to make it look like the user has moved to another page. The new URL might not correspond to a real resource on the back end, and when the user presses Refresh in the browser, your system should be prepared to deal with it.

For example, if the user presses the refresh button and the browser tries to load /portal/settings/app.bundle.js, your system has to know that it should go to the portal-service and load /app.bundle.js instead of /settings/app.bundle.js.

You have several options to handle this:

1. Have the gateway rewrite the URLs before forwarding the request to the static website service. This requires a gateway that can match and rewrite URLs based on a regex prefix. Some gateways can do that (Spring Cloud Gateway, Envoy), while others can’t (AWS API Gateway). Alternatively, if your gateway supports custom filters, you could write one to deal with this problem.

2. Let the static site service deal with it, by either setting up the appropriate rules in Nginx or wrapping the static site in a framework of your choice (e.g. Spring) and letting it handle the routing issues. The latter option will add slight complications all over the place and will require more compute resources.

3. Don’t host static sites behind the gateway; instead, put them on a CDN, or somewhere else. You’ll need to deal with CORS (generally not a big problem), and you may have to control authorization separately.

Protocols and TLS offloading

Generally, all gateways can handle HTTP/1.1 and TLS offloading, and by now majority can handle incoming HTTP/2 connections. It will be surprising if your top gateway choice cannot do these.

When it comes to protocols, the main differentiators are:

1. Whether the gateway can handle WebSockets: most can, but some can’t (AWS API Gateway, Netflix Zuul 1). This may be either critical for you, or completely irrelevant. However, if you pick a gateway that doesn’t support WebSockets and it turns out that you need them, you will spend a lot of effort on workarounds (that happened to us).

2. Whether the gateway can use HTTP/2 to communicate with the back-end services. For most people, it will not be critical, but you do get better performance with HTTP/2, so it’s definitely a nice feature to have.

With TLS, your biggest problem will be not whether the gateway can handle it, but how you’re going to store, distribute, and rotate the certificates securely. Certificate renewal can be painful, especially if you do it infrequently, and I suggest awarding bonus points to gateways that support easy certificate provisioning/rotation.

For example, Kubernetes Nginx ingress controller can be set up to renew the certificates from Let’s Encrypt automatically. On AWS, using a certificate from the AWS Certificate Manager is quite painless.

But on Azure, as of writing this, you cannot just link a certificate from the Key Vault to the Application Gateway.

Alternatively, you don’t have to do TLS offloading on the gateway. Instead, you could place another piece of infrastructure in front of the gateway (e.g. AWS Elastic Load Balancer) and have it offload TLS. This is the setup that we went with, and I think we got the best of both worlds: easy TLS certificate management and other features from AWS ELB, and an API gateway that fit our service ecosystem like a glove.

Other filters

Broadly, filtering is the stuff that the gateway can do between receiving the incoming request and forwarding it to the back-end service, or between receiving and forwarding the response.

In addition to the features mentioned in previous sections, filters can provide extra protection (rate limiting, header sanitization), fault-tolerance (retries), or other useful features (gzip compression, redirects). I’d classify these in the same category as heated car seats in Canadian winter: they’re handy, but if they’re not present, there are other ways to deal with these problems. That said, depending on your scale and architecture, some of these may turn out to be very useful to have specifically in the gateway, so I suggest thinking about that in advance.

Logging and request tracing

All gateways have some request logging capability; some can add tracing information to the requests for easier request correlation in the logs, e.g. tracing in Envoy. Tracing can be very useful when the request might be handled by multiple services, though as with most other filters, there are other solutions for it.

For me, the main differentiator is the timeliness of the logs. A gateway that logs to stdout or a file can have its logs shipped to a log aggregator like Splunk or SumoLogic in under a second. AWS API Gateway logs can show up in AWS CloudWatch after as much as a minute, and log aggregators often poll the S3 bucket with logs less often than that.

This difference may not seem like much, but which one would you rather have when you’re trying to troubleshoot a live issue?

Developer friendliness

Once you have a gateway, someone will have to feed it and take it out for walks. Let’s think through that ahead of time.

Organizational fit

Let’s imagine you just got a gateway. Now, someone will have to:

• Deploy and manage it.

• Deploy and maintain the configuration, including the routes.

• Handle unforeseen problems and requirements.

• Run multiple environments (production, test environments, local development environments).

• Promote the configuration changes between the environments.

• Upgrade and apply security patches for the gateway, the host server, and, if you’re using one, the container.

• Have a disaster recovery plan in case everything goes wrong.

All of this has to be done efficiently, and while keeping production errors to a minimum.

Also, consider how the gateway will fit into your organization. If the gateway is entirely different from the rest of your service ecosystem, and if the process to configure it is complicated, you will end up with one or a handful of people who will be “gateway people”, and every time the gateway will need to be touched, one of them will have to be involved.

This can lead to more handoffs, slower deliveries, avoiding the gateway (or worse, working around it), and key-person risk.

If configuring the gateway is easy, and if the gateway looks and feels like just another service in your system, then any experienced developer will be able to work on it.

For this reason, I recommend the following:

1. Pick a gateway that most of your developers will be comfortable with.

2. Take an infrastructure-as-code approach to the gateway deployment and configuration, keeping the configuration in source control and deploying both the gateway and the configuration automatically.

3. Make it easy to set up and manage the routes.

4. Pick a gateway that’s easy to patch and upgrade; ideally, one that is distributed through a repository like Docker Hub, npm, or Maven.

Managing the routes

Broadly, there are ways to make the route configuration easy to maintain:

1. Keep the all route configuration in a central repository and deploy it automatically, or

2. Whenever a new service is created, update the route on the gateway, either through a REST call or through service discovery.

At this point, every gateway supports one or the other or both. But whatever the approach, keep in mind that you may want to occasionally repoint the route from one service to another without changing the service names, or have a route pointed to something outside your service ecosystem.

At HOOPP, we’ve set up the route configuration in our regular configuration git repository, along with the rest of the gateway config. Any developer can submit a pull request with a change, and many have. The pull request has to pass a CI build that lints the config files, and be approved by someone on the Core Platform team (usually happens within 5-60 min). Then the config is automatically applied to test environments, and, if those start up successfully, to production. The gateway configuration is updated maybe every couple weeks; aside from a few initial hiccups, it did not cause any production issues. But if an issue does happen, git has the full revision history, and any change is easy to revert.

This is not the only way to do it, and probably not the best, but it was easy to set up and results in the least amount of work for everyone involved.

Local development experience

Something that’s easily overlooked: if you’d like the local development environment to resemble production as closely as possible, then you’d need to find a way to run the gateway on the developers’ workstations, including providing a route map that’s appropriate for local development, i.e. one that points to localhost routes.

At HOOPP, we found that some developers hated the idea of running the gateway locally, and some thought it was essential. You may need to accommodate both, but if you have to pick, I’d pick the run-the-gateway-locally side, and make the local startup and config management as fast and as painless as possible.

Gateways-as-libraries have an unfair advantage here. Since you already know how to start and manage configuration for your other services, and gateway-as-library is just another service, you will have an inherent head start when working with them.

Other considerations

There are other criteria when picking a gateway, but most of them tend not to be big differentiators for the major gateway solutions:

• Documentation: is it thin on information, or is it extensive and detailed?

• Answer availability: is it easy to search for answers on StackOverflow, Google, Github, or the vendor’s forum?

• Maturity: more mature products tend to be more secure and more battle-tested.

• Adoption: is it popular? Winners typically attract more resources for improvements, documentation, security patches, plugins, etc.

• Security: what’s the vendor’s security posture? Have they had security audits, is there a bug bounty program, etc? If you’re not sure how to evaluate that, I suggest involving someone with information security expertise.

Making the choice

Unless you’re prepared to have a team dedicated to the gateway, you’re almost certainly better off picking one that (1) is easy for you to automate and integrate into your systems, and that (2) your developers can support with minimal additional training.

Once you apply all the constraints from the preceding 2,800 words of the post, you will find that gateway-as-library in your organization’s language of choice is likely the front-runner, provided that it exists and satisfies the requirements in this post.

In the absence of that, the top choice would be the gateway that can be easily deployed on your existing infrastructure, like KrakenD or Envoy on a container-based infrastructure. This usually gives enough options that you don’t have to venture farther afield to pick another gateway.

The fundamental idea underpinning these habits is that the development team is doing well when it is delivering the right, high quality features quickly, with an emphasis being on the quick turnaround time for each feature, i.e. the cycle time. To do that, all team members need to be on the same page, and the development process for each feature should be moving along with minimum lag or rework. Basically, the development should be lean.

The corollary of this is that the team members need to:

• Be able to work together, possibly in the same complex codebase, with a large subset of the team working on the same feature until it’s done and then smoothly moving on to the next one;

• Continuously keep each other up-to-date on major developments;

• Smoothly hand over the work to the next person as it becomes ready, e.g. hand over the requirements to developers, hand over built features to QA or product owner for testing, etc.;

• Proactively unblock themselves if unforeseen blockages occur, such as a need to consult someone or get an external dependency moving.

To do that well, the team needs technology, architecture, and development practices that support rapid, quality development; good planning and product management practices; and, of course, great communication.

To build great communication, I found it useful to help the team members develop these specific communication habits.

1. Communicate continuously

Here, “continuously” being in the sense of “continuous integration“, not “constant, non-stop communication”; constant communication is undesirable because then no work gets done. Instead, continous communication, like continuous integration, should happen as soon as it needs to happen, not on a set schedule or at the end of some task. This practice is best illustrated by an example.

Meet the team

Let’s say three developers, Sam, Mandy, and Ali, are working on the same codebase and they’ve agreed on the technical design. Sam starts writing the code, but as he works, he keeps noticing imperfections in the design and gradually adjusts the code to account for those. Next day, when the time comes to integrate the code with Mary and Ali, it turns out that Sam’s code has substantially deviated from the original design, and the other two have to do a significant amount of rework.

Sam could have avoided this by letting his teammates know that he found a problem with the design and offering a correction. Mandy and Ali could have reviewed the correction and suggested improvements. Thus, for the cost of a quick interruption, Sam could have ensured that the team delivered faster and better.

Similarly, Mandy doesn’t need to dot every “i” in the code before handing the feature to the product owner for testing or showing the code to the other developers for a code review. Instead, she could start showing partially ready work, so that her colleagues could sooner discover any issues with the code, the requirements, or the design.

Building the habit

This continuous communication does not naturally occur to everyone, especially if they have worked in environments that do not encourage it. There are several mechanisms to make it happen:

• Suggest it to the team; some team members will likely agree and will start doing it.

• A technical lead managing junior developers could just decree it.

• In a team with more experienced members, it will likely be necessary for the team to meet, discuss, and agree that this is expected.

• With new hires, I have set the expectation of continuous communication in the interviews and during the onboarding.

In either case, it is worth emphasizing that the regular communication is not for the scrum master, the manager, or the product owner to check on the work, but for the team to keep the work flowing and to discover surprises as early as possible.

Whatever the method, it’s essential to keep the team members accountable to this standard. In the case of Sam above, if he is new to this habit, someone could casually ask him whether he needs to update Ali or Mandy on anything. If Sam shows a persistent pattern of not updating his teammates, this may warrant a 1-on-1 discussion with a warning that his actions are affecting the team.

2. Shared communication channels over 1-on-1 communication

The purpose of shared communication channels is to make the communication scale efficiently.

Discussing something one central place is more efficient than discussing it one-on-one and updating each teammate later. If Ali sends a pull request approval privately to Sam, then Mary will not know about it. She may not be needed to review the pull request, and may not even be interested in it now, but by posting the request in a shared chat channel, Ali would give her an opportunity to be aware that it is happening without doing any extra work.

Shared communication channels could take many forms, all of them with the purpose of making the communication scale:

• Scheduled meetings to review or discuss something;

• Chat channels, group email, and mailing lists;

• Ad-hoc conversations;

• Shared task boards, task lists;

• Wiki, or other documentation;

• Whiteboard with a design on it and a big “do not erase” sign.

Whatever the channel, I find the focus on the efficiency is critical; otherwise the team could get bogged down with needless meetings. Not everyone needs to show up to talk with an external vendor for two hours; instead, one or two people could go, and then update everyone in a 5-minute meeting, or, better yet, by email.

Similarly, I’ve given up on multi-hour planning on the first day of a sprint because most topics are not interesting to half of the people present, and detailed plans for work later in the sprint tend to change anyway. Instead, by combining the principles of efficient and continuous communication, we could shorten the sprint planning meeting to a 15-minute overview of upcoming features, and do the detailed design and planning as the team gets close to starting the work on the next item.

Well-organized written communication, such as documentation, or diagrams, tends to scale well but has a higher upfront cost. Chat channels and email tend to scale moderately. Meetings tend to drop in efficiency as more people join them, so it’s important to keep them focused and require presence only of those people who need to be there. Ad-hoc, one-on-one or group communication scales the worst but is best for resolving blocking questions.

Building the habit

The two parts to building this habit are figuring out the right communication channels and developing the habit of using them.

If the team does not meet regularly to discuss and decide something, Agile frameworks such as Scrum provide a good start, with meetings like planning, daily scrum, and retrospectives. These build the habit of regularly meeting together for a specific purpose. Over time, however, the team would benefit from reorganizing meetings, reducing some and adding others, like group code reviews or requirements reviews.

With written communication channels, I’ve gotten excellent returns from having a Wiki and a group chat like Slack.

The other side of this habit is redirecting the communication to the shared channels as much as possible. In my experience, repeated suggestions usually get the team most of the way there. For example, if a developer would ask me to review a piece of code, I’d suggest doing this with a substantial subset of the team instead. Or when someone sends a request for another team to do something, I’d ask to CC the rest of our team. Repeating the message of the importance of using the shared channels in the retrospectives also tends to help.

3. Proactive resolution of blockers

Extending the idea of not waiting to communicate, this habit has the team members act immediately when they find an unforeseen blockage.

For example, suppose a business analyst Fiona is almost done preparing the requirements when she realizes that she still doesn’t know what the source for a specific field should be. She could wait until the next scheduled meeting with the client, Bill, to find out, but that could mean delaying the implementation. Instead, ideally she would either call Bill right away or walk over to his desk to clarify this. Failing that, she would preferably email, or leave a voicemail, and only then fall back to the scheduled meetings.

Like with any interaction, this depends on whether Bill is receptive to unexpected calls. If they are infrequent and help resolve blockages, he probably will be. In fact, he may be glad that Fiona asked the right question that helped deliver a better product.

Similarly, if Ali (the developer) is troubleshooting a complicated bug and needs to involve another team, he should first try to walk over or call them right away and ask them to help, instead of sending an email and waiting a couple of days for a response.

Since ad-hoc, one-on-one communication scales the worst, this should be generally reserved for unforeseen blockages or urgent situations that were not identified during the planning.

This habit also helps in larger organizations where requests sometimes tend to fall between the cracks or get stalled at the bottom of someone’s to-do list. A polite in-person or phone question about the delivery timeline sometimes gets the request moving surprisingly quickly.

Building the habit

This habit requires the team members to overcome the anxiety of interrupting or inconveniencing someone. Unfortunately, the only way to become more comfortable with interrupting others is to do it enough times. Here, a more experienced team member could help someone new to this habit by setting an example or taking them along to talk to someone who did not expect them.

I usually introduce this to the new team members as soon as possible by taking them with me to talk to someone the first time they need something. With existing team members, often just a suggestion is enough (“Why don’t we ask her now?”), with an offer to come along if necessary.

With a well-designed client library, the service consumer gets:

• Less typing on the keyboard,

• Compile-time error checks instead of run-time checks,

• Easier mocking and unit testing,

• Hinting and autocomplete when using an IDE,

And in general all the good things that lead to reduced time to delivery on features and bug fixes.

These benefits are offset by the cost of writing and maintaining a client library, so it may not make sense to do so when the consumer should be allowed to use any language they want, e.g. in case of external-facing APIs. Or when there’s only one consumer for the client library, like a combination of tightly coupled back-end and front-end in a simple application.

Fortunately, there are often tools to make it easy to write and maintain client libraries.

A simple example

Let’s suppose we’re writing a service that maintains the list of users. If the service consumer wants to add another user, they submit a POST request like this:

{
    “name”: “Bob Li”,
    “email”: “bli@somewhere.com”
    // More details
}

It returns 200 OK if everything went fine; otherwise, it returns a 4xx/5xx error code with a message in the body.

This will work fine and will do its job. The first consumer of the service will probably write their own copy of the model and will use a convert it to JSON with a simple one-liner that is available in their language of choice. They may add a wrapper for mocking the service in the unit tests as well. The second consumer will probably do the same but in a separate piece of code, different from the first one. And so will the third.

There is, of course, a chance that they will all get together and decide to do this in a central, shared library that will have message models, call signatures, and some convenience functionality like retries or logging. This assumes that the service consumers are all aware of each others’ actions, have the time and willingness to get together to do this, and that the company culture supports taking time out of writing features to do foundational work. From my observations, unless there’s planning in advance, the odds of this shared library emerging aren’t high.

Writing a client library

In the above case, the advance planning would be writing the client library and making it available to the consumers of the service. This doesn’t have to be a lot of work. For the same user service, the library would include AddUserMessage type, possibly a return type, and a piece of code that wraps the process of making the HTTP call and handling the response.

For example, in Spring framework with Kotlin, we could write the client library using Feign like this:

class AddUserMessage(val name: String, val email: String)

// This will be auto-implemented by Feign
@FeignClient(”<url>”)
interface UserService {
    // Throws exception on failure
    @PostMapping()
    fun addUser(user: AddUserMessage)  
}

This is a basic blocking implementation and could use some improvement, but the service consumers already get: compile-time error checking, easy mocking for unit tests, and the IDE will help autocomplete the function and class names.

In cases of communicating over a queue, the client library would hide implementation details for the queue call, allowing the caller to think just about the call to the service and not worry about the underlying details. In case of async calls or long-running requests, the library could also provide correlation between the requests and responses, handling de-duplication, or managing other complexities. The client library could also manage retries, extracting meaningful error details, and emit progress notifications.

Automating client library generation

On a larger scale, there are some interesting examples of client libraries being generated automatically. For example, Michael Bryzek showed how Flow.io maintains a large number of microservices by generating client libraries from JSON declarations. The same generation system produces mock clients for testing; these mock clients have, according to Michael, mostly removed the need for integration testing.

In another example, AWS SDKs have all the signs of being automatically generated. Given Amazon’s other internal automation, it sounds unlikely that they chose instead to use an army of engineers to type out the highly repetitive implementation code.

Taking the idea to the next step, if the service consumer only uses the client library, then they do not need to be aware of the underlying protocol. Then we can switch from HTTP and REST to something like grpc and Protocol Buffers, and use the same declaration to generate multiple client libraries for several languages. At this point, unlike AWS above, we lose the option of supporting an arbitrary client language through an HTTP endpoint.

Taking this another step further, client libraries don’t even need to be just for services in the “microservice” sense. For example, an internal data warehouse could be considered a service. To improve cycle time for the whole company, the team maintaining the data warehouse could publish a shared library that provides type definitions and mappings (auto-generated from the database schema) that correspond to the underlying tables and views which it allows querying. Over time, the library could be enriched with change notification updates, transparent calls to cache instead of the database, query limits, and other restrictions that otherwise would have to be communicated to the developers.