Brain on Fire » Blog Tim McCormack says words

We need a better approach for pandemic self-isolation in modern America, given our baseline individualism, nuclear families, and nearly nonexistent social safety nets. Neighborhood resilience is an important step, but I think we can build on that to support people who need other people to come in their houses—such as those with other illnesses, those who need childcare, and other situations.

This is inspired by the needs of those with children. Children need other children to play with, the children need to be supervised, and the adults cannot work (from home or elsewhere) if the children are in the throes of cabin fever. And most challenging of all, children are terrible at hygiene, so any families that have children playing together should assume they are all in the same "transmission pool".

Here I propose a model for ad-hoc cell-based resilience. If you have research papers or studies that have bearing on this. please send them my way. If you know any experts on epidemiology or community resilience in epidemics, I would love to get their thoughts on all this.

This is not a guide, it is a discussion prompt. If feedback looks positive, perhaps someone could turn this from a specification into a public-friendly protocol, but this has not yet received review of any sort, so please do not share it as if it is an expert's recommendation.

One of the disconcerting things about the modern world is that I can follow news about a hurricane devastating Florida but enjoy a wonderful sunny day up here in Massachusetts. I can even watch in real time as trees are whipped about and homes are flooded. It doesn't feel like it's really happening.

And that's how it felt when the COVID-19 outbreak started in China a month or so ago. I read reports of quarantine, of overcrowded hospitals, of panic and fear. But here in the US, everything was calm.

I think that people are used to seeing images of danger comfortably on the other side of the glass of their TV or phone screen. The danger is Elsewhere, a place that cannot be reached from Here. Elsewhere cannot reach out its bony finger and strike at the heart of Here. It's disconnected, a fantasy land.

But viruses travel, and with cities and airplanes, they travel fast. The glass shatters.

Now it is here, in Somerville, Massachusetts, USA. It is increasing exponentially, or so I can only imagine, due to the deficit of tests. (Likely because of the lack of tests.) And even just yesterday, people were walking and driving and biking as if everything were perfectly normal. The virus also travels invisibly. The hurricane is here, bearing down on us, but to most people, so far, it just feels like an unusually windy day.

I've heard a lot of people get fatalistic about taking precautions, and talk about how they're inevitably going to get sick with COVID-19 even if they wash their hands, maintain distance, even if they partially self-isolate (e.g. work from home). This can be due to living with others, having a face-to-face customer service job, or just eating food that has passed through the hands of others.

And it's true. They're most likely going to come down with COVID-19 at some point, regardless of what they do.

The key—the thing people aren't talking about—is when. And that makes all the difference.

We use an air popper to make popcorn at home, and there are always a few unpopped kernels at the bottom. Far less than for microwave popcorn, and not enough to worry about waste-wise, but a few. I became curious about whether these were just unpopped, or actually unpoppable.

Verdict in my N=1 experiment: Yes, almost all of them can be repopped! The easiest thing is to just toss 'em back in the popper for next time. Throw 'em back, they're not big enough yet. ;-)

I became interested in finding The Perfect Load Balancer when we had a series of incidents at work involving a service talking to a database that was behaving erratically. While our first focus was on making the database more stable, it was clear to me that there could have been a vastly reduced impact to service if we had been able to load-balance requests more effectively between the database's several read endpoints.

The more I looked into the state of the art, the more surprised I was to discover that this is far from being a solved problem. There are plenty of load balancers, but many use algorithms that only work for one or two failure modes—and in these incidents, we had seen a variety of failure modes.

This post describes what I learned about the current state of load balancing for high availability, my understanding of the problematic dynamics of the most common tools, and where I think we should go from here.

(Disclaimer: This is based primarily on thought experiments and casual observations, and I have not had much luck in finding relevant academic literature. Critiques are very welcome!)

TL;DR

Points I'd like you to take away from this:

• Server health can only be understood in the context of the cluster's health
• Load balancers that use active healthchecks to kick out servers may unnecessarily lose traffic when healthchecks fail to be representative of real traffic health
• Passive monitoring of actual traffic allows latency and failure rate metrics to participate in equitable load distribution
• If small differences in server health produce large differences in load balancing, the system may oscillate wildly and unpredictably
• Randomness can inhibit mobbing and other unwanted correlated behaviors