Our service discovery caught its own failure and switched itself off
A three-replica cluster kept disagreeing with itself. The cause was in our own code: the peer-discovery routine had a catch block that disabled discovery when multicast failed, and on managed Kubernetes multicast always fails. Here's why we deleted gossip entirely and elected a leader with a blob lease instead, and why the membership you can open in Storage Explorer beats the membership you have to infer.
Jul 2, 20265 min read