Redis Backplane for SignalR

Adding a second API instance broke real-time updates. Fixing that surfaced a worse bug that had been there the whole time: every connected user was receiving every load, whether or not they had access to it.

We added API instances behind a load balancer to handle growth. Real-time updates immediately started failing, but only for some people, and only sometimes - which is the worst way for anything to fail.

The problem

A SignalR hub only knows about the connections held by the instance it is running on. Once there was more than one instance, a user only received updates published by whichever server they happened to be connected to. Everything published anywhere else never reached them, and nothing anywhere reported an error. The grid just quietly stopped being true.

Digging into it turned up a second problem that had nothing to do with scaling, and had been shipping happily for a long time: every connected user was receiving updates for every load. Updates were not scoped by access at all. The front end only rendered what a user was allowed to see, so nobody noticed - but the data was going over the wire to browsers that had no business holding it.

Constraints

• Scope it: A user should only receive updates for loads they actually have access to.
• Deliver it everywhere: Every user, on every instance, should get the updates they are entitled to.
• Ship it fast: This was actively affecting production, so the fix could not wait on a redesign.

What I built

• A Redis backplane: so every API instance publishes and subscribes through shared channels instead of broadcasting only to the connections it holds.
• SignalR groups from existing access keys: reusing the access model already enforcing tenant isolation everywhere else rather than inventing a second one.
• Group membership on connect: a user joins only the groups their access keys entitle them to, at the moment the connection is established.
• Scoped publishing: a grid update goes to the groups with access to that load, not to everyone on the hub.

Both problems closed at once. No missed updates regardless of which instance a user lands on, no refresh needed to see current data, and no load going out to a browser that should not receive it.

What I would do differently

Shipping fast meant keeping the existing structure and schema as close to untouched as I could. The cost of that is still in the code: when one field changes, we send the whole grid row. I would move to sending only the changed fields as deltas and merging them on the front end - the same change I would make to the grid this feeds , and the one piece of this I would call unfinished rather than done.

What I learned

Before this I understood SignalR as a way to broadcast. Building this taught me that groups are the actual unit of design - that deciding who is in which group, and when they join it, is most of what makes a real-time feature correct rather than merely fast. The backplane was the easy half.