Evolution of a Kendo Grid

It started fully client-side. Today it is a million rows and 81 columns under near-constant real-time updates. Three architectures later, here is what held up — and what I would skip if I started over.

When we first started work on our Kendo grid it was entirely client-side. During development that worked without any issues. The plan was to move branches over one at a time, which meant the data was small and manageable, and there was no need to build anything more complicated.

That changed quickly once the first branch started using the new software. We went through a few different iterations as the amount of data, the number of users, and the frequency of updates all kept growing.

Today the grid holds roughly 1 million rows and 81 columns, with near-constant real-time updates. This is how we got there, and what I would do differently if I were starting over.

Client-side

This was the original design, written early on by someone without a formal engineering background. It was reasonable for a first version or a prototype, but built without much thought toward future scaling or performance.

Client-side rendering stopped being viable almost immediately, with only one branch onboarded — slow grid load times and, in some cases, outright timeouts. While the data set is small, client-side is simple and easy to set up. It just does not scale gracefully; it stops working all at once, without warning.

Stored procedure, live query

The next step was the smallest change we could make to get off client-side. We moved the query into a stored procedure and added offset-based paging — effectively the same query as the original client-side load, but paged on the server instead of the client. It was the obvious move: not only the easiest, but the fastest to implement, and it improved front-end performance and data transfer speed without a redesign.

It did not last. As more branches moved onto the new software the query slowed down, and it finally became unusable because it was reconstructing a temporary table on every single page request. What had worked for a few branches fell apart as more records and more branches came online. We knew this was a temporary solution when we implemented it. Sometimes you patch a problem quickly knowing it is not the right answer — in this case it bought enough time to move further down the road, but it always had to be replaced.

The flat table

The current approach, already in place by the time I joined, uses a dedicated load grid table. That table is kept in sync by a backend service any time a load or order is updated, and the front end only ever queries it directly. No joins, no calculations at read time. That is what makes it hold up at scale.

The backend service is almost an exact replica of the original stored procedure for the temporary table. The only difference is that we defined a real table in the database, moved the procedure into the service, and had the service update the table directly.

This is the version I work on now, including scoping real-time updates through the SignalR and Redis backplane — which, notably, never caused performance problems. Each user only ever receives a single update per row, and the front end ignores updates for rows that are not currently displayed.

Other problems we ran into

Historical loads were killing performance: Most users rarely need to see loads that are already released to accounting or settled; they care about active loads. We added a filter that excludes historical loads by default, controlled by a URL query param, so users can opt in when they actually want them back. That fixed most — but not all — of the slowness admins were seeing when viewing every load.

Offset pagination gets worse the deeper you page: That is the downside of the offset paging we are still using. Even with the historical filter, admins who can see all loads still hit slow responses on the later pages. This is one of the bigger problems we face right now, and one we are actively discussing and white-boarding.

The localStorage date filter bug: This one was two separate problems stacked together. On the backend, Kendo’s built-in filtering defaults date columns to DateTime2 while we use DateTime, so each date column has to be explicitly configured in EF Core. On the front end, grid state and filters are saved to localStorage when a user leaves the page so we can restore it when they return — but the state is serialized into a string, and when we deserialize it and re-apply the filters, Kendo does not parse strings back into dates. Any user who filtered on a date field, left, and came back got a broken grid. We now parse those strings into date objects ourselves before handing them to Kendo. Every new date column has to be added to that list by hand, which is an easy step to miss.

What I would change

• Cursor-based pagination instead of offset: It would fix the deep-page slowdown, but it is genuinely hard here. Every column is filterable and multi-sort is supported, which makes a reliable dynamic cursor much more complex than the textbook version of the pattern.
• Query only the columns currently visible: Rather than pulling all 81 on every request.
• Virtualized infinite scroll instead of paging: Users have asked for page sizes above our 100-row cap and would honestly rather scroll than page. This is the one I would want most, but it only makes sense once the pagination and query-scoping problems above are actually solved.
• A view instead of a flat table: The grid can fall out of sync when a row update fails. A view would solve that, at the cost of recomputing calculated fields on every read instead of storing them. A good candidate for testing one against the other.

If I were starting over I would skip the client-side and stored-procedure phases entirely and go straight to the denormalized table built for read performance. And I would build pagination around the filter and sort complexity I actually had, instead of defaulting to offset paging.