The Canvas: Gigapixel on a Tablet

A whole-slide image (WSI) is a digital scan of a glass microscope slide — the tissue a pathologist (the doctor who diagnoses disease by looking at cells) reads to find cancer. Scanned at full magnification, one slide is gigapixel: on the order of 100,000 × 100,000 pixels, which is billions of pixels — far more than fits in a phone or laptop's memory at once. So the very first constraint is brutal: the image the pathologist needs to study cannot be loaded the way a normal photo is loaded.

Three facts make it harder still, and they are the facts that wrote this whole design:

• A whole-slide image is gigapixel — it does not fit in memory.
• Pathologists view on phones and, mostly, tablets, over flaky hospital networks.
• The canvas must render hundreds of thousands of annotations — and stay smooth.

On a real slide, the AI has already marked the tissue into classes, each with a live count. A single dense region can carry thousands of marks:

A traditional viewer loads the entire image and every mark up front. That is slow, and on a smaller machine it simply crashes. The rest of this day is the two moves that avoid that: never hold the whole image, and never draw what is not visible.

The core move is to stop treating the slide as one huge picture and start treating it as thousands of small ones. This happens once, automatically, the moment a slide is uploaded — and it turns a gigapixel-in-memory problem into a small-tiles-on-demand problem.

Here is the mechanism. On upload, an S3 event (S3 is Amazon's file storage; an "event" is a notification it fires when a file lands) kicks off a Fargate task — a container that runs without a server to manage — which converts the slide from its scanner format (NDPI, SVS, TIFF) into DZI tiles. DZI (Deep Zoom Image) is a standard format that pre-cuts an image into a pyramid of small square tiles, one set per zoom level. The conversion uses OpenSlide (a library that reads pathology scanner formats) and Pyvips (a fast, low-memory image processor).

The viewer then never asks for the whole image. It asks only for the handful of tiles under the current view, at the current zoom, and quietly fetches the tiles just outside the frame so panning feels instant. That is the whole trick.

A tile pyramid is what "pre-cut into a pyramid" means concretely: the same image stored at many magnifications, each magnification chopped into small fixed-size tiles (256 × 256 pixels is typical). The top of the pyramid is a single tiny thumbnail of the whole slide; the bottom is the full 100k × 100k image cut into thousands of tiles. Each zoom level is its own layer.

Why a pyramid instead of one big grid? Because zoom maps directly onto levels. When the pathologist is looking at the whole slide, the viewer serves a few tiles from a small level — low resolution, but that is all the screen can show anyway. When they zoom into one duct, the viewer switches to a deep level and serves only the few tiles that cover that duct at full resolution. The amount of pixel data moved over the network stays roughly constant no matter how far in or out they are, because it is always "a screen's worth of tiles."

The viewport is the rectangle of the slide currently visible on screen. At any moment the viewer computes which tiles of the current level fall inside that rectangle, requests just those, and preloads a one-tile ring around them so a pan does not stall waiting on the network.

This is why the viewer stays responsive on a tablet on hospital Wi‑Fi. The whole image never loads — only the tiles the viewport needs — and adjacent-tile preload hides the network jitter, so a pan reveals already-fetched tiles instead of grey squares. The image side of the canvas is solved. The other side — the hundreds of thousands of annotations laid over the tiles — needs its own idea.

Tiling fixes the picture; it does nothing for the marks. Drawing 100,000 polygons every frame will choke a tablet regardless of how the pixels are streamed. The Adaptive Viewing Algorithm is Orca's answer: what loads depends on zoom, viewport, and device resources, not on how many marks exist in total.

Three rules drive it:

• Load only the annotations near the current position; unload the rest as the pathologist pans away.
• Let detail track zoom — draw more per mark when zoomed in, less when zoomed out.
• Flex the thresholds with the device: how many marks to keep is tuned against device memory, browser memory, and CPU, so a weaker machine simply keeps a lighter but still representative set.

The "how much detail" branch — heatmap versus simplified shapes versus full polygons — and the device budget are exactly what Day 6's patent formalises. Here we only need the shape of the loop: decide what is near, decide how detailed, then clamp the total to what the device can hold.

The Adaptive Viewing Algorithm asks "which marks are near the viewport?" thousands of times as the pathologist pans. Answering that by scanning all 100,000 marks every time would be far too slow. A QuadTree solves it: a tree that recursively subdivides 2‑D space into four quadrants, subdividing again only where marks are dense, so a "what is near here?" query touches a handful of nodes instead of the whole set.

The tree gives spatial queries that stay cheap as the mark count grows — the property the viewport-near check depends on. The same QuadTree does a second job when the view is zoomed out and a quadrant holds hundreds of marks: it collapses that crowd into one summary marker instead of drawing every point. That clustering behaviour, and the rest of the level-of-detail machinery, is the subject of Day 6, the patent.

None of this is a bespoke "canvas screen." The canvas is built the same server-driven way as every other page in Orca: the backend sends an ordered list of section objects, and the generic frontend renderer draws each one by type. (This is the SDUI idea from Day 4 — the UI is data.) A canvas is just four sections in a list, and it comes together one section at a time.

First the tool rail:

{
  "page": "canvas",
  "sections": [
    {
      "type": "toolbar",
      "tools": ["select", "pen", "shapes", "pan", "ai", "zoom"],
      "actions": { "load": { "url": "/canvas/9/sections/toolbar", "method": "GET" } }
    }
  ]
}

Then the gigapixel viewer fills the workspace, streaming the DZI tiles from the pyramid we built above:

{
  "type": "viewer",
  "source": "dzi_tiles",
  "actions": { "load": { "url": "/canvas/9/sections/viewer", "method": "GET" } }
}

Then the annotation classes switch on beside the viewer, bound to the AI's output dataset:

{
  "type": "annotations",
  "data": "ai_classes",
  "actions": { "load": { "url": "/canvas/9/sections/annotations", "method": "GET" } }
}

And finally the project image strip docks along the bottom, and the canvas is complete:

{
  "type": "image_list",
  "data": "project_slides",
  "actions": { "load": { "url": "/canvas/9/sections/images", "method": "GET" } }
}

Four sections — toolbar, viewer, annotations, image_list — and the hardest screen in the product is just another list the backend hands over. The tiling, the adaptive viewing, and the QuadTree all live inside the viewer and annotations sections; the page that composes them is plain SDUI.

Put the pieces together and the payoff is qualitative but honest: the viewer stays interactive — pan and zoom, even with dense annotations — on the actual device a pathologist uses, with a network that drops. It does that for two reasons and no others: the whole image never loads, only the tiles the viewport needs, and the whole set of marks never draws, only the ones near the view at a detail the device can afford. Adjacent-tile preload hides the network jitter on top.

This came straight from watching real users at real microscopes — the constraint wrote the architecture. Day 6 takes the "how much detail" decision, which we waved at here, and makes it precise: the adaptive-rendering method Orca filed as a patent.

• A whole-slide image is gigapixel (~100k × 100k px) and cannot be held in memory; pathologists view it on tablets over flaky hospital networks with hundreds of thousands of marks on top.
• The core move: on upload, an S3 event triggers a Fargate task (OpenSlide + Pyvips) that converts the scan into a DZI tile pyramid — turning a gigapixel-in-memory problem into a small-tiles-on-demand one.
• A tile pyramid stores the image at many magnifications, each cut into small tiles; zoom maps to a level, so the viewer always moves about "a screen's worth" of tiles.
• The viewer requests only the tiles under the current viewport at the current zoom, and preloads a ring of neighbours so panning does not stall.
• The Adaptive Viewing Algorithm loads only marks near the viewport, lets detail track zoom, and clamps the total to device/browser memory and CPU.
• A QuadTree makes "what is near here?" cheap by subdividing space only where marks are dense — and later collapses dense crowds into cluster markers.
• The canvas is SDUI: four sections (toolbar, viewer, annotations, image_list) in a backend-driven list — no bespoke canvas screen exists.

• [ ] I can explain why a gigapixel whole-slide image cannot be loaded like a normal photo.
• [ ] I can describe the upload → S3 event → Fargate (OpenSlide + Pyvips) → DZI pyramid pipeline.
• [ ] I can explain what a tile pyramid is and why zoom maps to a pyramid level.
• [ ] I can explain viewport-only tile loading and why adjacent-tile preload hides network jitter.
• [ ] I can state the three rules of the Adaptive Viewing Algorithm (near-only, detail-tracks-zoom, device-clamped).
• [ ] I can explain what a QuadTree is and why it keeps "what is near here?" cheap.
• [ ] I can explain how the canvas is assembled from four SDUI sections rather than a custom screen.