Case study: web rendering service
Introduction
Diffbot, a web data extraction company, engaged me to build a new web rendering service for their platform. The goal was lower rendering latency at crawl scale, delivered as a production-ready service that integrated with their existing SaaS platform. They were looking for someone who combined Chromium and client-side web expertise with backend service engineering. I prototyped the service, load-tested it, and built it out feature by feature until it was ready to take live traffic, which it began doing in stages after about two months.
I worked with Diffbot as an independent contractor. This case study is my own account of the work and is not endorsed by Diffbot.
About Diffbot
Diffbot extracts and structures knowledge from web pages at scale and makes it available through its platform and APIs. Its Knowledge Graph is built by crawling the public web and structuring what it finds with machine learning.
The Opportunity for Faster Web Crawling at Scale
The internet changes constantly, and keeping a knowledge graph current depends on how fast pages can be crawled and rendered. Rendering is the expensive step in that loop, which makes it the step worth rebuilding. The work called for backend service engineering and Chromium expertise in the same person.
Why I was a fit
Rendering work of this kind sits between two specialisms that rarely overlap: the internals of the Chromium rendering process, and the backend engineering needed to run it as a reliable service under load. I had spent years on both sides, across several companies and industries, which is the combination the project needed.
How I approached it
The first priority was establishing that the service could render pages at low latency under real concurrency. I ran load tests throughout, tuning for latency, stable resource usage and predictable behaviour under sustained traffic rather than for a single best-case number. From there the work was methodical: go through the API surface the new service had to match, implement it feature by feature, and cover each one with automated tests so that behaviour was verified rather than assumed.
The Results
After about two months the service was ready to begin taking production traffic, rolled out in stages alongside the system it was replacing. Staging the rollout was deliberate: it let each slice of traffic be checked against the old path before the next slice moved, so that any differences surfaced in a controlled way rather than all at once. My engagement covered the build and the start of that migration.
