CASE STUDY: WEB RENDERING SERVICE
Introduction
The CEO of a web data extraction company needed help with upgrading their web rendering service. The old service was taking relatively long to render web pages, and they had already developed a proof-of-concept that sped up the process. They required help in prototyping and developing a production-ready replacement for the old service, which would seamlessly integrate with their SaaS platform. The company saw an opportunity in our expertise in client-side web technologies as well as web systems, web services, and web APIs development. Our support with developing the service allowed the company to start replacing the old service with a faster new version after around 2 months. The new web URL rendering endpoint had almost 2.5x lower latency.
The Story of the Web Data Extraction Company
The core value proposition of the company is extracting and structuring knowledge from web pages at scale and giving access to it through an easy-to-use yet powerful platform or APIs. Throughout the years, they have been extracting and structuring data using Knowledge Graphs from all of the public internet. They do it using a plethora of machine learning algorithms.
The Opportunity for Faster Web Crawling at Scale
The internet is constantly changing at a high pace, and in order to keep up with the changes, web crawling speed is crucial. The client already had an initial idea of how to increase extraction API performance but required someone with strong backend engineering skills and Chromium expertise to make it happen.
Why They Chose DevPeer
The CEO of the company reached out to us as he was specifically looking for a software engineer with expertise in the Chromium rendering process as well as deep backend experience. This mix of experience, supported by working for multiple companies across different industries, was what made us an ideal candidate for the project.
How DevPeer Responded
The initial focus was on making sure that the service could render web pages with optimized latency at scale. Lots of load tests were run to ensure minimum latency with optimal usage of hardware resources as well as stability and reliability. The next phase involved analyzing features within the existing rendering service that was being replaced. Bit by bit, the API features were implemented in the new service, while making sure they met user expectations by including well-crafted automated tests.
The Results
After 2 months, the rendering web service was ready to start gradually rolling out to production and replacing some traffic from the old one. Serving some user requests with the new service revealed minor issues and discrepancies, which were swiftly fixed and had minimal or unnoticeable impact on end-user experience. Customer satisfaction went up as the web crawling jobs were greatly sped up. Additionally, server hardware was utilized better, which led to noticeable cost savings.