At the recent VB Transform 2026 conference, Toby Roberts, Vice President of Engineering at Zillow, alongside Arvind Jain, CEO of Glean, shared practical lessons on deploying large-scale AI systems. Zillow's representative emphasized that measuring AI's return on investment (ROI) is only accurate when organizations establish baseline evaluation benchmarks before building their systems. This approach has allowed Zillow to achieve measurable operational efficiency gains driven by practical AI adoption rather than chasing industry hype.
Context & Core Challenges
As a leading U.S. real estate tech platform handling up to 80% of annual transactions, Zillow faces the challenge of maintaining a seamless customer experience. Homebuyers typically undergo a months-long journey, interacting across various channels ranging from customer support agents and mortgage specialists to real estate agents. According to Roberts, a single chatbot could never maintain this complex flow of information throughout the entire process. Therefore, the real obstacle lies not in preparing raw data, but in building a 'persistent context layer' that tracks the customer's journey regardless of the channel they use. Zillow quickly decided to develop its own custom context layer rather than relying entirely on third-party chat interfaces.
Technical Analysis & Technology
Technically, Zillow chose to build its own orchestration framework instead of routing customer queries directly through a single large language model API. Leveraging 20 years of machine learning experience from products like Zestimate, the company focuses on using smaller, fine-tuned models for specialized tasks. Concurrently, Zillow's internal system runs thousands of Glean agents via the Glean MCP gateway to automate repetitive tasks. This solution centralizes data integration, preventing departments like legal or marketing from building redundant individual pipelines. Additionally, model routing and 'precomputed context' technologies eliminate the need to load data from scratch, significantly reducing latency and cutting token consumption by up to 50% when calling large models like Claude.
Expert Insights & Perspective
Commenting on this architecture, Glean CEO Arvind Jain noted that large language models alone cannot deliver true enterprise automation without being tightly integrated with internal context. Jain explained that redundant integrations across separate departments represent a massive hidden cost that many enterprises overlook when investing in AI. Meanwhile, Roberts warned engineers not to over-rely on automated permission controls within AI architectures for sensitive data. To ensure legal compliance, Zillow proactively implemented hard-coded business rules and independent compliance checks on top of its existing architecture.
Impact & Future Outlook
Zillow's success in proving that AI boosted code deployment by 40% was made possible because they had established DORA metrics years prior. This offers a valuable lesson for enterprises struggling to measure the actual return on investment in emerging technologies. Optimizing costs through context management and small model routing is set to become the dominant trend for driving sustainable, real-world AI adoption. Zillow's practical, hype-free approach reaffirms that data governance and upfront measurement planning are the true keys to the success of large-scale AI projects.