Back to Portfolio
GCP Vertex AI RAG

Howie Case Study

Enterprise-Grade Multimodal RAG on Google Cloud

Howie Architecture Screenshot

The Challenge

In modern enterprises, critical knowledge is rarely confined to text documents. It is locked inside video tutorials, webinars, and unstructured PDFs. Most off-the-shelf RAG (Retrieval-Augmented Generation) systems fail to ingest or synthesize this multimodal data effectively, leading to “knowledge silos” where AI assistants cannot answer fundamental questions.

Furthermore, deploying these systems often leads to runaway cloud costs and “silent failures” where the AI hallucinates because it cannot retrieve the correct context.

The Solution

I architected “Howie” as a reference implementation for a production-ready, multimodal knowledge base on Google Cloud Platform. The system uses a microservices architecture, separating the heavy, offline data ingestion pipeline from the lean, high-performance FastAPI serving layer on Cloud Run.

Key Technical Innovations

  • Multimodal Ingestion Pipeline: Leveraging Gemini 2.5 Pro, I built a pipeline that does not just transcribe video but “watches” it, extracting structured steps and timestamps to allow users to query visual actions.
  • Data Consistency & “Ghost Nodes”: During development, I diagnosed a critical state-management issue where vector stores and document stores became desynchronized. I implemented a robust, persistent Docstore pattern to guarantee that every retrieved vector is correctly rehydrated with its original context.
  • Idempotent Infrastructure: To ensure reliability, the infrastructure uses “get-or-create” scripting patterns, allowing the entire environment to be torn down and rebuilt reliably.
  • Cost-Aware Operations: I implemented a “Deploy-on-Demand” architecture with automated scripts to manage the lifecycle of expensive Vertex AI Endpoints.

The Outcome

A scalable, verifiable AI assistant that provides accurate, sourced answers from mixed media. This architecture serves as a blueprint for companies looking to unlock their internal video and document archives without incurring massive technical debt.

Want the deep dive? For a full technical breakdown of how I diagnosed the ‘Silent RAG Failure’ and engineered the ‘Manual Transmission’ solution, read the complete article on Medium.