How Narwal Helped a Leading Healthcare Technology Company Build AI Observability and Governance into its AI Platform

Summary

Narwal.ai helped a leading US-based healthcare technology company transform its production AI platform from a black box into a measurable enterprise system. By embedding AI observability, cost visibility, adoption analytics, and model-quality governance into the platform, Narwal gave leadership its first complete view of how AI was performing in production. After closing a five-month observability gap, Narwal delivered an executive dashboard in just three weeks and extended the same foundation into user-adoption tracking and model evaluation, enabling data-driven decisions on AI performance, cost, and quality.

Customer Challenge

The company’s AI agent was live, but the systems meant to monitor it weren’t talking to each other, and leadership was making decisions without the data to back them.

• A Five-Month Connectivity Gap: Snowflake Cortex usage and cost data had no working pipeline into Datadog, leaving nearly five months without reliable AI observability.
• No Cost or Usage Visibility: Leadership had no consistent way to see AI usage patterns, cost attribution, or historical trends for the newly launched agent.
• No Visibility into User Adoption: There was no structured way to track whether users were actually adopting the CAM Agent, or how usage was trending over time.
• No Framework for AI Quality Governance: Without a model-evaluation framework, there was no systematic way to catch quality issues before they affected users or leadership confidence.

Narwal’s Solution

Narwal treated AI observability and adoption analytics as one connected foundation, first closing the data gap, then building the reporting and evaluation layers on top of it.

• Closed the AI Observability Gap: Built an Azure DevOps pipeline to read metrics from Snowflake Cortex and publish them into Datadog, resolving a connectivity gap that had persisted for nearly five months.
• Executive AI Operations Dashboard: Delivered a working executive dashboard to leadership within three weeks, giving stakeholders their first real view into AI platform performance.
• Cost and Usage Reporting: Added cost attribution reporting, five monitoring alerts, usage-history ingestion, and a backfill of more than 64,000 metric-series to restore historical visibility.
• User Adoption Tracking: Partnered with the CAM Agent team to build and deploy a User Adoption Tracking API to production, giving leadership a consistent view of how usage was trending.
• Model-Quality Evaluation Framework: Stood up a Snowflake-based model-evaluation framework, deployed a benchmark model evaluation to production, and began extending evaluation coverage into Capital Budget workflows.
• ML Pipeline Health Check: Audited existing ML pipelines and shared findings across CAM, AIQ, Snowflake, and platform stakeholders to surface risks before they became incidents.

Business Outcomes

• Nearly Five Months of Blind Spots Closed: The company gained its first working Snowflake-to-Datadog observability pipeline after almost five months without one.
• Executive Visibility in Three Weeks: Leadership gained complete historical visibility into AI usage and costs, restoring over 64,000 production metrics.
• Adoption Now Measurable: User adoption tracking moved from nonexistent to a production-deployed capability, giving leadership a real signal on how the AI agent was actually being used.
• A Foundation for Model-Quality Governance: The model-evaluation framework gave the company a repeatable way to catch quality issues early, extending beyond the initial CAM Agent into Capital Budget workflows.

Why Narwal

For enterprises investing heavily in AI, success depends on more than launching models, it depends on knowing how they perform in production. Narwal transformed an AI platform that leadership couldn’t measure into one they could monitor, govern, and continuously improve, proving that AI observability is what makes enterprise AI accountable at scale.