We built a Conversational Data Analyst using Agentic AI. Here’s what actually worked (and what didn’t).
7 mins
read
l
Vasudev Gupta
l
April 9, 2026
Let’s craft intelligent solutions together—turning your data into unstoppable growth.
Most AI analytics systems today are built on a deceptively simple idea: connect a large language model to your data, and it becomes a universal interface for analysis. One model to understand intent, generate queries, retrieve documents, reason across sources, and visualize insights.
We started with that assumption.
It didn’t just underperform—it failed in ways that exposed a deeper truth: the architecture itself was flawed.
The issue wasn’t intelligence. It was that we were forcing a probabilistic system to behave like a deterministic execution engine. That abstraction works in controlled demos. It collapses under real analytical complexity.
LLMs operate as probabilistic sequence generators. They simulate reasoning through token prediction. Analytical systems, however, rely on deterministic execution—ordered steps, explicit dependencies, and verifiable outputs.
As soon as we introduced multi-step queries—especially those combining SQL with document retrieval—the system began to degrade. The model would start strong, correctly identifying relevant operations, but lose intermediate state as reasoning progressed. Constraints disappeared. Outputs remained fluent, but internally inconsistent.
This wasn’t hallucination. It was a failure of stateful computation under token constraints.
We weren’t building a smarter system—we were stretching the wrong abstraction.
Imagine trying to use a paintbrush—probabilistic, fluid, artistic—to tighten screws (deterministic, precise, mechanical).
The turning point came when we reframed the problem:
Analytics is not about generating answers. It’s about executing workflows.
A single business question can require:
• decomposing intent
• querying structured systems
• retrieving unstructured context
• aligning outputs
• presenting insights
These are fundamentally different tasks. Trying to compress them into one model guarantees trade-offs.
So we broke the system apart.
Neo is built as a system of specialized agents, each responsible for a distinct part of the workflow:
.png)
At first glance, this looks like a simple pipeline. It isn’t.
The defining characteristic of Neo is that it operates as a closed-loop system:
This shift—from linear pipelines to iterative execution—is what makes the system robust.
The Planning Agent converts natural language into a structured plan. It does not attempt to answer the query—it defines how the system should arrive at the answer.
Instead of implicit reasoning, we generate explicit execution structure.
You can think of it as compiling intent into a task graph:
Task Graph = {
tasks: [T1, T2, T3...],
dependencies: [(T1 → T3), (T2 → T3)...]
}
This eliminates a major failure mode of LLM systems: losing track of intermediate steps. The system no longer has to “remember” what it’s doing—the structure is externalized and enforced.
But task graphs alone aren't sufficient. Neo employs a proprietary graph-based context retrieval architecture that combines semantic understanding with structured relationship mapping. This maintains smaller context windows while achieving higher precision—resulting in faster planning, more accurate execution, and fewer hallucinations.
The Analyst Agent is where real work happens. It operates across both structured and unstructured environments, introducing something most LLM systems lack entirely: execution awareness.
Structured Data Flow (SQL Systems)
Instead of blindly generating SQL, the system executes in a loop:
1. Inspect Schema
2. Generate SQL
3. Execute Query
4. Validate Output
5. If error → Repair → Retry
This transforms SQL generation from a one-shot guess into a self-correcting process.
Unstructured Data Flow (RAG Systems)
For documents, the system follows a structured retrieval pipeline:
Query → Embedding → Vector Search → Re-rank → Context Extraction
But retrieval alone isn’t enough. The system compresses and structures outputs so they can be aligned with numerical data—enabling cross-source reasoning.
Secure Data Boundary
"Your data never leaves your infrastructure. Neo operates entirely within your ecosystem."
"More critically, Neo is grounded in your actual data. Every number is traceable. Every insight is verifiable. The system doesn't hallucinate—it validates its reasoning against real data state. You're seeing reality, not approximation."
One of the most critical design decisions was ensuring that raw data never leaves the execution layer:
Database → Execution Layer → Aggregated Results → LLM
NOT:
Database → LLM
This ensures both security and compliance, while still enabling reasoning over results.
The real challenge isn’t SQL or RAG individually—it’s connecting them.
Consider a query like:
“How are premium product sales performing against budget, and how does that align with our strategy?”
Answering this requires:
• computing variance from structured data
• extracting intent from documents
• aligning both across time and context
• identifying gaps between performance and narrative
This is a cross-modal reasoning problem, and it cannot be solved reliably in a single pass.
By separating planning and execution, Neo creates intermediate states that allow structured and unstructured outputs to be reconciled. This is what enables true analytical reasoning, not just data retrieval.
Most systems treat visualization as a formatting step. We found that to be a critical mistake.
Visualization is a decision problem. The system must choose:
• what to show
• how to show it
• what to emphasize
Instead of static rules, Neo models visualization as:
Chart = f(Data, Intent, Context)
The Visualization Agent selects representations dynamically, annotates key insights, and generates explanations that tie everything back to the original query.
The result is not just a chart—it’s a justified analytical output.
Building Neo wasn’t just about what worked—it was about identifying what fundamentally doesn’t.
End-to-End LLM Pipelines
Using a single model for the entire workflow produced outputs that looked correct but weren’t internally consistent. As complexity increased, the system would skip steps, merge logic, or lose dependencies entirely.
The issue is structural: without explicit execution graphs, multi-step reasoning breaks.
Stateless Querying
Processing each query independently simplified design but destroyed continuity. Follow-up questions require implicit context, and without memory, the system must reconstruct everything from scratch.
This not only increases error rates but makes iterative analysis impossible.
Direct SQL Generation Without Feedback
Without execution feedback, SQL generation becomes guesswork. Errors are not corrected—they propagate.
We saw frequent failures:
• incorrect joins
• schema mismatches
• empty results treated as valid
The absence of retry loops made the system brittle.
Naïve RAG
Basic retrieval improved recall but introduced noise. Retrieved documents were often unstructured and difficult to align with numerical outputs.
Without context compression and structured extraction, RAG cannot support analytical reasoning.
Static Visualization Logic
Rule-based chart selection failed under real-world variation. Data shape, scale, and intent cannot be captured through fixed mappings.
Visualization must be dynamic and reasoning-driven.
Each of these failures pointed to the same conclusion:
Analytics systems cannot be built as generation pipelines. They must be designed as stateful, orchestrated systems with explicit execution control.
"What distinguishes Neo is operational intelligence. The system knows when to act autonomously and when to surface constraints. It doesn't fail silently—it explains limitations clearly, whether missing data or schema restrictions. This is decision-making behavior, not error handling."
"Neo understands your data landscape. It comprehends schema structure, table relationships, and business context. When asked 'Why did retention drop in Q3?', it knows which tables connect and which metrics matter. Contextual reasoning, executed with precision."
At the heart of Neo is its execution loop:
While (goal_not_satisfied):
plan = PlanningAgent.update()
result = AnalystAgent.execute(plan)
feedback = validate(result)
Every step produces a trace:
Execution Trace = {
plan,
queries,
retrieved_docs,
errors,
retries,
outputs
}
This is what transforms the system from a black box into a traceable, debuggable, enterprise-ready system.
At this point, the real question is not whether Neo works—it’s why systems like this are necessary.
Traditional BI tools are built for predefined questions. They rely on static dashboards and fixed logic. The moment a question deviates, the system breaks and requires manual intervention.
Neo operates differently.
It allows users to interact at the level of questions, while handling the complexity of execution underneath. More importantly, it enables cross-source, multi-step reasoning - something traditional systems were never designed for.
This represents a shift from:
• data access
to
• decision systems
Generative AI produces outputs.
Agentic AI takes actions.
In analytics, this distinction is critical. Producing an answer is not enough. The system must arrive at that answer through a process that is:
• structured
• iterative
• verifiable
This is what Neo enables - a system that doesn’t just respond, but operates.
If an AI system cannot:
• decompose problems into executable steps
• interact with real data systems
• recover from failure
• and explain its reasoning
it is not an analytics system.
It is an interface.
Neo exists because real-world data problems demand more than intelligence. They demand structure, orchestration, and iteration.
And that is exactly what agentic analytics delivers.
