Why Your AI Vendor's Demo Looked Great … and Their Delivery Didn't
Here are three main reasons this happens, and how to avoid it in the future.
You've seen it happen.
Maybe more than once.
A vendor walks into the room with a demo that does exactly what you need. The model works. The interface is clean. The use case maps perfectly to your business problem. Three months later, the project is stalled, the timeline has doubled, and the vendor is explaining why production is "more complex than expected."
You're not imagining things. The gap between demo and delivery is one of the most predictable failure patterns in AI projects. Understanding why it happens is the difference between choosing your next vendor wisely and repeating the same expensive lesson.
The Demo Is Designed to Impress, Not to Ship
The part nobody mentions during the sales process: a demo is a controlled environment. The data is curated. The edge cases are removed. The model is tuned to perform well on a narrow set of inputs that map perfectly to the story the vendor wants to tell.
None of that is dishonest, exactly. It's how demos work in every industry. But AI has a unique problem that makes the demo-to-production gap wider than most buyers realize.
In traditional software, what you see in a demo is roughly what you get. The interface might change, the features might shift, but the underlying mechanics are the same. In AI, the demo and the production system can be fundamentally different things. A model that performs at 95% accuracy on a curated dataset might drop to 70% on your real-world data. And 70% accuracy in production isn't a minor gap. It's the difference between a useful system and an expensive distraction.
Three Structural Reasons Demos Don't Translate
This isn't about bad vendors. Some of the firms that demo beautifully are genuinely talented teams. The problem is structural.
1. Your data isn't their data.
The demo ran on clean, well-labeled, representative data. Your data has gaps, inconsistencies, legacy formats, and domain-specific edge cases that nobody mentioned during the sales cycle because nobody asked the right questions. The vendor assumed your data would look like their training data. It doesn't. It never does.
This is where most projects hit their first real delay. The vendor scopes two weeks for data integration and spends two months discovering that your actual data environment looks nothing like what they planned for.
2. Production requirements were never part of the conversation.
A demo doesn't need to handle latency constraints, concurrent users, security requirements, system integrations, or graceful failure modes. Production does. These aren't minor implementation details. They're architectural decisions that should shape the entire system design from day one.
When a vendor builds the demo first and worries about production later, they're often building on a foundation that can't support what comes next. Retrofitting a demo into a production system is like renovating a house by starting with the wallpaper. At some point you have to deal with the plumbing, and by then, everything built on top is at risk.
3. The team that sold you isn't the team that builds it.
This one is particularly common with larger firms, but it happens at boutiques too. The senior architect who diagnosed your problem in the sales process hands off to a more junior team for delivery. That team is capable, but they weren't in the room when the nuances of your business were discussed. They're working from a scope document, not from understanding.
The result is technically competent work that misses the point. The model works. It just doesn't solve the problem you actually have.
What Buyers Miss During Evaluation
If you've been through this cycle, you might think the answer is better due diligence. Ask more questions. Check more references. That helps, but most evaluation processes focus on the wrong signals.
Buyers tend to evaluate what a vendor can build. The better question is how they build it.
Ask about their diagnostic process before any code is written. How do they assess whether your data can support the proposed solution? What happens when they discover it can't? Do they have a prototyping phase that uses your actual data, or do they go straight from demo to development? What does their team structure look like, and will the people in the room today be the people building tomorrow?
The answers to these questions tell you more than any demo ever will. A vendor who has a thoughtful diagnostic process, who builds with your real data early, and who keeps senior talent through delivery is structurally less likely to hit the demo-to-production wall. Their process is designed to surface problems before those problems become expensive.
The Diagnostic Step Most Vendors Skip
There's a phase that belongs between "this looks promising" and "let's build it" that most AI engagements skip entirely: an honest assessment of whether the proposed solution will work with your specific data, in your specific environment, for your specific use case.
This isn't a proof of concept. A POC is typically designed to prove the vendor's approach works. A diagnostic assessment is designed to find out whether it will. The difference matters. One is built to succeed. The other is built to tell the truth.
A proper diagnostic answers the questions that should have been asked before the contract was signed: Is the data sufficient? Is the problem well-framed? Are the success metrics aligned with what the business actually needs? What are the real risks, and what would mitigation look like?
When this step gets skipped, the entire project becomes the diagnostic. You're paying production prices for discovery work. By the time the real challenges surface, there's too much momentum and too much spend to change course easily.
Choosing Differently Next Time
If your last AI vendor delivered a great demo and a disappointing project, you're in good company. The pattern is common enough that it's become one of the primary reasons organizations develop AI skepticism. The problem isn't that AI doesn't work. It's that the way AI is sold doesn't match the way it needs to be built.
The organizations that break this cycle change how they buy. They insist on a diagnostic phase with real data before committing to a full build. They keep senior talent accountable through delivery, not just sales. And they ask the uncomfortable question early: what happens when this doesn't work as well as the demo?
The best time to ask that question is before you sign. The second best time is now.
Your AI Project Didn’t Work. Do You Know Why?
Consider the benefits of a third-party post-mortem. Seriously.
Cultivating resilience: New growth begins where the old lessons were learned.
Most organizations that walk away from a failed AI initiative make the same mistake twice: they move on without understanding what went wrong.
It's understandable. The project already cost time, money, and internal credibility. The last thing anyone wants is to spend more resources examining a failure. But that instinct, closing the chapter and moving on, is exactly what leads to the same problems showing up in the next attempt.
The Pattern We Keep Seeing
Here's what typically happens. An organization invests in an AI initiative. Maybe a recommendation engine, an internal knowledge system, or an automation pipeline. Six months in, the project stalls. The vendor blames the data. The internal team blames the vendor. Leadership pulls the plug and everyone agrees to "revisit AI next year."
When next year comes, a new vendor pitches a new approach. It sounds different enough to feel like a fresh start. But underneath, the same structural problems are waiting: unclear success criteria, underestimated data complexity, misalignment between what the business needs and what the technical team built.
This cycle is expensive. And it's avoidable.
Why Internal Reviews Fall Short
Some organizations do attempt a post-mortem after a failed AI project. But when the people who built it are the same people reviewing it, the analysis has natural blind spots.
This isn't about blame or competence. It's human nature. Teams that lived inside a project for months have assumptions baked so deeply into their thinking that they can't see them anymore. They know what they intended the architecture to do. They know why they made certain tradeoffs. What they can't see is where those intentions diverged from reality. They were too close to the work.
An internal review tends to surface symptoms: the model wasn't accurate enough, the data was messy, the timeline was too aggressive. Real observations, sure. But rarely root causes.
What an Independent Post-Mortem Actually Uncovers
An independent third party brings something no internal team can: fresh eyes with deep expertise and no attachment to the decisions that were already made.
A good AI post-mortem goes beyond "what happened." It answers three questions that actually matter for your next initiative:
1. Was this the right problem to solve with AI? Not everything that looks like an AI problem is one. Sometimes the real issue is a data pipeline problem, a process design problem, or something better solved with straightforward automation. An independent review can make this call without the sunk-cost bias that affects everyone who was involved in the original decision.
2. Where did the technical approach break down? AI projects fail for specific, diagnosable reasons. Was the training data representative of production conditions? Was the problem framed correctly for the chosen model architecture? Were the evaluation metrics aligned with actual business outcomes? Answering these requires genuine depth. A surface-level review produces surface-level answers.
3. What needs to be true for the next attempt to succeed? This is the question that makes the investment worthwhile. A post-mortem isn't an autopsy. It's a blueprint. What data needs to exist, and in what form? What does the team need to look like? What are the realistic milestones and decision points? An independent assessment can lay this out clearly because they're designing for what works, not defending what was already tried.
The Cost of Skipping This Step
Consider the math. If your failed AI initiative cost $200,000 (a conservative number for a mid-market project) and you start a new one without understanding why the first one failed, you're betting another $200,000 that the problems were circumstantial rather than structural.
Sometimes they are circumstantial. More often, they're not.
The research supports this. Industry data shows 42% of organizations abandoned most of their AI initiatives in 2025. The majority of generative AI pilots never reach production impact. These aren't isolated failures. They're patterns. And patterns have identifiable causes.
A post-mortem that costs a fraction of the original project can save you from repeating the full cost of failure. And it gives your team and your leadership something harder to quantify but just as valuable: confidence that the next initiative is built on solid ground, not optimism.
What to Look for in a Post-Mortem Partner
Not every consultant can do this well. Here's what matters:
Research-grade diagnostic ability. The person reviewing your project needs to understand AI systems at a fundamental level. Not just how to use tools, but why certain approaches fail under certain conditions. There's a real difference between someone who can identify that your model underperformed and someone who can tell you why it underperformed and what architectural decision caused it.
Cross-scale experience. AI fails differently at different organizational scales. Someone who's only worked with startups won't understand the integration challenges of enterprise systems. Someone who's only worked with Fortune 500 firms won't understand the resource constraints of a growing company. You want someone who's seen both ends.
Honesty over billable hours. The post-mortem partner should be willing to tell you that your next AI project shouldn't be an AI project at all. If the assessment is designed to generate follow-on work rather than genuine insight, that's not an assessment. It's a sales pitch.
Moving Forward
If your organization has shelved an AI initiative, or is watching one struggle right now, the worst thing you can do is nothing. The second worst is starting over without learning from what happened.
An independent post-mortem turns a failed investment into a clear path forward. It's a small step, but it changes the odds for everything that comes after.
You can afford the post-mortem. The question is whether you can afford to skip it.
Agentic AI - defined by context?
A blog post by this group made a good stab at defining “Agentic AI” by, in part, describing both what it is and contrasting it with what it is not. A very useful framing, illustrated by the diagram below. Something to help me think about the potential of combining RAG principles with multi-step, complex work flow and problem solving it a useful way.
The Next Iteration of RAG ?
Every aspect of LLM-based solution development remains in flux, with architectures evolving rapidly toward paradigms of greater power and business relevance. Recently I have encountered compelling arguments for what next-generation RAG should look like. A post by Brij Kishore Pandey — something along the lines of "Stop doing RAG like it's 2023" — resonated strongly, as I recognized that several advances that might be helpful if incorporated into my own work.
———————————————————-
The boundary between RAG and agentic approaches is itself becoming harder to define as agentic methods grow in strength, reliability, and ease of use. That said, the core principle remains constant: assembling the right context is central to effective LLM-based solutions, regardless of the architectural pattern used.
———————————————————-
Before mapping out Cognosa 2.0, I intend to study the graphic from Kis' post carefully. I am not, however, convinced that a Swiss army knife architecture will ever represent the optimal solution for every use case. Having already implemented enhanced retrieval types with tunable parameters, flexible filtering, and customized semantic chunking, I question whether every element of the proposed architecture is necessary — or even beneficial — for my intended use cases. More study is warranted. That said, my experience with agents, Claude Code, and the composability of skills and sub-agents gives me good reason to engage seriously with this framework and extract what is most applicable. Balancing the utility of generalizability and the power of finely honed improvements for narrow or speceific use cases is a common trade-off for product managers; perhaps investment is best defined in the context of my next customer!
Experimenting with a new LLM: QWEN
We have established an Alibaba Cloud account to evaluate their model offerings. Initial impressions are consistent with expectations: their models trail the frontier but perform comparably to Llama 3.1 and similar large open-source models — perhaps six to twelve months behind the leading providers. This week I plan to run direct comparisons against our existing Cognosa use cases, which is straightforward given the platform's ability to route identical queries across models and compare outputs side by side.
I have reduced the frequency of these comparisons, however, for two reasons. First, my daily satisfaction with Claude continues to increase. Second, ongoing disappointment with OpenAI — not only in coding tasks but across technical domains including DNS, Docker Compose, and infrastructure troubleshooting — reduces my incentive to benchmark regularly. Gemini has emerged as my preferred backup assistant and likely outperforms ChatGPT in most of my workflows, though I continue to alternate between them. I have also discontinued running large open-source models locally on my Mac Studio: inference is too slow, output quality insufficient, and my API spend across frontier providers remains well below any meaningful optimization threshold — at least for now.
Semantic Chunking
Or, more simply: how smart, customized RAG solutions deliver better answers
It probably won’t surprise anyone that a consultancy focused on semantic technologies believes that semantic decomposition — or the more headline-friendly term semantic chunking — really matters. Whether you’re building a generative AI assistant or a more traditional analytics workflow, better decisions usually come down to having the right information at the right moment. That might mean real-time sensor data driving an automated process, or it might mean helping a human answer a detailed question about a product, regulation, or technical system. In either case, identifying the relevant context quickly and accurately is what makes the answer useful.
For small or tightly scoped knowledge bases, this isn’t hard. A simple chatbot can scan most or all of the available material and produce a reasonable response. But once you’re dealing with hundreds or thousands of pages — manuals, specifications, discussions, updates, and historical notes — brute-force approaches stop working. You can’t realistically pass all that material to an LLM every time, and retraining a custom model is expensive, slow, and often impractical when the data is changing.
This is exactly why Retrieval-Augmented Generation (RAG) has become so popular. Instead of training the model on everything, you retrieve just the most relevant context and use that to support generation. But this is also where many RAG systems start to struggle. If your documents are simply chopped into uniform blocks and indexed, the retrieved “chunks” can end up being a kind of word salad — loosely related text that looks relevant on the surface but doesn’t actually answer the question being asked.
That’s where semantic chunking comes in. On the Cognosa AI-as-a-Service platform, we load data in a way that tries to preserve meaning, not just token counts. The goal is to capture complete ideas — instructions, definitions, explanations, constraints — at a level of detail that makes sense for each source. A 300-page user manual needs to be handled very differently from a collection of recipes or short biographies. Product catalogs introduce their own challenges, especially when many items share similar attributes and options and the system must not mix them up.
You’ll often hear people talk about different “chunking strategies,” and a quick search will turn up plenty of debates about which approach is best. But when you step back, most of these discussions are really circling the same principle: keep meaningful, coherent information together, and don’t break associations that matter. The specifics vary by use case and data source, but the underlying goal is the same — make it easy for the system to retrieve context that actually helps answer the question.
There are, of course, many ways to implement this. You can describe semantic chunking in algorithmic terms, or focus on whether the parsing is done with LLMs, traditional NLP tools, rule-based approaches, or some hybrid of all three. Those choices matter for cost, performance, and reliability, but they’re ultimately tactical. The strategy is not “use an LLM” or “use a specific algorithm”; the strategy is to organize knowledge in a way that consistently supports high-quality answers.
In the real world, data is rarely neat or homogeneous. A collection of journal articles or a large email archive is about as simple as it gets. Most enterprise knowledge bases are far messier: user manuals, standards and regulations, technical specifications, customer correspondence, chat logs, spreadsheets, diagrams, tables, and scanned documents all mixed together. Making this usable requires organizing content based on what it means and how it relates to likely questions — not just where it appears in a file.
At CogWrite, we’ve built and refined RAG collections using a range of tools and techniques across many projects. We maintain a growing library of ingestion routines and have created vector databases for different types of domain-specific AI solutions. This allows us to adapt quickly while still tailoring the approach to the data and the problem at hand.
One example is our work around Casambi, an open-source, low-energy Bluetooth lighting control protocol. There’s a wealth of publicly available material about Casambi — documentation, specifications, guides, presentations, and discussions — but it exists in many formats and levels of detail. We built a Casambi document collection specifically to support a question-and-answer workflow, and it has become one of our internal testbeds for experimenting with more sophisticated semantic chunking techniques and retrieval strategies.
A sample query and response from that system is shown below.
A simple query about a complext progocol implemented around the world for high quality industryial lighting control and management.
Cognosa Capabilities: LLM access options
Open Source or Proprietary Large Language Models
Queries With — or Without — Context
Cognosa supports two complementary ways of using AI. First, users can ask questions directly to a large language model, much like using a standard chatbot. Second—and far more powerful—users can query an LLM that is enhanced with organization‑specific data collections they are authorized to access. Both modes work with either free, open‑source models (no per‑token charges) or commercial models such as those from Anthropic or OpenAI, using the customer’s own API key.
Augmented Queries with Company Data
Organizations often maintain multiple data collections: industry research, regulatory guidance, internal policies, customer service resources, product documentation, and more. Cognosa can connect any or all of these collections to an LLM, giving users precise, context‑rich answers. In the example shown, this Cognosa workspace offers access to an open‑source Llama‑3 70B model (optimized for accuracy and speed) or Anthropic’s Claude 4.5 Sonnet model. Queries sent to Claude generate charges directly to the customer’s Claude API account, while Cognosa saves and logs all queries and responses for later review.
Illustrative Token Costs
As of this writing, Anthropic’s pricing for the Sonnet 4 family (200k‑token context window) is approximately:
• $3 per 1M input tokens
• $15 per 1M output tokens
A typical context‑augmented query may include about 3,000 input tokens (a short question plus relevant retrieved context) and return about 1,000 output tokens. At six such queries per workday, this usage comes to roughly 126.72 USD per month—regardless of whether it is one person making multiple queries or several users making fewer queries each.
By contrast, a direct LLM query (no contextual augmentation) might use only 450 tokens in and 350 tokens out, totaling an estimated 34.85 USD per month for the same number of daily queries.
Examples
The figures below shows how Cognosa displays both types of queries:
a) direct questions to an LLM, and
b) augmented questions answered using your organization’s specialized data collections.
Query without context:
Query with context: (1 of 3)
Query with context: (2 of 3)
Query with context: (3 of 3)
Cognosa solutions v. Custom GPTs
One reason we created the Cognosa platform was that while some customers had quick success using Custom GPTs in OpenAI’s ecosystem, over time they found that the no-code approach could not answer certain queries reliably, could not scale to support larger or more complex datasets, or became very expensive at higher usage levels.
This post compares OpenAI’s GPT Builder — how it works, what it does well, and where it differs from a purpose-built Retrieval-Augmented Generation (RAG) solution like Cognosa.
Setting the stage: Custom GPTs vs. a purpose-built RAG solution
Many organizations experimenting with AI start by testing Custom GPTs inside the OpenAI ecosystem. It’s an appealing option: upload some files, add instructions, and instantly get a conversational assistant that seems aware of your documents.
But the simplicity of a Custom GPT comes with inherent limitations. It offers convenience, not full control.
A purpose-built RAG solution — like the one Cognosa provides — takes a different approach. Your data, structure, semantics, indexing strategy, and retrieval logic become first-class components, giving you transparency, customization, and scalability that a generic builder cannot match.
Before jumping to the side-by-side comparison, it helps to understand what Custom GPTs actually do — and just as important, what they don’t.
What a Custom GPT Does Provide
Same base model, customized behavior
A Custom GPT uses the same underlying LLM that powers ChatGPT. You supply instructions and files, but you do not retrain or fine-tune the model’s weights.
Knowledge file uploads
You can upload PDFs, spreadsheets, or text files so the GPT can reference them during a conversation — e.g., “Use these spec sheets when answering product questions.”
Custom instructions
You define how the GPT should behave, how it should use its knowledge files, and what persona or style it should adopt.
Optional tools, actions, and APIs
You can enable external API calls, web browsing, or code execution to extend the assistant’s capabilities.
Internal retrieval from uploaded files
At runtime, the GPT can draw from your uploaded files for additional context. They serve as an internal reference set, not as training data.
No use of your data for public training
OpenAI states that Custom GPTs are not used to train or improve OpenAI’s public models
What a Custom GPT does not (or at least not clearly) do
It is not fine-tuning.
Uploading files does not train new model weights. The GPT uses your files as additional context, not as learned parameters.
It does not replace a full RAG architecture.
You cannot control chunking, embeddings, vector indexes, or retrieval pipelines. The internal mechanics aren’t exposed and may not resemble a traditional RAG pipeline.
It still inherits normal LLM limitations.
Hallucinations, context-window limits, and confusion between similar content can still occur without careful structuring.
It does not permanently embed your data
The GPT uses the base model’s tokenizer and inference engine. Your “knowledge files” act as temporary reference material, not integrated training data.
RAG vs Custom GPT — how they compare
Below is a side-by-side summary of how Cognosa’s RAG-first approach compares with OpenAI’s Custom GPT builder.
| Solution Area | Cognosa / RAG Approach | Custom GPT (OpenAI) |
|---|---|---|
| Data ingestion & indexing | Purpose-built vector database, tuned chunking, overlaps, similarity thresholds, and retrieval pipelines designed for your data and query patterns. | Upload files within limits; generic ingestion and chunking with minimal visibility or control. |
| Granular control over retrieval | Full control of embeddings, vector size, retrieval type, chunk size, and parameters. Tuned during customer validation. | Only files + instructions; embedding and retrieval logic remain opaque. |
| LLM usage | Use open-source or commercial LLMs; deploy in our cloud, your cloud, or on-prem. Open-source avoids per-token fees. | Tied to OpenAI’s API pricing; enterprise costs can be significant. |
| Model fine-tuning | Optional fine-tuning of open-source models; or use smaller models when retrieval matters more than reasoning. | No fine-tuning; behavior controlled only through prompts and context. |
| Latency & maintenance | Performance based on chosen infra; small 24×7 workloads run for low hundreds/month. | Minimal ops overhead; OpenAI manages everything. |
| Traceability & control | Transparent retrieval with logs, auditing, and explicit control over data access. | Opaque retrieval; limited insight into contributing content. |
| Setup speed | RAG is complex to build alone, but Cognosa provides a turnkey managed setup. | Very fast: upload files + instructions and begin testing. |
| Operational cost | Predictable: setup + monthly cost based on hosting, model, and usage. | Ongoing OpenAI token charges; low barrier to start, expensive at scale. |
Next time, we’ll take an advanced look at some of how this magic works: the actual UI that gives advanced users access to alternative retrieval strategies and parameters!
Cognosa in the Spotlight: Why AI-as-a-Service Might Be Right for Your Business
It all begins with an idea.
Intro
Every few years, a technology shift quietly changes how work gets done.
AI-driven automation—especially Retrieval-Augmented Generation (RAG)—is one of those shifts. Yet many organizations aren’t benefiting from it. Not because the technology isn’t ready, but because their own data isn’t being used to make AI truly useful.
That’s the gap CogWrite Semantic Technologies built Cognosa to close.
Welcome to Cognosa in the Spotlight, a new blog series that explains AI-as-a-Service in practical, business-friendly terms—and shows how Cognosa will soon help small and mid-sized organizations put their own knowledge to work.
Why generic AI isn’t enough for real business tasks
ChatGPT-style assistants are impressive, but they do not:
know your company policies
know your products or parts inventory
understand your compliance rules
reflect your training materials
capture years of institutional knowledge or workflows
Without your specialized data, a general-purpose model can only give general-purpose answers.
What changes everything is AI that’s augmented with your private, domain-specific knowledge.
The power of AI + your proprietary data (RAG)
Retrieval-Augmented Generation (RAG) lets an AI system draw on your actual documents, procedures, manuals, specifications, and knowledge bases—while keeping that data private and secure.
This unlocks practical, high-value use cases such as:
Customer service augmentation – Answer customer questions from your real documentation, accurately, 24/7.
Compliance guidance – Give staff precise, policy-correct answers in regulated environments.
Technical support & troubleshooting – Integrate repair manuals, parts catalogs, service history, and SOPs.
Employee onboarding & training – Help new hires ramp faster with an “expert in chat form.”
Knowledge management & internal search – Replace clunky portals with natural-language answers.
In short: RAG turns your company’s documents into an always-available expert.
Why “AI-as-a-Service” matters
Most businesses don’t want to build and maintain their own:
vector databases
embedding pipelines
content “chunking” and indexing
API orchestration layers
prompt templates
monitoring and feedback loops
They do want the benefits.
AI-as-a-Service means you get the capability without the infrastructure, engineering overhead, or specialized in-house team. Cognosa’s mission is to make this accessible, reliable, and turnkey.
Where Cognosa is today—and what this blog will cover
CogWrite Semantic Technologies is approximately two months away from onboarding new beta users for Cognosa, our AI-as-a-Service platform.
Over the coming weeks, Cognosa in the Spotlight will explore:
what problems AI can realistically solve
how RAG works under the hood
how businesses can prepare their data
where AI provides immediate ROI—and where caution is warranted
practical examples of real-world solutions
how small and medium-sized teams can deploy AI effectively
Our goal is simple:
Help you identify the points in your workflow where an AI assistant—enhanced with your proprietary knowledge—can save time, reduce cost, and improve accuracy.
In future posts, we’ll walk through real-world solutions made possible by platforms like Cognosa and show how they can be implemented cost-effectively—without spending hundreds of thousands of dollars or paying $15,000/month for enterprise LLM access.