As I continue to see former colleagues in Management Consultancies make the same error over and over I thought I would give you the solution, because frankly its embarrassing.
Problem Statement
Management Consultancies staff are being told to utilise LLM’s but the report outputs often make subject matter mistakes that create embarrassment.
Background AI-assisted Report Controversy
Fortune: Deloitte was caught using AI in a government report with fabricated references
Deloitte Australia charged AUD $290,000 for a report that used AI, and a researcher found fake citations; Deloitte later issued a partial refund after admitting use of generative AI. FortuneNDTV: Deloitte used Azure OpenAI GPT-4o in drafting, leading to fictitious references
Australian government reissued corrected report after AI-generated errors were flagged by an academic; future contracts may include strict AI clauses. www.ndtv.comAustralian Financial Review: Deloitte to refund government and admits AI errors
Confirms partial refund after acknowledgement that AI was used in drafting a consulting report. The Australian Financial Review
AI Architectures for Valid Outputs
Creating viable reports from complex data requires multiple sources, that should be a given in any consultancy environment. However the current issue is that a single source is being trusted to validate the relationships internally and without any reference points.
In the same way that Bayesian Networks uses Qualitative Data Sources (shown as people below) as Context, Meaning, Taxonomies so LLM’s can use the same architectures to deliver high quality reports. Essentially this is a complex Agent that first validates meaning the draws in Expert Knowledge to drive the outputs.
LLMs and NLMs
LLMs (large language model) combined with NLMs (narrow language model) are essential for critical tasks like Biomedical Research, Healthcare, Defence, Security, Government although the highest immediate financial benefit would be in Investment Banking, Trading Platforms and in E-commerce.
Implementing Narrow Language Models NLMs
A small fine-tuned transformer
A classifier or extractor
A rules-based or template-driven engine
A retrieval-indexed expert corpus
A prompt-constrained micro-model
Pairing LLMs and NLMs
LLMs ? breadth, fluency, reasoning
NLMs ? precision, authority, constraint
LLMs and NLMs High-Level Architecture
Large Language Model (LLM)
- Natural language interface
- Multi-step reasoning
- Conversation flow control
- Output synthesis
Narrow Language Models (NLMs)
- Authoritative domain knowledge
- Policy and compliance enforcement
- Deterministic or bounded outputs
- Reduced hallucination risk
Common NLM Implementations
An NLM in this pattern may be implemented as:
- Fine-tuned small transformer
- Domain-specific classifier
- Rules or policy engine
- Retrieval-backed expert corpus
- Prompt-constrained micro-model
Design Principle
LLMs provide linguistic intelligence; NLMs provide epistemic authority.
Epistemic authority refers to the sources or individuals that we trust for valid knowledge, influencing our beliefs and actions.
Epistemic authority plays a crucial role in how we understand and engage with information, especially in areas like science and finance.
The true value of NLMs is as External Counterpoints, not as embedded function as some LLMs operate.