AI Knowledge Agent

Overview

The AI Knowledge Agent turns your knowledge base into a 24/7 support agent that answers customer questions across every channel — email, WhatsApp, Instagram, Facebook Messenger, Telegram, SMS, Slack, and the web widget. It is built on top of the existing Knowledge Base (RAG) and AI Replies modules, adding four customer-facing capabilities and an admin Insights dashboard.

Where the base AI Replies feature sends one-off generated answers, the Knowledge Agent feature adds the missing pieces that turn that into a sellable, measurable customer-support replacement:

• Visible citations — every answer cites the source document the agent used (clickable links to the original page or doc).
• Graceful fallback — when confidence is low, the agent says "I don’t know” instead of hallucinating.
• Customer thumbs feedback — visitors rate every reply 👍 or 👎 across all channels via signed public links.
• Insights dashboard — deflection rate, knowledge gaps, top sources, question clusters, hours saved.
• Per-workspace quotas — monthly question limit and per-answer cost cap so resellers can sell tiered access.

How It Works End-to-End

A single customer message flows through these stages:

1. Inbound — the customer sends a message on any channel.
2. Quota check — if the workspace exceeded its monthly KB question limit, retrieval is skipped and the canned fallback is returned.
3. Retrieval — RAGService searches the knowledge base (Pinecone vector search if configured, MySQL FULLTEXT otherwise) and returns the top-K matching chunks with their document title and URL.
4. Prompt assembly — AIManager builds a system prompt that includes the retrieved sources numbered [1], [2], … and instructs the LLM to cite them inline.
5. LLM call — the configured provider (OpenAI / Anthropic / Gemini / Mistral) generates a draft answer.
6. Confidence gate — if heuristic confidence falls below the workspace threshold OR no chunks fed the answer, the canned fallback message replaces the LLM output. The raw LLM text is preserved in ai_original_content for audit.
7. Per-channel render — AIResponse::customerContent($format) produces an HTML body for email, a plain body for WhatsApp/SMS/Telegram/Slack, or a structured payload for the web widget — with the inline citations footer appended.
8. Logging — one row is written to kb_query_logs with the question, answer, cited chunk IDs, confidence, was_fallback flag, and a signed feedback token.
9. Outbound — the channel sender appends the "Was this helpful?" thumbs links (formatted for that channel) and dispatches the reply.
10. Customer rates — clicking 👍 or 👎 hits the public signed endpoint /kb/feedback/{token}/{verdict}, which updates kb_query_logs.was_helpful and shows a friendly thank-you page.

Insights Dashboard

Find the dashboard at /knowledge-base/insights — or via the "Insights" button on the Knowledge Base page header. It is the headline customer-visible feature of the Knowledge Agent module.

KPI Cards

Dashboard Panels

• Most-Asked Questions — cluster of semantically similar answered questions. Similar phrasings ("what is your refund policy" vs "what’s the refund policy") collapse to one row with a count badge.
• Top Gap Clusters — the same clustering applied to fallback-triggered questions, with a one-click "Fix" button that deep-links to the Knowledge Base Q&A form with the question pre-filled.
• Knowledge Gaps — flat list of recent unanswered questions, each with an "Add to KB" button.
• Top Sources — which KB documents are getting cited most. Helps identify dead-weight content.
• Recent Answers — live tail of Q&A pairs with admin thumbs for internal tuning.

Range and Channel Filters

The whole page can be filtered by time range (7 / 30 / 90 days) and by channel (email, WhatsApp, Instagram, widget, etc.). Channel pills only show channels that have been used in the selected window, so a workspace using only WhatsApp won’t see a noisy bar of inactive options.

Visible Citations

When the LLM uses retrieved content, it emits inline citation markers like [1], [2] after each claim. The Knowledge Agent expands these into a numbered "Sources:" footer formatted for the channel:

Only sources the LLM actually referenced are listed — if retrieval returned 5 chunks but the answer only cited 2, the footer shows just those 2. Hallucinated citation numbers that don’t match retrieved sources are silently dropped.

Toggle this globally in Settings → AI → Knowledge Agent → Show citations in replies.

Graceful Fallback

When AI confidence is below the workspace’s confidence_threshold, OR no knowledge base content matched the question, the customer sees a configured fallback message instead of a guessed answer. The default message is:

I’m not sure about that one. Would you like me to connect you with someone from our team?

The raw LLM output is still stored on the Message record (ai_original_content) so admins can review what the agent would have said.

Toggle and customize in Settings → AI → Knowledge Agent:

• Replace low-confidence answers — on/off switch.
• Fallback message — free text up to 1,000 characters. Leave blank to use the default copy.

Customer Thumbs Feedback

Every KB-grounded reply ships with a "Was this helpful?" control. The customer clicks 👍 or 👎; the click fires-and-forgets to a public signed URL; kb_query_logs.was_helpful updates; the customer sees a thank-you page.

The Public Endpoint

Route: GET /kb/feedback/{token}/{verdict}

• No authentication — the signed token is the credential.
• Throttled to 60 requests per minute per IP.
• Idempotent — re-clicking the same link sets the same value.
• Returns a small standalone HTML page (dark-mode aware) so customers landing from email links see a friendly confirmation, not the MailTrixy login screen.

Token Security

Tokens follow the format {kb_query_logs.id}.{16 hex chars} where the hex is the first 16 characters of sha256_hmac(APP_KEY, "kb_query_log:{id}"). That binds each token to a specific log row, and the only mutation possible is flipping a single boolean column — so even a leaked token has near-zero blast radius.

Per-Channel Rendering

• Email — styled HTML block at the bottom of the message with 👍 / 👎 link buttons. Auto-injected by EmailSendService::injectKbFeedback().
• Widget — inline buttons rendered by widget/v1.js from the structured ai object in the chat history response.
• WhatsApp / SMS / Telegram / Slack — plain-text footer appended to the body with the two URLs.

Quotas and Cost Caps

The Knowledge Agent has its own metered limits separate from the workspace-global monthly_cost_limit. This lets resellers sell the Knowledge Agent feature as a tiered upsell without affecting the rest of the AI allowance.

Both caps are hard gates — once hit, the customer sees the fallback regardless of whether Replace low-confidence answers is enabled.

Channel Support Matrix

Database Schema

The feature adds one new table and column-set additions to two existing tables:

kb_query_logs (new)

• One row per AI reply that consulted the knowledge base.
• Columns: workspace_id, conversation_id, channel, question, answer, cited_chunk_ids (JSON), confidence, was_fallback, was_helpful, provider, model, tokens_in, tokens_out, cost_usd.
• Indexed on (workspace_id, created_at), (workspace_id, was_fallback), (workspace_id, confidence).

messages (added columns)

• ai_was_fallback (bool, nullable)
• ai_feedback_token (string 64, nullable)
• ai_cited_sources (JSON, nullable)
• ai_original_content (text, nullable)

ai_configs (added columns)

• kb_show_citations (bool, default true)
• kb_fallback_enabled (bool, default true)
• kb_fallback_message (text, nullable)
• kb_monthly_question_limit (uint, default 0 = unlimited)
• kb_per_answer_cost_cap (decimal 8,4, default 0 = no cap)

Key Services and Files

Configure It

1. Go to Settings → AI.
2. Scroll to the Knowledge Agent panel.
3. Toggle Show citations in replies on (default).
4. Toggle Replace low-confidence answers with a fallback message on (default).
5. Optionally customize the fallback message.
6. Optionally set a monthly question limit and per-answer cost cap.
7. Click Save Configuration.

If your workspace already has knowledge base content, you don’t need to do anything else — the next inbound message that triggers an AI reply will produce a logged answer visible in Knowledge Base → Insights.

Troubleshooting

• Insights shows 100% fallback rate. Either there is no KB content yet (the empty-state card will tell you), or every answer is below your confidence threshold. Lower the threshold temporarily under Settings → AI → Confidence Threshold to see what the LLM is actually outputting.
• Thumbs links don’t appear in email. Check that the Message has ai_feedback_token populated and ai_was_fallback is false. Tokens are not issued for fallback replies (nothing to rate) or playground/test calls.
• Widget shows duplicate "Was this helpful?" text. Make sure conversation.channel is chat or live_chat. Plain-text footer is only suppressed for those exact values; a custom channel string falls through to the plain footer.
• Citations missing even though sources came back. The LLM did not actually emit [N] markers in its answer. appendCitationsFooter() intentionally only lists sources the answer references — this is to avoid fake citations. Re-tune the system prompt or pick a stronger model.