Enterprise AI Chatbot v2

Today, many organizations are developing generative AI chatbots for internal use – to increase operational efficiency, automate internal support workflows, and provide employees with rapid access to critical organizational knowledge. This not only reduces manual overhead but can also boost productivity across the organization. 

For enterprise organizations, the development of a GenAI Chatbot can be particularly complex. The solution must be secure, scalable, and extensible. This requires specialized knowledge and a complex architecture. While each organization has unique needs, the underlying approach and architecture design are applicable for enterprise organizations across sectors. 

AllCloud has extensive experience in the development of GenAI chatbots for enterprises. While the AllCloud AI team takes a tailored approach to developing custom solutions for each organization, there are several core components of the architecture that serve as best practices.

• • Prioritize Secure, Decoupled Architecture: Design a modular, enterprise-grade application deployed within an a secured cloud Landing Zone, ensuring independent scaling for the UI, AI agent, and external integrations. Incorporate strict security measures like Single Sign-On (SSO) and robust encryption.
  • Adopt Agent-Based Orchestration (MCP/Gateway): Implement a core Agent Service that uses a protocol like MCP and a gateway to dynamically discover and invoke tools. This abstracts the complexity of integrating with existing systems, allowing for easy future expansion.
  • Establish Automated, Factual RAG: Build a highly automated Retrieval-Augmented Generation (RAG) ingestion pipeline that processes organizational data from various sources (e.g., SharePoint) into a vector database, ensuring the chatbot is grounded in factual, quotable information.
  • Integrate Observability and Persistence: Ensure all short and long-term user context is securely retained across sessions and deploy comprehensive AI observability tools to monitor trace latency, control token costs, and fine-tune system reliability.

Numerous enterprise organizations are seeking to develop internal GenAI chatbots to enhance the efficiency of their internal support and operations. A chatbot can reduce time spent by employees searching for answers in internal documentation across various sources, such as HR or compliance and it can eliminate human error caused by misinterpretation or oversight of information. From an operational standpoint, it can reduce the time for opening a support ticket and it can also lower the total number of support tickets opened, allowing service centers to improve their efficiency and freeing them to deal with complex issues. 

To fully meet an organization’s needs, the goal is to build a high-availability, cost-effective, and scalable enterprise GenAI chatbot. It should function as a central “one stop bot” for the organization, using natural language to interact with employees, query systems, organizational knowledge bases, and internal service systems. It should allow automation of tickets and request creations for various service centers, reducing manual tickets and speeding the average resolution time. In addition, the solution should be expandable – allowing for easy integration of additional tools and systems in the future by the local development team.

For a recent engagement, the AllCloud AI team created a custom enterprise GenAI chatbot for a large healthcare organization. While the architectural core components and implementation were designed to meet the organization’s specific needs, they can serve as guidelines and best practices for other enterprise organizations seeking to develop their own solution.   

First, the chatbot was designed as a secure, modern  enterprise-grade application deployed entirely on the AWS Cloud within the existing Landing Zone for the organization. The system was built with a decoupled, modular architecture, ensuring the user interface, core AI agent reasoning, and external integrations operate and scale independently. 

Security is deeply embedded, with integrated Microsoft Entra ID via OIDC for Single Sign-On (SSO), alongside strict data encryption in transit and at rest using AWS KMS. The system utilizes the RAG (Retrieval-Augmented Generation) approach, MCP ( Model Context Protocol), AI observability, and various AWS managed services, alongside microservices architecture. 

One of the main goals of the solution was to create a user-friendly, responsive interface that guarantees security and privacy, while also being scalable to meet future development and organizational goals.

The user-facing layer is a stateless, responsive web application built using React/Next.js and deployed on an Amazon EKS cluster. It serves as the exclusive entry point for user interactions and handles authentication securely via Entra ID. The frontend streams LLM responses, maintains chat UI threads, and forwards the authenticated user’s access and ID tokens to backend services for strict authorization.

The core of the chatbot is the Agent Service, built using Python and the LangChain framework. This agent is responsible for receiving queries, determining intent, and iteratively executing actions using the various tools foundational models hosted on Amazon Bedrock.

To interact with external systems without bloating the core agent code, the architecture leverages the Model Context Protocol (MCP) to standardize tool interactions, utilizing the AWS Bedrock AgentCore Gateway to dynamically discover and invoke these MCP-based tools. The AgentCore Gateway acts as an intelligent bridge that allows the agent to dynamically discover, search, and invoke various tools. These tools are provided by dedicated MCP servers, enabling the agent to execute highly specific tasks like creating ServiceNow tickets or querying the AWS Bedrock Knowledge Base.

The goal is to separate the core logic of the chatbot from other components, and allow accessibility by using open-standard protocols for connecting to various tools and data sources. Furthermore, the Gateway is extendable, allowing for the addition of other tools in the future, without affecting the core component of the agent. In the future, the agent could connect to HR systems, IT systems, and other core business areas, while maintaining tight control over authorization, allowing for different levels of authorization for accessing specific information or data.

To facilitate seamless system interactions, a specialized ServiceNow Facade service abstracts the complexities of the ServiceNow API. This dedicated layer empowers employees to manage service tickets directly through the chat interface, enabling them to verify the status of current tickets or generate new support incidents instantly. 

The specific ServiceNow configuration implemented ensures that the chatbot leverages the authenticated user’s identity to execute API calls within ServiceNow directly. This creates a modular, extensible integration layer that allows the organization to safely connect to any external API or data source.

To ground the chatbot in factual organizational data, the solution features a highly automated document ingestion pipeline. The flow operates as follows:

1. Data Extraction: Amazon AppFlow executes on a daily schedule to extract newly created or modified documents from SharePoint, storing the raw files and metadata into S3 bucket.
2. Event-Driven Processing: An S3 event triggers an AWS Step Functions orchestration workflow using Amazon EventBridge.
3. Format Conversion: Specialized AWS Lambda functions within the Step Function automatically parse and convert disparate file types (like .msg emails, images, or .pptx presentations) into clean PDF or Markdown formats suitable for vectorization. It also ensures files are accurately prepared and adapted to be included in the Knowledge Base.
4. Vectorization & Storage: Processed documents are stored in S3 bucket and synced to an AWS Bedrock Knowledge Base. Bedrock automatically chunks the text and utilizes an Amazon OpenSearch Serverless (Vector Engine) database to store and manage the high-dimensional embeddings.When users ask policy questions, the agent queries and returns answers complete with direct citations and links back to the original SharePoint documents.

Persistence and observability are crucial for ensuring the effectiveness of the chatbot, both to improve the content it provides users, as well as to understand technical considerations, such as latency, and improve the responsiveness of the system. AI observability also supports effective FinOps, so an organization can understand how much it is spending on interactions.

Because the frontend is stateless, all memory – including short-term session data, long-term user chat history, and thread history is securely persisted in an Amazon Aurora PostgreSQL database with the PGVector extension. To optimize system responsiveness, AWS ElastiCache for Redis manages the caching of prompts and LLM outputs, effectively reducing latency for repeat interactions.

Finally,to guarantee enterprise reliability and monitor agent interaction quality, the architecture incorporates Arize Phoenix, deployed directly onto the EKS cluster. This provides deep observability, allowing administrators to track end-to-end trace latency, monitor token consumption for cost control, and fine-tune system prompts.

By leveraging AWS native services, GenAI frameworks, and a highly modular MCP-driven design, an enterprise organization can build a chatbot that goes beyond simple Q&A. It can create a robust digital assistant prepared to scale to thousands of users, securely executing complex workflows, while keeping organizational and operational data readily accessible. Furthermore, by abstracting system complexity, the modular architecture approach makes it easy for developers to extend the system and integrate new tools in the future, to expand the chatbot’s capabilities and meet the growing needs of the organization.

Ready to build or scale your enterprise GenAI solution?
Contact us today to design a secure, custom architecture tailored to your organization’s needs.