> For the complete documentation index, see [llms.txt](https://docs.openmind.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.openmind.com/build/concepts/7_project_structure.md). # Project Structure Project structure and flow diagram ### Project Structure ```tree . ├── cmd/ # Main entry point │ └── main.go ├── config/ # Agent configuration files ├── internal/ # Core packages │ ├── actions/ # Action orchestrators │ ├── backgrounds/ # Background task orchestrators │ ├── cloudsession/ # Cloud session management │ ├── config/ # Configuration loading │ ├── fuser/ # Input fusion logic │ ├── geometry/ # Geometry / spatial helpers │ ├── hooks/ # Function hook registry │ ├── httpclient/ # Shared HTTP client │ ├── inputs/ # Input/sensor orchestrators │ ├── knowledgebase/ # RAG knowledge base client │ ├── llm/ # LLM integration │ ├── logger/ # Logging utilities │ ├── mcp/ # MCP client + orchestrator │ ├── memory/ # Agent memory │ ├── metrics/ # Prometheus metrics server │ ├── providers/ # I/O providers (TTS, ASR, audio) │ ├── runtime/ # Core runtime manager │ ├── tracer/ # Execution tracer │ ├── util/ # Shared utilities │ ├── vad/ # Voice activity detection │ ├── ws/ # WebSocket client │ └── zenoh/ # Zenoh integration (CDR codec, session) ├── pkg/ # Public/importable packages (e.g. memory) ├── plugins/ # Plugin implementations │ ├── actions/ # speak, emotion, navigation, unitree, etc. │ ├── backgrounds/ # Background task plugins │ ├── inputs/ # asr/, vlm/, face_presence, etc. │ └── llm/ # OpenAI, Gemini, DeepSeek, Ollama, etc. └── Makefile # Build system ``` The system is based on a loop that runs at a fixed frequency set by the config field `hertz` (`cfg.Hertz` in Go). The loop looks for the most recent data from various sources, fuses the data into a prompt (typical length \~1 paragraph), sends that prompt to one or more LLMs, and then sends the LLM responses to virtual agents or physical robots for conversion into real world actions. > **Note:** In addition to the core loop running at `hertz`, a robot will have dozens of other control loops running at rates of 50-500 Hz (for physical stabilization and motions), 2-30 Hz for sensors such as LIDARS and laserscan, 10 Hz for GPS, 50 Hz for odometry, and so forth. The `hertz` setting refers only to the basic fuser cycle that is best thought of as the refresh rate of the robot's core attention and working memory. ### Flow Diagram ![](https://2120135774-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Ft8vMsruGqqhYpVx9qhd5%2Fuploads%2Fgit-blob-3aa8c57c15a8fc307ebcc4c444c91f5f040c159f%2Fom1-flow-diagram.png?alt=media) #### Sensory Input & AI Captioning The system's perception begins with a suite of Sensory Inputs that gather data from the environment and its own internal state. These inputs are multi-modal and include: * Environmental: Audio, Video, Lidar, GPS * Platform State: Battery, Wallet This raw data is then processed by an AI Captioning layer. This layer transforms the raw sensory streams into a structured, machine-readable format. This includes: * ASR (Automatic Speech Recognition) for audio. * VLM (Vision-Language Model) for describing video feeds. * Spatial/NAV data from GPS and Lidar. * Platform state, and 3D environmental data. Example services for this stage include `GoogleASRInput` for speech and `VLMGemini` / `VLMOpenAI` for vision. #### Fuser & Cortex LLM The Fuser is a critical component that integrates the processed sensory data from the AI Captioning layer with contextual and instructional data. This second set of inputs provides the necessary context for decision-making: * Governance: System Governance, System Prompt * User Intent: Input User Prompt * Knowledge: RAG (Retrieval-Augmented Generation), Background information The fused, comprehensive context is then passed to the Cortex LLM, which serves as the central reasoning engine or "brain" of the system. This Large Language Model is responsible for understanding the situation, interpreting the user's intent, and formulating a plan of action. The architecture is modular, allowing for different LLMs to be used, such as OpenAI, Gemini, DeepSeek, Ollama (local). #### Action & Orchestration The plan generated by the Cortex LLM is defined as a set of Actions. These high-level actions are then sent to the Action Orchestrator. This module translates the abstract plan into a series of concrete, low-level commands that can be executed by the system's hardware. The Orchestrator manages various output modalities, including: * TTS (Text-to-Speech) for verbal responses. * Sound effects. * Facial Expressions. * Physical Movement. Finally, the orchestrated commands are sent to the hardware layer to Execute Command, resulting in the system performing the desired action in the real world. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.openmind.com/build/concepts/7_project_structure.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.