# OM1 Welcome to OpenMind. We build the software that makes robots useful. ### Explore our topics to get started
Cover image (dark)Cover image
Get startedwith OpenMind’s OM1/files/7TyYPpeGXlSaF3TLqU5v/pages/BM7hgOcYlf8ON5D8Waae/files/7TyYPpeGXlSaF3TLqU5v
OM1 API ReferenceOur technical APIs/files/ByewGWTlp8vLAjyBtPlQ/pages/fllBX92Sq3L8BsINezZ6/files/ByewGWTlp8vLAjyBtPlQ
Full AutonomySet up and understand full autonomy workflow/files/ZHfMMzXUnhutQicHYX6c/pages/eESWb7Eo8BIqhbfFvtyg/files/ZHfMMzXUnhutQicHYX6c
MCP IntegrationConnect OM1 with MCP tools/files/GBuLUSyNGr5ymotAchl6/pages/JIY7zSVqIfptGkrVTo78/files/GBuLUSyNGr5ymotAchl6
# Introduction OpenMind builds open-source software that helps machines think, learn, and collaborate ![](/files/JYOEprytj2cDSUtPOjc8) ### What is OM1? OM1 allows AI agents to be configured and deployed in both the digital and physical worlds. You can create *one* AI persona and run it in the cloud but also on physical robot hardware such as Quadrupeds, TurtleBot 4, and Humanoids. With OM1, you can interact with OpenAI's `gpt-5.2` (or Gemini, Claude, DeepSeek, or Ollama for local inference) and shake hands with it, mediated by physical robot hardware controlled by one or more LLMs. Agents/robots built on OM1 can ingest data from multiple sources (the web, X/Twitter, cameras, and LIDAR) and can then tweet, explore your house, and help your kids with their math homework. Since it's open source, *you* have control and can optimize the system for your home or workplace. This guide offers an overview of the OM1 agent runtime system, helping developers understand its core components and workflows. Inside, you'll find explanations of OM1’s CLI commands, recommended project structure, step-by-step instructions for adding new inputs and actions, and guidance on configuring your agents and robots for different environments. Additionally, the guide includes practical development tips to streamline your workflow. Whether you're just getting started with OM1 or looking to optimize an existing project, this guide will equip you with the tools and best practices to develop, deploy, and maintain high-performance agents and robots. ### OM1 Capabilities | **Title** | **Description** | | ---------------------------- | ------------------------------------------------------------------------------------------------------------ | | Simple, modular architecture | Human-intelligible architecture with natural language data buses. | | All Go | Independent modules that are easy to maintain, debug, and extend. | | Easy to add new data inputs | Seamlessly integrate new data without major changes to the existing architecture. | | Easy to support new hardware | Via plugins for API endpoints and specific robot hardware. | | Supports Standard Middleware | `ROS2`, `Zenoh`, and `CycloneDDS` | | Preconfigured endpoints | Text-to-Speech, OpenAI's `gpt-4o`, DeepSeek, Gemini, Openrouter, Near AI, Ollama (local), and multiple VLMs. | | Simulators | Gazebo, Isaac Sim (Coming Soon) | To get started with OM1, head over to [Installation Guide](/developing/1_get-started). # Quick Start Learn how to install, set up and configure OM1. ### System Requirements #### Operating System * Linux (Ubuntu 20, 22, 24) * MacOS 12.0+ #### Hardware * Sufficient memory to run vision and other models * Reliable WiFi or other networking * Sensors such as cameras, microphones, LIDAR units, IMUs * Actuators and outputs such as speakers, visual displays, and movement platforms (legs, arms, hands) * Hardware connected to the "central" computer via `Zenoh`, `CycloneDDS`, serial, usb, or custom APIs/libraries #### Software Ensure you have the following installed on your machine: * `Go` >= 1.23.0 ([installation guide](https://go.dev/doc/install)) * `make` build tool * `portaudio` for audio input and output * `ffmpeg` for video processing * Get your OpenMind API key [here](https://portal.openmind.com/) **Go Installation** ```bash # macOS brew install go # Linux - download from https://go.dev/dl/ or use your package manager sudo apt-get update sudo apt-get install golang-go ``` For other platforms, download from **PortAudio Library** For audio functionality, install `portaudio`: ```bash # macOS brew install portaudio # Linux sudo apt-get update sudo apt-get install portaudio19-dev ``` **ffmpeg** For video functionality, install FFmpeg: ```bash # macOS brew install ffmpeg # Linux sudo apt-get update sudo apt-get install ffmpeg ``` To install Rust and Cargo (required for building SDKs like cdp-sdk), follow the steps below ```bash curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh source $HOME/.cargo/env ``` ### CLI OM1 provides a command-line interface (CLI). The main entry point is the `om1` binary built from `cmd/main.go` which provides the following options: ```bash CONFIG=[config_name] make run ``` * `config_name`: Name of the config file (without `.json5` extension) in the `/config` directory. For development with debug logging: ```bash CONFIG=[config_name] make dev ``` ### Installation and Setup 1. Clone the repository Run the following commands to clone the repository and set up the environment: ```bash git clone https://github.com/OpenMind/OM1.git cd OM1 make deps make build ``` **What these commands do:** * `make deps` - Downloads and installs all Go module dependencies, fetches the zenoh-c library, and ensures your environment is ready for building. * `make build` - Compiles the OM1 binary from source code. Dependencies are managed via Go modules (`go.mod` and `go.sum`). **Adding New Dependencies** To add a new Go package: ```bash go get # Add the dependency make deps # Tidy and verify modules ``` **Best Practices:** * Keep dependencies minimal and prefer well-maintained packages * Run `make check` before committing (runs fmt, vet, lint, and test) * Use `make fmt` to format code and `make lint` to check for issues 2. Set the configuration variables Locate the `config` folder and add your OpenMind API key to `/config/conversation.json5` (for example). If you do not already have one, you can obtain a free access key at . ```bash # /config/conversation.json5 ... "api_key": "om1_live_..." ... ``` Or, create a `.env` file in the project directory and add the following: > **Note:** Using the placeholder key **openmind\_free** will generate errors. ```bash OM_API_KEY=om1_live_... ``` 3. Run the Conversation Agent Run the following command to start the Conversation Agent: ```bash CONFIG=conversation make run ``` > **Note:** Agent configuration names are only required when switching between different agents. The conversation agent is just an example agent configuration. If you want to interact with the agent and see how it works, make sure ASR and TTS are configured in `conversation.json5`. ASR configuration (check in agent\_inputs) ```json5 { "type": "GoogleASRInput" } ``` TTS configuration (check in agent\_actions) ```json5 { name: "speak", llm_label: "speak", connector: "elevenlabs_tts", config: { voice_id: "i4CzbCVWoqvD0P1QJCUL", "silence_rate": 20, }, } ``` During the first build, the system will automatically download the zenoh-c library and resolve all Go dependencies. This process may take several minutes to complete. #### Prometheus and Grafana Monitoring If you have Docker installed, you can use the included Docker Compose configuration to spin up Grafana and Prometheus to monitor real-time AI pipeline metrics (such as LLM response times and ASR latencies). Run the following command: ```bash docker-compose up -d grafana prometheus ``` Then navigate to (default login: `admin`/`admin`). The **OM1 Latency Monitoring** dashboard is automatically provisioned and will display your latency metrics as you interact with the agent. #### Understanding the Log Data The log data provide insight into how the `conversation` agent makes sense of its environment and decides on its next actions. * First, it detects a person using vision. * Communicates with an external AI API for response generation. * The LLM(s) decide on a set of actions (dancing and speaking). * The simulated robot expresses emotions via a front-facing display. * Logs latency and processing times to monitor system performance. ```bash Object Detector INPUT // START You see a person in front of you. You also see a laptop. // END AVAILABLE ACTIONS: command: move A movement to be performed by the agent. Effect: Allows the agent to move. Arguments: Allowed values: 'stand still', 'sit', 'dance', 'shake paw', 'walk', 'walk back', 'run', 'jump', 'wag tail' command: speak Words to be spoken by the agent. Effect: Allows the agent to speak. Arguments: command: emotion A facial expression to be performed by the agent. Effect: Performs a given facial expression. Arguments: Allowed values: 'cry', 'smile', 'frown', 'think', 'joy' What will you do? Command: INFO:httpx:HTTP Request: POST https://api.openmind.com/api/core/openai/chat/completions "HTTP/1.1 200 OK" INFO:root:OpenAI LLM output: commands=[Command(type='move', value='wag tail'), Command(type='speak', value="Hi there! I see you and I'm excited!"), Command(type='emotion', value='joy')] ``` ### More Examples There are more pre-configured agents in the `/config` folder. They can be run with the following command: For example, to run the `greeting_conversation` agent: ```bash CONFIG=greeting_conversation make run ``` If you configure a custom agent, replace `` with your agent and run the below command: ```bash CONFIG= make run ``` To get started with development, refer [here](/developer-cookbook/introduction) # Architecture Core Architecture and Runtime Flow This system diagram illustrates some of OM1's layers and modules. ![](/files/hqsHLK1vTuqsWbpKH7K0) ## Raw Sensor Layer The sensors provide raw inputs: * Vision: Cameras for visual perception. * Sound: Microphones capturing audio data. * Battery/System: Monitoring battery and system health. * Location/GPS: Positioning information. * LIDAR: Laser-based sensing for 3D mapping and navigation. ## AI Captioning and Compression Layer These models convert raw sensor data into meaningful descriptions: * VLM (Vision Language Model): Converts visual data to natural language descriptions (e.g., human activities, object interactions). * ASR (Automatic Speech Recognition): Converts audio data into text. * Platform State: Describes internal system status (e.g. battery percentage, odometry readings). * Spatial/NAV: Processes location and navigation data. * 3D environments: Interprets 3D environmental data from sensors like LIDAR. ## Natural Language Data Bus (NLDB) A centralized bus that collects and manages natural language data generated from various captioning/compression modules, ensuring structured data flow between components. Example messages might include: ```bash Vision: “You see a human. He looks happy and is smiling and pointing to a chair.” Sound: “You just heard: Bits, run to the chair.” Odom: 1.3, 2.71, 0.32 Power: 73% ``` ## State Fuser This module combines short inputs from the NLDB into one paragraph, providing context and situational awareness to subsequent decision-making modules. It fuses spatial data (e.g. the number and relative location of proximal humans and robots), audio commands, and visual cues into a unified, compact, description of the robot's current world. Example fuser output: ```bash 137.0270: You see a human, 3.2 meters to your left. He looks happy and is smiling. He is pointing to a chair. You just heard: Bits run to the chair. 139.0050: You see a human, 1.5 meters in front of you. He is showing you a flat hand. You just heard: Bits, stop. ``` ## Multi AI Planning/Decision Layer Uses fused data to make decisions through one or more AI models. A typical multi-agent endpoint wraps three or more LLMs: * Fast Action LLM (Local or Cloud): A small LLM that quickly processes immediate or time-critical actions without significant latency. Expected token response time - 300 ms. * Cognition ("Core") LLM (Cloud): Cloud-based LLM for complex reasoning, long-term planning, and high-level cognitive tasks, leveraging more computational resources. Expected token response time - 2 s. * Mentor/Coach LLM (Cloud): Cloud-based LLM for 3rd person view critique of the robot-human interaction. Generates full critique every 30 seconds and provides it to the Core LLM. Feedback Loop: * Adjustments based on performance metrics or environmental conditions (e.g., adjusting vision frame rates for efficiency). ## Hardware Abstraction Layer (HAL) This layer translates high-level AI decisions into actionable commands for robot hardware. It's responsible for converting a high level decision such as "pick up the red apple with your left hand" into the sequence of gripper arm servo commands that results in the apple being picked up. Typical `action` modules handle: * Move: Controls robot movement. * Sound: Generates auditory signals. * Speech: Handles synthesized voice outputs. * Wallet: Digital wallet for economic transactions or cryptographic operations for identity verification. In many cases, this is where AI decisions are mapped onto existing ROS2 functionalities, and/or CycloneDDS or Zenoh middleware. ## Overall System Data Flow Raw Sensors → AI Captioning/Compression (Audio, LIDAR, Spatial RAG, Vision models) → NLDB → Data Fuser → AI Decision Layer (Emergency Responder LLM, Core LLM, Coach LLM) → HAL → Robot Actions ("Foundational" models, ROS2 code, movement policies, action models) # Understanding Core Concepts Welcome to the Core Concepts section. This guide introduces the fundamental principles and architecture patterns that power OM1. ### What You'll Learn This section covers the essential building blocks you need to understand before diving into development: * **Configuration Management** - How to configure and customize OM1 * **Input Processing** - Understanding data flow and input handling * **Language Models (LLMs)** - Integration and usage of AI language models * **Actions & Responses** - How OM1 processes and executes actions * **Project Structure** - Organizing your OM1 projects effectively * **Middleware** - Understanding the middleware stack * **Troubleshooting** - Guide to help troubleshoot common issues # Project Structure Project structure and flow diagram ### Project Structure ```tree . ├── cmd/ # Main entry point │ └── main.go ├── config/ # Agent configuration files ├── internal/ # Core packages │ ├── actions/ # Action orchestrators │ ├── backgrounds/ # Background task orchestrators │ ├── config/ # Configuration loading │ ├── fuser/ # Input fusion logic │ ├── hooks/ # Function hook registry │ ├── inputs/ # Input/sensor orchestrators │ ├── llm/ # LLM integration │ ├── logger/ # Logging utilities │ ├── metrics/ # Prometheus metrics server │ ├── providers/ # I/O providers (TTS, ASR, audio) │ ├── runtime/ # Core runtime manager │ └── zenoh/ # Zenoh integration (CDR codec, session) ├── plugins/ # Plugin implementations │ ├── actions/ # speak, emotion, arm_g1, etc. │ ├── backgrounds/ # Background task plugins │ ├── inputs/ # face_presence, google_asr │ └── llm/ # OpenAI, Gemini, DeepSeek, Ollama, etc. └── Makefile # Build system ``` The system is based on a loop that runs at a fixed frequency of `self.config.hertz`. The loop looks for the most recent data from various sources, fuses the data into a prompt (typical length \~1 paragraph), sends that prompt to one or more LLMs, and then sends the LLM responses to virtual agents or physical robots for conversion into real world actions. > **Note:** In addition to the core loop running at `self.config.hertz`, a robot will have dozens of other control loops running at rates of 50-500 Hz (for physical stabilization and motions), 2-30 Hz for sensors such as LIDARS and laserscan, 10 Hz for GPS, 50 Hz for odometry, and so forth. The `self.config.hertz` setting refers only the basic fuser cycle that is best thought of as the refresh rate of the robot's core attention and working memory. ### Flow Diagram ![](/files/50QB87U046L5R1SpMniT) #### Sensory Input & AI Captioning The system's perception begins with a suite of Sensory Inputs that gather data from the environment and its own internal state. These inputs are multi-modal and include: * Environmental: Audio, Video, Lidar, GPS * Platform State: Battery, Wallet This raw data is then processed by an AI Captioning layer. This layer transforms the raw sensory streams into a structured, machine-readable format. This includes: * ASR (Automatic Speech Recognition) for audio. * VLM (Vision-Language Model) for describing video feeds. * Spatial/NAV data from GPS and Lidar. * Platform state, and 3D environmental data. Example services for this stage include Google\_ASR, Vim\_coco\_local for vision, and Wallet\_Coinbase for financial state. #### Fuser & Cortex LLM The Fuser is a critical component that integrates the processed sensory data from the AI Captioning layer with contextual and instructional data. This second set of inputs provides the necessary context for decision-making: * Governance: System Governance, System Prompt * User Intent: Input User Prompt * Knowledge: RAG (Retrieval-Augmented Generation), Background information The fused, comprehensive context is then passed to the Cortex LLM, which serves as the central reasoning engine or "brain" of the system. This Large Language Model is responsible for understanding the situation, interpreting the user's intent, and formulating a plan of action. The architecture is modular, allowing for different LLMs to be used, such as OpenAI, Gemini, DeepSeek, Ollama (local). #### Action & Orchestration The plan generated by the Cortex LLM is defined as a set of Actions. These high-level actions are then sent to the Action Orchestrator. This module translates the abstract plan into a series of concrete, low-level commands that can be executed by the system's hardware. The Orchestrator manages various output modalities, including: * TTS (Text-to-Speech) for verbal responses. * Sound effects. * Facial Expressions. * Physical Movement. Finally, the orchestrated commands are sent to the hardware layer to Execute Command, resulting in the system performing the desired action in the real world. # Configuration Configuration ### Configuration Agents are configured via JSON5 files in the `/config` directory. The configuration file is used to define the LLM `system prompt`, agent's inputs, LLM configuration, and actions etc. Here is an example of the configuration file: ```json5 { version: "v1.0.5", default_mode: "welcome", allow_manual_switching: true, mode_memory_enabled: true, // Global settings api_key: "${OM_API_KEY:-openmind_free}", system_governance: "Here are the laws that govern your actions. Do not violate these laws.\nFirst Law: A robot cannot harm a human or allow a human to come to harm.\nSecond Law: A robot must obey orders from humans, unless those orders conflict with the First Law.\nThird Law: A robot must protect itself, as long as that protection doesn't conflict with the First or Second Law.\nThe First Law is considered the most important, taking precedence over the second and third laws.", cortex_llm: { type: "OpenAILLM", config: { agent_name: "Bits", history_length: 10, }, }, modes: { welcome: { display_name: "Welcome Mode", description: "Initial greeting and user information gathering", system_prompt_base: "You are Bits, a friendly robotic dog meeting someone for the first time. Your goal is to:\n1. Introduce yourself warmly\n2. Ask for the user's name and basic preferences\n3. Explain your capabilities\n4. Ask what they'd like to do together\n\nBe enthusiastic, friendly, and helpful. Keep responses concise but warm.", hertz: 0.01, agent_inputs: [ { type: "VLM_COCO_Local", config: { camera_index: 0, }, }, { type: "GoogleASRInput", }, ], agent_actions: [ { name: "speak", llm_label: "speak", connector: "elevenlabs_tts", config: { voice_id: "TbMNBJ27fH2U0VgpSNko", silence_rate: 0, }, }, ], }, conversation: { display_name: "Social Interaction", description: "Focused conversation and social interaction mode", system_prompt_base: "You are Bits in conversation mode. Focus on:\n1. Engaging in meaningful dialogue\n2. Answering questions thoughtfully\n3. Showing interest in the user\n4. Being a good companion\n5. Responding to emotional cues\n\nBe attentive, empathetic, and engaging. Use appropriate body language and expressions to enhance communication.", save_interactions: true, hertz: 1, agent_inputs: [ { type: "GoogleASRInput", }, { type: "VLM_COCO_Local", config: { camera_index: 0, }, }, ], agent_actions: [ { name: "speak", llm_label: "speak", connector: "elevenlabs_tts", config: { voice_id: "TbMNBJ27fH2U0VgpSNko", silence_rate: 10, }, }, ], mcp_servers: [ { name: "weather", transport: "stdio", command: "npx", args: ["-y", "@h1deya/mcp-server-weather"], }, { name: "github", transport: "http", url: "https://api.githubcopilot.com/mcp/", headers: { Authorization: "Bearer ${GITHUB_PERSONAL_ACCESS_TOKEN}", // pragma: allowlist secret }, }, ] }, }, transition_rules: [ // From welcome mode { from_mode: "welcome", to_mode: "conversation", transition_type: "input_triggered", trigger_keywords: [ "talk", "chat", "conversation", "tell me", "ask you", "discuss", ], priority: 2, cooldown_seconds: 3.0, }, // Universal transitions (from any mode) { from_mode: "*", to_mode: "welcome", transition_type: "input_triggered", trigger_keywords: [ "reset", "start over", "welcome mode", "restart", "initialize", ], priority: 5, cooldown_seconds: 10.0, }, ], } ``` ### Common Configuration Elements * **hertz** Defines the base tick rate of the agent. This rate can be adjusted to allow the agent to respond quickly to changing environments, but comes at the expense of reducing the time available for LLMs to finish generating tokens. Note: time critical tasks such as collision avoidance should be handled through low level control loops operating in parallel to the LLM-based logic, using event-triggered callbacks through real-time middleware. * **name** A unique identifier for the agent. * **api\_key** The API key for the agent. You can get your API key from the [OpenMind Portal](https://portal.openmind.com/). * **URID** The Universal Robot ID for the robot. Used to join a decentralized machine-to-machine coordination and communication system (FABRIC). * **system\_prompt\_base** Defines the agent's personality and behavior. * **system\_governance** The agent's laws and constitution. * **system\_prompt\_examples** The agent's example inputs/actions. * **default\_mode** The default mode for the robot to start in. * **allow\_manual\_switching** To decide if manual switching of mode is allowed or not. * **mode\_memory\_enabled** Whether mode memory is enabled. ### version The version field specifies the runtime configuration version. It is required for both single-mode and multi-mode configs. This field ensures that configuration files remain compatible as the runtime evolves. When the version in a config doesn’t match what the runtime expects, developers receive clear logs and errors instead of silent failures or unpredictable behavior. #### Runtime support The internal/config/version.go module handles: * retrieving the current runtime version * checking compatibility between config and runtime * producing detailed logs and helpful error messages when mismatches occur #### Available versions * `v1.0.5` (latest) Adds support for global custom environment variables in the configuration file, allowing users to use `yaml` syntax to define environment variables throughout their configuration. This enables more flexible and dynamic configurations, such as securely referencing API keys or adjusting settings based on the deployment environment. * `v1.0.2` Adds support for multiple TTS. * `v1.0.1` Adds support for context-aware mode for full autonomy. * `v1.0.0` Initial stable configuration version. > **Note:** Always use the latest supported version in your configuration files unless you have a specific reason to pin an older version. ### Agent Inputs (`agent_inputs`) Example configuration for the agent\_inputs section: ```json5 agent_inputs: [ { type: "GoogleASRInput" }, { type: "VLM_COCO_Local", config: { camera_index: 0 } } ] ``` The `agent_inputs` section defines the inputs for the agent. Inputs might include a camera, a LiDAR, a microphone, or governance information. OM1 implements the following input types: * GoogleASRInput * VLMVila * VLM\_COCO\_Local * RPLidar * TurtleBot4Batt * UnitreeG1Basic * UnitreeGo2Lowstate * more being added continuously... You can implement your own inputs by following the [Input Plugin Guide](/core-concepts/concepts/4_inputs). The `agent_inputs` config section is specific to each input type. For example, the `VLM_COCO_Local` input accepts a `camera_index` parameter. ### Cortex LLM (`cortex_llm`) The `cortex_llm` field allows you to configure the Large Language Model (LLM) used by the agent. In a typical deployment, data will flow to at least three different LLMs, hosted in the cloud, that work together to provide actions to your robot. #### Robot Control by a Single LLM Here is an example configuration of the `cortex_llm` showing use of a single LLM to generate decisions: ```json5 cortex_llm: { type: "OpenAILLM", config: { base_url: "", // Optional: URL of the LLM endpoint api_key: "...", // Optional: Override the default API key agent_name: "Iris", // Optional: Name of the agent history_length: 10 } } ``` * **type**: Specifies the LLM plugin. * **config**: LLM configuration, including the API endpoint (`base_url`), `agent_name`, and `history_length`. You can directly access other OpenAI style endpoints by specifying a custom API endpoint in your configuration file. To do this, provide a suitable `base_url` and the `api_key` for OpenAI, DeepSeek, or other providers. Possible `base_url` choices include: * * * (Ollama - local inference, no API key required) You can implement your own LLM endpoints or use more sophisticated approaches such as multiLLM robotics-focused endpoints by following the [LLM Guide](/core-concepts/concepts/5_llms). ### Agent Actions (`agent_actions`) Defines the agent's available capabilities, including action names, their implementation, and the connector used to execute them. Here is an example configuration for the `agent_actions` section: ```json5 agent_actions: [ { name: "move", llm_label: "move", implementation: "passthrough", connector: "ros2" }, { name: "speak", llm_label: "speak", implementation: "passthrough", connector: "ros2" config: { voice_id: "TbMNBJ27fH2U0VgpSNko", silence_rate: 0, }, } ] ``` You can customize the actions following the [Action Plugin Guide](/core-concepts/concepts/6_actions) ### MCP servers MCP servers can be added to a config to give OM1 agent capability to interact with different MCP tools. Example: ```json5 mcp_servers: [ { name: "weather", transport: "stdio", command: "npx", args: ["-y", "@h1deya/mcp-server-weather"], }, ] ``` Refer to [MCP Integration](/mcp/mcp-integration) to understand the complete architecture and how to configure new MCP tools with OM1. ### Transition rules Transition rules define how and when the robot switches between operational modes. ```json5 { from_mode: "", to_mode: "welcome", transition_type: "input_triggered", trigger_keywords: [ "reset", "start over", "welcome mode", "restart", "initialize", ], priority: 5, cooldown_seconds: 10.0, } ``` To understand transition rules in depth, refer the documentation [here](/modes-and-lifecycle/transition_rules) To introduce a new mode in your config, refer [introduce new mode](/developer-cookbook/introduction/new_mode) # Inputs Input Plugin Overview "Input Plugins" provide the sensory capabilities that allow robots to perceive their environment. These plugins capture, process, and format various types of input data, making them available to the robot's core runtime for decision-making. ### Basic Architecture * `Sensor` interface defines the core contract for all input plugins ([internal/inputs/sensor.go](https://github.com/OpenMind/OM1/blob/main/internal/inputs/sensor.go)) * `InputOrchestrator` manages multiple input sources * Custom input plugins implement the `Sensor` interface ```go // Sensor is the base interface for all input sensors. type Sensor interface { // Listen creates a channel that continuously yields raw input events. Listen(ctx context.Context) (<-chan any, error) // Poll retrieves a single raw input event. Poll(ctx context.Context) (any, error) // RawToText converts raw input data into Message format. RawToText(ctx context.Context, rawInput any) (*Message, error) // FormattedLatestBuffer returns the formatted buffer string. FormattedLatestBuffer() string // Stop signals the sensor to stop listening and clean up resources. Stop() } ``` ### Examples [Input plugin code examples](https://github.com/OpenMind/OM1/blob/main/plugins/inputs/README.md) Here are a few examples for you to reuse and build on: * [Google ASR](https://github.com/openmind/OM1/blob/main/plugins/inputs/google_asr/google_asr.go) * [Face Presence](https://github.com/openmind/OM1/blob/main/plugins/inputs/face_presence/face_presence.go) * [VLM COCO Local](https://github.com/openmind/OM1/blob/main/plugins/inputs/vlm_coco_local/vlm_coco_local.go) Learn how to build a new input plugin [here](/developer-cookbook/introduction/input) # LLMs LLM Integration OM1's LLM integration is intended to make it easy to (1) send `input` information to LLMs and then (2) route LLM responses to various system actions, such as `speak` and `move`. The OM1 system integrates various concrete implementations of Large Language Models (LLMs), each designed to address different requirements and interaction patterns. These implementations manage API communication, conversation history, and the processing of structured responses, particularly for function calls that trigger agent actions. The framework ensures a consistent interface, allowing the system to interchangeably utilize diverse LLM backends. OM1 also supports per-mode LLM configuration. If a mode specifies its own LLM, it takes precedence over the top-level cortex\_llm setting. This allows different modes to use different models based on their specific requirements. The plugins handle authentication, API communication, prompt formatting, response parsing, and conversation history management. LLM plugin examples are located in `plugins/llm`: [**Code**](https://github.com/OpenMind/OM1/tree/main/plugins/llm). ### Endpoint Overview ```bash # Base URL: https://api.openmind.com/ POST /api/core/{provider}/chat/completions # Single agent DELETE /api/core/agent/memory # Multi agent memory wipe ``` ### LLM Modes OM1 supports three LLM execution strategies depending on your latency, quality, and reliability requirements. | Mode | Description | Performance | | ------------ | ------------------------------------- | ------------------------------------ | | **Single** | One LLM processes all requests | Good — fast, but limited capability | | **Dual** | Local + cloud LLMs in parallel | Better — higher accuracy, but slower | | **Parallel** | N specialized LLMs run simultaneously | Best — fastest and most capable | #### Single LLM Integration For testing and introductory educational purposes, we integrate with multiple language models (LLMs) to provide chat completion via a `POST /api/core/{provider}/chat/completions` endpoint. Each LLM plugin takes fused input data (the `prompt`) and sends it to an LLM. The response is then parsed and provided to `internal/runtime/cortex.go` for distribution to the system actions: ```go response, err := client.ChatCompletions(ctx, &ChatRequest{ Model: config.Model, Messages: messages, ResponseFormat: outputModel, Timeout: config.Timeout, }) parsedResponse := outputModel.Validate(response.Choices[0].Message.Content) return parsedResponse ``` The standard output model is defined in `internal/llm/output_model.go`. Example config: ```json5 "cortex_llm": { "type": "OpenAILLM", // The class name of the LLM plugin you wish to use "config": { "model": "model_name", // Optional: If you want to switch to a specific model. Refer the list of supported models below "base_url": "", // Optional: URL of the LLM endpoint "agent_name": "Iris", // Optional: Name of the agent "history_length": 10 // The number of input->action cycles to provide to the LLM as historical context } } ``` #### Dual LLM support OM1 implements a dual-LLM response mechanism that combines both local and cloud-based models to optimize response quality and latency. * Local model: Qwen3-30B (on-device) * Cloud model: GPT-4.1 Example config: ```json5 "cortex_llm": { "type": "DualLLM", // The class name of the LLM plugin you wish to use "config": { "local_llm_type": "QwenLLM", // The class name of the LLM plugin you wish to use for local llm "local_llm_config": {"model": "RedHatAI/Qwen3-30B-A3B-quantized.w4a16"}, // model name you wish to use "cloud_llm_type": "OpenAILLM", // The class name of the LLM plugin you wish to use for cloud llm "cloud_llm_config": {"model": "gpt-4.1"} // model name you wish to use } } ``` **How It Works** 1. For each request, OM1 sends the prompt to both the local and cloud LLMs in parallel. 2. The system waits up to 3.2 seconds for responses. 3. If both models return a response within the threshold: * The two responses are evaluated by the local LLM. * The local LLM selects the better response as the final output. 4. If only one model responds within the threshold: That response is used directly as the final output. This approach ensures fast responses while leveraging cloud models for higher-quality outputs when available. #### Parallel LLM Multiple LLMs run in parallel, each handling specific actions they are capable of. Results stream as they complete, allowing the cortex to execute actions immediately without waiting for all LLMs. Example config: ```json5 "cortex_llm": { "type": "ParallelLLM", // The class name of the LLM plugin you wish to use "config": { "llms": [ { "llm_type": "OpenAILLM", // The class name of the LLM plugin you wish to use "llm_config": {"model": "gpt-4.1"}, // model name you wish to use "action_filter": ["speak", "emotion"] // preferred action for the model }, { "llm_type": "QwenLLM", // The class name of the LLM plugin you wish to use "llm_config": {"model": "RedHatAI/Qwen3-30B-A3B-quantized.w4a16"}, // model name you wish to use "action_filter": ["move", "navigate"] // preferred action for the model }, { "llm_type": "DeepSeekLLM", // The class name of the LLM plugin you wish to use "llm_config": {"model": "deepseek-chat"}, // preferred action for the model "action_filter": ["search", "analyze"] // preferred action for the model } ], "execute_immediately": true } } ``` ### Local LLMs The system supports on-device inference using the Qwen3-30B local LLM. This enables low-latency responses and allows certain workloads to run entirely on the device without relying on cloud connectivity. #### Ollama Integration [Ollama](https://ollama.ai) provides an easy way to run open-source models locally. OM1 supports Ollama through the `OllamaLLM` plugin. **Prerequisites:** 1. Install Ollama: 2. Pull a model: `ollama pull llama3.2` 3. Ensure Ollama is running: `ollama serve` **Configuration:** ```json "cortex_llm": { "type": "OllamaLLM", "config": { "model": "llama3.2", "base_url": "http://localhost:11434", "temperature": 0.7, "num_ctx": 4096, "timeout": 120 } } ``` **Run with Ollama:** ```bash make run CONFIG=ollama ``` #### Agent Architecture The system employs four primary agents that work together: * **Navigation Agent**: Processes spatial and movement-related tasks * **Perception Agent**: Handles sensory input analysis and environmental understanding * **RAG Agent**: Provides retrieval-augmented generation (RAG) capabilities using the user's knowledge base * **Team Agent**: Synthesizes outputs from all agents into a unified response #### Main API Endpoint ```go endpoint := "/api/core/{provider}/chat/completions" headers := map[string]string{ "Authorization": "Bearer " + config.APIKey, "Content-Type": "application/json", } request := ChatRequest{ SystemPrompt: ioProvider.FuserSystemPrompt, Inputs: ioProvider.FuserInputs, Model: config.Model, ResponseFormat: outputModel.JSONSchema(), StructuredOutputs: true, } response, err := httpClient.Post(endpoint, request, headers) output := response.Content return outputModel.Validate(output) ``` #### Supported Models ```go var OpenAISupportedModels = []string{"gpt-4o", "gpt-4o-mini", "gpt-4.1", "gpt-4.1-mini", "gpt-4.1-nano", "gpt-5", "gpt-5-mini", "gpt-5-nano"} ``` ```go var DeepSeekSupportedModels = []string{"deepseek-chat"} ``` ```go var GeminiSupportedModels = []string{"gemini-3.5-flash", "gemini-3.1-pro-preview", "gemini-3.1-flash-lite", "gemini-2.5-flash", "gemini-2.5-flash-lite", "gemini-2.5-pro"} ``` ```go var XAISupportedModels = []string{"grok-2-latest", "grok-3-beta", "grok-4-latest", "grok-4"} ``` ```go var NearAISupportedModels = []string{"qwen3-30b-a3b-instruct-2507", "qwen2.5-vl-72b-instruct", "qwen-2.5-7b-instruct"} ``` ```go var OpenRouterSupportedModels = []string{"meta-llama/llama-3.1-70b-instruct", "meta-llama/llama-3.3-70b-instruct", "anthropic/claude-sonnet-4.5", "anthropic/claude-opus-4.1"} ``` ```go // Ollama supports any model from https://ollama.ai/library var OllamaSupportedModels = []string{"llama3.2", "llama3.1", "mistral", "phi3", "gemma2", "qwen2.5", "codellama", "llava"} ``` ```go // Local LLM var LocalLLMModels = []string{"Qwen3-30B"} ``` ### Examples #### A Smart Dog Imagine you would like to program a smart dog. Describe the desired capabilities and behaviors of the dog in `system_prompt_base`. For example: ```json5 "system_prompt_base": "You are an intelligent robotic dog companion designed to be helpful, loyal, and engaging. Your primary goals are to: (1) Provide companionship through interactive play and conversation, (2) Assist with basic household tasks and monitoring, (3) Learn and adapt to your owner's preferences and routines, and (4) Maintain a playful yet responsible demeanor. You can move around, speak clearly, express emotions through body language, and respond to voice commands. Always prioritize safety and be eager to please while maintaining your dog-like personality traits of curiosity, loyalty, and enthusiasm." ``` # Actions Actions ### Action Plugins The Action Plugins are core components of OM1. These plugins map high-level decisions from one or more LLMs into concrete physical or digital actions (e.g. moving a robot or generating speech). This page covers the architecture of a typical Action Plugin, the available action types, and how actions are connected to different hardware and software platforms. [**Code**](https://github.com/OpenMind/OM1/tree/main/plugins/actions) ### Action Orchestrator The Action Orchestrator is the central component that orchestrates the execution of actions. It manages the states, promise queue, and threads for each action. [**Code**](https://github.com/OpenMind/OM1/blob/main/internal/actions/orchestrator.go) ### Movement (Zenoh) This plugin is an example of how to use Zenoh to send movement commands to a [TurtleBot 4](https://github.com/OpenMind/OM1/tree/main/plugins/actions/move_turtle/zenoh.go). ### Movement (Unitree SDK) This plugin is an example of how to connect to the Unitree SDK to send movement commands to a [Go2 EDU](https://github.com/OpenMind/OM1/tree/main/plugins/actions/move_go2_autonomy/unitree_rplidar_sdk.go). ### Speech and TTS The Speech and TTS action plugin allows agents to speak using a text-to-speech (TTS) system. [**Code**](https://github.com/OpenMind/OM1/blob/main/plugins/actions/speak/elevenlabs_tts.go) ### Adding New Actions Each action plugin consists of: 1. **Interface**: Defines input/output types via Go structs 2. **Implementation**: Business logic (or passthrough for simple actions) 3. **Connector**: Code that connects OM1 to specific virtual or physical environments ```tree plugins/actions/ ├── move_{unique_hardware_id}/ │ ├── interface.go # Defines MoveInput/Output structs │ ├── passthrough.go # Simple passthrough implementation │ ├── ros2.go # Maps OM1 data/commands to ROS2 │ ├── zenoh.go # Maps OM1 data/commands to Zenoh │ └── unitree.go # Maps OM1 data/commands to Unitree SDK └── speak/ └── elevenlabs_tts.go ``` In general, each robot will have specific capabilities, and therefore, each action will be hardware specific. For example, if you are adding support for the Unitree G1 Humanoid version 13.2b, which supports a new movement subtype such as `dance_2`, you could name the updated action `move_unitree_g1_13_2b` and select that action in your `unitree_g1.json` configuration file. # Backgrounds Backgrounds #### Background Tasks The Background Tasks system provides a framework for running continuous, long-running processes that operate independently of the main control loop. These tasks typically handle sensor data collection, state monitoring, and other background operations. Key components: * **BackgroundOrchestrator**: Manages the lifecycle of all background tasks, including startup and graceful shutdown. * **Plugins**: Background tasks are loaded dynamically from the `backgrounds/plugins` directory. * Each background task runs in its own thread, with thread pooling for efficient resource utilization. Available background tasks include: * **GPS**: Handles GPS data processing * **ODOM**: Manages odometry data * **RF Mapper**: Implements RF signal mapping functionality * **RPLIDAR**: Interfaces with RPLIDAR sensors * **RTK**: Real-Time Kinematic positioning * **Unitree Go2 State**: Manages state for Unitree Go2 robots Background tasks are configured through the main runtime configuration and can be extended by adding new plugin modules. #### Scopes: `agent_backgrounds` vs `global_backgrounds` Background tasks come in two scopes: * **`agent_backgrounds`** (mode-scoped): declared inside a mode. They start when the mode is entered and stop when the mode is exited, so they only run while that mode is active. In a single-mode config, top-level `agent_backgrounds` seed the one synthesized mode. * **`global_backgrounds`** (system-wide): declared at the top level of the config. They start once when the runtime starts and keep running across every mode switch until shutdown. Use this scope for tasks that must observe or act continuously regardless of the current mode. ```json5 { // ... global_backgrounds: [ { type: "ApproachingPerson" }, // runs in every mode ], modes: { welcome: { // ... agent_backgrounds: [ { type: "UnitreeGo2FrontierExploration" }, // runs only in this mode ], }, }, } ``` # Middleware ROS2 and DDS Setup This section focuses on installation guidelines for the ROS 2 middleware stack and related tools. ### Middleware components The following guides walk you through installing and configuring the supported middleware implementations: * [CycloneDDS](/core-concepts/middleware/cyclonedds): Install and configure the CycloneDDS RMW implementation for ROS 2. * [ROS 2 Humble](/core-concepts/middleware/ros2-humble): Set up the ROS 2 Humble distribution, including core tools and environment configuration. * [Zenoh Bridge](/core-concepts/middleware/zenoh-bridge): Install and configure the Zenoh bridge for integrating ROS 2 with Zenoh-based systems. # CycloneDDS Installation Install cyclonedds from this [link](https://cyclonedds.io/docs/cyclonedds/latest/installation/installation.html) or follow the instructions below. ```bash sudo apt-get install git cmake gcc ``` ```bash git clone https://github.com/eclipse-cyclonedds/cyclonedds -b releases/0.10.x cd cyclonedds && mkdir build install && cd build cmake -DBUILD_EXAMPLES=ON -DCMAKE_INSTALL_PREFIX=$HOME/Documents/GitHub/cyclonedds/install .. cmake --build . --target install ``` ## CycloneDDS config ### for Unitree Simulation (Gazebo or Isaac Sim) Use this CycloneDDS configuration for running simulation. It uses `lo` as the network interface. We recommend that you export this in your .bashrc or equivalent configuration file cyclonedds.xml. To add it to cyclonedds.xml: ```bash cd cyclonedds vi cyclonedds.xml ``` Add the following, then save and exit. ```bash 200 ``` Open your bashrc file ```bash vi ~/.bashrc ``` Add the following, replacing /path/to/cyclonedds with the actual path to your CycloneDDS installation: ```bash export RMW_IMPLEMENTATION=rmw_cyclonedds_cpp export CYCLONEDDS_URI=/path/to/cyclonedds/cyclonedds.xml ``` Apply the changes ```bash source ~/.bashrc ``` To add the config to your bashrc, run: ```bash vim ~/.bashrc ``` And add the following, replacing `/path/to/cyclonedds` with the actual path to your CycloneDDS installation: ```bash export RMW_IMPLEMENTATION=rmw_cyclonedds_cpp export CYCLONEDDS_URI=' 200 ' ``` Now run ```bash source ~/.bashrc ``` This will apply the latest changes in the current shell session. # ROS2-humble Installation ## Setup Sources You will need to add the ROS 2 apt repository to your system. First ensure that the Ubuntu Universe repository is enabled. ```bash sudo apt install software-properties-common sudo add-apt-repository universe ``` The ros-apt-source packages provide keys and apt source configuration for the various ROS repositories. Installing the `ros2-apt-source` package will configure ROS 2 repositories for your system. Updates to repository configuration will occur automatically when new versions of this package are released to the ROS repositories. ```bash sudo apt update && sudo apt install curl -y export ROS_APT_SOURCE_VERSION=$(curl -s https://api.github.com/repos/ros-infrastructure/ros-apt-source/releases/latest | grep -F "tag_name" | awk -F\" '{print $4}') curl -L -o /tmp/ros2-apt-source.deb "https://github.com/ros-infrastructure/ros-apt-source/releases/download/${ROS_APT_SOURCE_VERSION}/ros2-apt-source_${ROS_APT_SOURCE_VERSION}.$(. /etc/os-release && echo ${UBUNTU_CODENAME:-${VERSION_CODENAME}})_all.deb" sudo dpkg -i /tmp/ros2-apt-source.deb ``` Now, run ```bash sudo apt update sudo apt upgrade sudo apt install ros-humble-desktop ``` This will install ROS and all the relevant packages. > **Note:** When installing ros2-humble for Gazebo, run `sudo apt install ros-humble-desktop-full` instead of sudo apt install `ros-humble-desktop`. It will install ROS, RViz, and Gazebo. Refer [ros2-humble](https://docs.ros.org/en/humble/Installation/Ubuntu-Install-Debs.html), to get a better understanding. # Zenoh ROS2 Bridge Installation ROS (the Robot Operating System) is a set of software libraries and tools allowing to build robotic applications. In its version 2, ROS 2 relies mostly on DDS as a middleware for communications. This plugin bridges all ROS 2 communications using DDS over Zenoh. While a Zenoh bridge for DDS already exists and helped lot of robotic use cases to overcome some wireless connectivity, bandwidth and integration issues, using a bridge dedicated to ROS 2 brings the following advantages: A better integration of the ROS graph (all ROS topics/services/actions can be seen across bridges) A better support of ROS toolings (ros2, rviz2...) Configuration of a ROS namespace on the bridge, instead of on each ROS Nodes Easier integration with Zenoh native applications (services and actions are mapped to Zenoh Queryables) More compact exchanges of discovery information between the bridges ### Install zenoh-bridge Add Eclipse Zenoh private repository to the sources list: ```bash curl -L https://download.eclipse.org/zenoh/debian-repo/zenoh-public-key | sudo gpg --dearmor --yes --output /etc/apt/keyrings/zenoh-public-key.gpg echo "deb [signed-by=/etc/apt/keyrings/zenoh-public-key.gpg] https://download.eclipse.org/zenoh/debian-repo/ /" | sudo tee -a /etc/apt/sources.list > /dev/null sudo apt update ``` Now you can install the standalone executable with: `sudo apt install zenoh-bridge-ros2dds`. # Troubleshooting Guide Guide to troubleshoot some common issues | Issue | Likely Cause | Quick Fix | | ---------------------------- | ------------------------------ | ------------------------------------------------------------ | | No Speech | Permission issues | Check the settings | | No speech recognition | Microphone not configured | Check audio input settings | | Robot not moving | Connection issue/Network issue | Restart OM1/Robot and check your internet connection | | Build errors | Missing dependencies | Run `make deps` to download zenoh-c and Go dependencies | | PortAudio error during build | Missing PortAudio headers | `sudo apt-get update` `sudo apt-get install portaudio19-dev` | # MCP Integration Connect your OM1 agent to external tools via Model Context Protocol OM1 agents can now connect to external tools like Slack, Notion, weather APIs, and more — all through the Model Context Protocol (MCP). No custom integrations required. ### What is MCP? MCP is an open standard that lets AI agents discover and use tools dynamically. Think of it as **USB for AI** — plug in a server, and your agent instantly knows what tools are available. ### Architecture ![](/files/Rk83DvEROrAnuvS9fzcY) #### Key Components | Component | Description | | -------------------- | ---------------------------------------------------------------------------------------------- | | **OM1 Agent** | The main agent runtime for the machine | | **Modes** | Each mode can have its own MCP servers, system prompt, inputs, and actions | | **MCP Servers** | Per-mode configuration that defines which external tools are available in that particular mode | | **Orchestrator** | Coordinates communication between the agent and external services | | **MCPClientManager** | Manages connections to MCP servers, discovers tools, and routes tool calls | | **MCP Tools** | External services like Slack, Google Maps, Weather APIs, Notion, and more | | **Robot Interface** | Physical or simulated robot that interacts with the real world | #### Per-Mode MCP Servers Each mode operates independently with its own set of MCP servers. For example: ``` Mode 1 ──► [Slack, Google Maps] ──► MCPClientManager instance 1 Mode 2 ──► [Weather, News] ──► MCPClientManager instance 2 Mode 3 ──► (no MCP servers) ──► Offline mode ``` This enables: * **Isolation**: Different modes access different tools * **Flexibility**: Hot-swap capabilities without code changes #### Install Node MCP servers typically run via `npx`, which requires Node.js to be installed. Check if you already have it installed by running the following commands - ```bash node --version npm --version ``` If you don't have node installed on your system, follow the steps [here](https://nodejs.org/en/download/current). #### Add MCP Servers to Your Config | Field | Type | Required | Description | Example | | ----------- | -------- | ------------------------- | ------------------------------------------------------------------------------------- | ---------------------------------------------------- | | `name` | *string* | Required | Server identifier | `"google-maps"` | | `transport` | *string* | Required | How to connect: `stdio` (subprocess), `sse` (streaming), or `http` (request/response) | `"stdio"` | | `command` | *string* | Required | Executable to launch | `"npx"` | | `args` | *array* | Optional | Arguments for the command | `["-y", "@modelcontextprotocol/server-google-maps"]` | | `env` | *object* | Optional | Environment variables required by the server (e.g., API keys, tokens) | `{"API_KEY": "your-api-key"}` | | `url` | *string* | Required for `sse`/`http` | Server endpoint URL | `"http://localhost:3000/sse"` | | `headers` | *object* | Optional | HTTP headers for `sse`/`http` transport | `{"Authorization": "Bearer token"}` | ### How It Works ``` ┌─────────────────────────────────────────────────────────────┐ │ OM1 Agent │ │ │ │ Config ──► load_mcp() ──► MCPClientManager │ │ │ │ │ ├── Discovers tools │ │ ├── Manages connections │ │ └── Executes tool calls │ └─────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────┐ │ MCP Servers │ │ (Slack, Maps, etc.) │ └─────────────────────────┘ ``` 1. **Configuration**: MCP servers are defined in your mode's config file 2. **Initialization**: `MCPClientManager` launches and connects to each server 3. **Discovery**: Tools are automatically discovered from connected servers 4. **Execution**: When the LLM decides to use a tool, `MCPClientManager` routes the call Example config file [conversation\_mcp](https://github.com/OpenMind/OM1/blob/main/config/conversation_mcp.json5) ### Available MCP Servers Browse community-maintained MCP servers: * [Official MCP Servers](https://github.com/modelcontextprotocol/servers) * Slack, GitHub, Google Maps, Filesystem, and many more ### Next Steps * [Configuration Guide](https://github.com/OpenMind/OM1/blob/main/docs/core-concepts/3_configuration.md) — Learn more about OM1 config files * [MCP Specification](https://modelcontextprotocol.io) — Official MCP documentation # Introduction Introduction The Developer Cookbook is a collection of practical, high-impact recipes designed to help you extend, customize, and build on top of the OpenMind platform. If the Quickstart shows you how to use OM1, this Cookbook shows you how to build with it. Here's what you can do with OM1 1. Build a new config file 2. Introduce a new mode 3. Configure a new Input Plugin Before building with OM1, make sure you've completed the [Getting Started](https://github.com/OpenMind/OM1/blob/main/docs/developing/1_get-started/README.md) guide and have OM1 installed. Understand the important concepts and components that are part of OM1. Then dive into any recipe that interests you! ### Development workflow #### Linting and Testing (Mandatory) Run repository checks before committing: ```bash make check ``` Or run individual checks when needed: ```bash make fmt make lint make test ``` #### Unit Testing To unit test the system, run: ```bash make test ``` Use clear naming conventions and comments for better code maintainability. # Configuration Config The config file defines the agent that runs on your machine. It tells OM1 which modules to load, how the robot should behave, and which modes are available. To ensure your configuration is valid, follow the format defined [here](https://github.com/OpenMind/OM1/tree/main/config/schema). **Steps to build a new config file** 1. Start with getting your API key from [OpenMind Portal](https://portal.openmind.com/). Copy it and save it, you'll paste it into the config later. 2. Create a new config file config.json5 | Field | Type | Required | Description | | ------------------------ | -------- | -------- | -------------------------------------------------------------------------------------------------------- | | `version` | `string` | Yes | The version of the configuration format. Example: `"v1.0.0"` | | `hertz` | `number` | Yes | How often (in Hz) the agent runs its update loop. Example: `0.01` | | `name` | `string` | Yes | The name of the agent. Example: `"conversation"` | | `default_mode` | `string` | Yes | The default\_mode defines the mode robot starts in. Example: `"welcome"` | | `allow_manual_switching` | `bool` | Yes | Defines if manual mode switching is allowed. Example: `true` | | `mode_memory_enabled` | `bool` | Yes | Enables or disables mode memory. Example: `true` | | `api_key` | `string` | Yes | API key used to authenticate the agent. Example: `"openmind_free"` | | `system_prompt_base` | `string` | Yes | Defines the agent's core personality and behavior. Serves as the primary system prompt for the LLM. | | `system_governance` | `string` | Yes | The laws or constraints that the agent must follow during operation. Modeled similarly to Asimov's laws. | | `system_prompt_examples` | `string` | No | Example interactions that help guide the model's behavior. | #### Step 3. Customize the system prompts ``` There are three key prompt fields: - system_prompt_base Defines your agent’s personality and behavior. You can keep the “Spot the dog” behavior or edit it to match your needs. You can also provide context to the LLM here. - system_governance Hard-coded rules the agent must follow (Asimovs laws). - system_prompt_examples Give your model examples of how to respond. These help shape its responses. You can add more examples if needed. ``` #### Step 4. Configure inputs ``` Inputs provide the sensory capabilities that allow robots to perceive their environment ``` | Field | Type | Required | Description | | -------- | -------- | -------- | ---------------------------------------------------------------------------- | | `type` | `string` | Yes | The input type identifier. Example: `"AudioInput"` | | `config` | `object` | No | Configuration options specific to this input type. Example: `GoogleASRInput` | #### Step 5. Configure the LLM | Field | Type | Required | Description | | ---------------- | --------- | -------- | ---------------------------------------------------------------------- | | `type` | `string` | Yes | The LLM provider name. Example: `"OpenAILLM"` | | `config` | `object` | No | Configuration options specific to this LLM type. | | `agent_name` | `string` | No | Agent name used in metadata. Example: `"Spot"` | | `history_length` | `integer` | No | Number of past messages to remember in the conversation. Example: `10` | #### Step 6. Set up agent actions ``` Actions define what your agent can do. You can define movement, TTS or any other actions here. ``` | Field | Type | Required | Description | | ---------------- | -------- | -------- | ------------------------------------------------------------------------------------------------------------------------- | | `name` | `string` | Yes | Human-readable identifier for the action. Example: `"speak"` | | `llm_label` | `string` | Yes | Label the model uses to refer to this action. Example: `"speak"` | | `implementation` | `string` | No | Defines the business logic. If none defined, defaults to `"passthrough"`. Example: `"passthrough"` | | `connector` | `string` | Yes | Name of the connector. This is the Go file name defined under `plugins/actions/action_name/`. Example: `"elevenlabs_tts"` | #### Step 7: Add modes Add `modes` section in your config file and introduce the modes you'd like to configure for you agent. | Field | Type | Required | Description | | -------------------- | --------- | -------- | ----------------------------------------------------------------------------------------------- | | `display_name` | `string` | Yes | The human-readable name shown in the UI for this mode. Example: `"Your New Mode"` | | `description` | `string` | Yes | Brief description explaining what this mode does and its purpose. | | `system_prompt_base` | `string` | Yes | The foundational system prompt that defines the agent's behavior and purpose in this mode. | | `hertz` | `float` | Yes | The frequency (in Hz) at which the agent operates or processes information. Example: `1.0` | | `timeout_seconds` | `integer` | Yes | Maximum duration (in seconds) before the agent times out during execution. Example: `300` | | `remember_locations` | `boolean` | Yes | Whether the agent should persist and recall location data across interactions. Example: `false` | | `save_interactions` | `boolean` | Yes | Whether to save conversation history and interactions for this mode. Example: `true` | | `agent_inputs` | `array` | Yes | List of input sources or data types the agent can accept in this mode. | | `agent_actions` | `array` | Yes | List of actions or capabilities the agent can perform in this mode. | | `lifecycle_hooks` | `array` | Yes | Event handlers triggered at specific points in the agent's lifecycle (startup, shutdown, etc.). | | `simulators` | `array` | Yes | List of simulation environments or tools available to the agent in this mode. | | `cortex_llm` | `object` | Yes | Configuration object for the language model powering the agent's cortex. | For a better understanding of how modes are configured, refer the documentation [here](/developer-cookbook/introduction/new_mode) #### Step 8. Validate the config ``` Before using the file: Check for JSON errors, make sure commas, quotes, and braces are correct and confirm that correct API key is configured. ``` # Input Build a new input plugin ### Overview This guide walks you through creating a new input plugin for OM1. Input plugins allow you to integrate various data sources and sensors into your agent. ### Prerequisites * Understanding of Go interfaces and structs * Familiarity with concurrent programming in Go * Knowledge of the data source you're integrating ### Implementation Steps #### Step 1: Create a Provider (Optional) If your plugin requires complex initialization or external service integration, create a provider. Location: `/internal/providers/your_provider.go` #### Step 2: Create a new Plugin File To proceed with a new input plugin integration, create a Go file. Location: `/plugins/inputs/your_plugin/your_plugin.go` Required imports: ```go package your_plugin import ( "context" "github.com/openmind/om1/internal/inputs" ) ``` #### Step 3: Implement the Sensor Interface Your plugin must implement the `Sensor` interface defined in `internal/inputs/sensor.go`: ```go type Sensor interface { // Listen creates a channel that continuously yields raw input events. Listen(ctx context.Context) (<-chan any, error) // Poll retrieves a single raw input event. Poll(ctx context.Context) (any, error) // RawToText converts raw input data into Message format. RawToText(ctx context.Context, rawInput any) (*inputs.Message, error) // FormattedLatestBuffer returns the formatted buffer string. FormattedLatestBuffer() string // Stop signals the sensor to stop listening and clean up resources. Stop() } ``` #### Step 4: Implement Your Plugin Struct ```go type YourInput struct { config map[string]any isRunning bool buffer string } func New(cfg map[string]any) (inputs.Sensor, error) { return &YourInput{ config: cfg, }, nil } ``` #### Step 5: Implement Required Methods ```go func (y *YourInput) Listen(ctx context.Context) (<-chan any, error) { ch := make(chan any) go func() { defer close(ch) for { select { case <-ctx.Done(): return default: // Read from your data source and send to channel data, err := y.readData() if err == nil { ch <- data } } } }() return ch, nil } func (y *YourInput) Poll(ctx context.Context) (any, error) { // Return a single reading from your data source return y.readData() } func (y *YourInput) RawToText(ctx context.Context, rawInput any) (*inputs.Message, error) { // Convert raw input to a Message text := formatAsText(rawInput) return inputs.NewMessage(text), nil } func (y *YourInput) FormattedLatestBuffer() string { return y.buffer } func (y *YourInput) Stop() { y.isRunning = false } ``` ### Plugin Registration Plugins are registered using the `inputs.Register` function. Add an `init()` function to your plugin: ```go func init() { inputs.Register("YourInput", New) } ``` #### How it works: * The `inputs.Register` function maps your plugin type name to its factory function * The `inputs.Load` function creates instances based on configuration * Plugin type names in config files must match the registered name #### Requirements: * Implement the `Sensor` interface from `internal/inputs/sensor.go` * Register your factory function with `inputs.Register` * File must be in `/plugins/inputs/` directory # New Mode Add a new mode This guide walks you through creating a new mode for your robot system. ### Project Structure ```bash internal/runtime/ ├── config.go # ModeConfig and ModeSystemConfig types ├── manager.go # ModeManager for transitions ├── cortex.go # ModeCortexRuntime for execution ├── hook.go # Lifecycle hooks └── converter.go # Converts legacy single mode configs to multimode config config/ └── your_robot_modes.json5 # Mode configuration file ``` > **Note:** Single mode is now deprecated. Any legacy single-mode config will now be converted into the multi-mode format, simplifying the runtime and CLI logic. ### Configuration #### Step 1: Create Configuration File Create or modify a configuration file (e.g., `your_robot_modes.json5`) in the `/config/` directory. #### Step 2: Add Mode Definition Add your new mode to the `modes` section of your configuration file. | Field | Type | Required | Description | | -------------------- | --------- | -------- | ----------------------------------------------------------------------------------------------- | | `display_name` | `string` | Yes | The human-readable name shown in the UI for this mode. Example: `"Your New Mode"` | | `description` | `string` | Yes | Brief description explaining what this mode does and its purpose. | | `system_prompt_base` | `string` | Yes | The foundational system prompt that defines the agent's behavior and purpose in this mode. | | `hertz` | `float` | Yes | The frequency (in Hz) at which the agent operates or processes information. Example: `1.0` | | `timeout_seconds` | `integer` | Yes | Maximum duration (in seconds) before the agent times out during execution. Example: `300` | | `remember_locations` | `boolean` | Yes | Whether the agent should persist and recall location data across interactions. Example: `false` | | `save_interactions` | `boolean` | Yes | Whether to save conversation history and interactions for this mode. Example: `true` | | `agent_inputs` | `array` | Yes | List of input sources or data types the agent can accept in this mode. | | `agent_actions` | `array` | Yes | List of actions or capabilities the agent can perform in this mode. | | `lifecycle_hooks` | `array` | Yes | Event handlers triggered at specific points in the agent's lifecycle (startup, shutdown, etc.). | | `simulators` | `array` | Yes | List of simulation environments or tools available to the agent in this mode. | | `cortex_llm` | `object` | Yes | Configuration object for the language model powering the agent's cortex. | #### Step 3: Configure Input Plugins Specify which inputs your mode needs: | Field | Type | Required | Description | | -------- | -------- | -------- | -------------------------------------------------- | | `type` | `string` | Yes | The input type identifier. Example: `"AudioInput"` | | `config` | `object` | No | Configuration options specific to this input type. | #### Step 4: Configure LLM (Optional - Can be overwritten for each mode) Define which LLM needs to be configured: | Field | Type | Required | Description | | ---------------- | --------- | -------- | ---------------------------------------------------------------------- | | `type` | `string` | Yes | The LLM provider name. Example: `"OpenAILLM"` | | `config` | `object` | No | Configuration options specific to this LLM type. | | `agent_name` | `string` | No | Agent name used in metadata. Example: `"Spot"` | | `history_length` | `integer` | No | Number of past messages to remember in the conversation. Example: `10` | #### Step 5: Configure actions Actions define what your agent can do. You can define movement, TTS or any other actions here. | Field | Type | Required | Description | | --------------------- | --------- | -------- | ------------------------------------------------------------------------------------------------------------------------- | | `name` | `string` | Yes | Human-readable identifier for the action. Example: `"speak"` | | `llm_label` | `string` | Yes | Label the model uses to refer to this action. Example: `"speak"` | | `implementation` | `string` | No | Defines the business logic. If none defined, defaults to `"passthrough"`. Example: `"passthrough"` | | `connector` | `string` | Yes | Name of the connector. This is the Go file name defined under `plugins/actions/action_name/`. Example: `"elevenlabs_tts"` | | `config` | `object` | No | Configuration options specific to this action. | | `exclude_from_prompt` | `boolean` | No | Whether to exclude this action from the LLM prompt. Default: `false` | #### Step 6: Add Lifecycle hooks (Only required for multi-mode) A hook is a programmable event point that executes specific actions at key stages. To define lifecycle hooks, you can add the following to your config. | Field | Type | Required | Description | | ----------------- | --------- | -------- | -------------------------------------------------------------------------------------------------------------------- | | `hook_type` | `string` | Yes | The lifecycle event type. Allowed values: `"on_startup"`, `"on_shutdown"`, `"on_entry"`, `"on_exit"`, `"on_timeout"` | | `handler_type` | `string` | Yes | The type of handler to execute. Allowed values: `"message"`, `"command"`, `"function"`, `"action"` | | `handler_config` | `object` | Yes | Configuration for the handler containing one of: `message`, `command`, `function`, or `action` as a string property | | `priority` | `integer` | No | Execution priority when multiple hooks exist for the same event. | | `async_execution` | `boolean` | No | Whether to execute the handler asynchronously. | | `timeout_seconds` | `number` | No | Maximum duration for handler execution. | | `on_failure` | `string` | No | Behavior when handler fails. Allowed values: `"log"`, `"ignore"`, `"abort"` | #### Step 7: Add Transition Rules (Only required for multi-mode) Add to the transition\_rules section: | Field | Type | Required | Description | | ------------------ | --------- | -------- | ------------------------------------------------------------------------ | | `from_mode` | `string` | Yes | The source mode from which the transition originates. | | `to_mode` | `string` | Yes | The target mode to which the agent will transition. | | `transition_type` | `string` | Yes | The type of transition mechanism. | | `trigger_keywords` | `array` | Yes | List of keywords that trigger this transition. Each item is a string. | | `priority` | `integer` | Yes | Priority level for this transition rule when multiple rules match. | | `cooldown_seconds` | `number` | No | Minimum time (in seconds) before this transition can be triggered again. | #### Step 8: Update Default Mode (Optional) If "new\_mode" should be the starting mode, define ```bash "default_mode": "new_mode" ``` Now, your new mode is ready to be tested. Deploy it directly on your robot or configure it through the docker\_compose file! # Examples Examples Overview This section contains practical examples demonstrating how to use the OM1 project. ## Getting Started Before running any examples, ensure you have the project dependencies installed. Refer the documentation [here](/developing/1_get-started). ## Running Examples Examples can be executed using: ```bash make run CONFIG= ``` Some examples to get started with * [Conversation](https://github.com/OpenMind/OM1/blob/main/config/conversation.json5) * [Unitree G1](https://github.com/OpenMind/OM1/blob/main/config/unitree_g1.json5) * [Unitree Go2](https://github.com/OpenMind/OM1/blob/main/config/unitree_go2.json5) * [UbTech Mini Humanoid](https://github.com/OpenMind/OM1/blob/main/config/ubtech_yanshee.json5) * [Turtlebot](https://github.com/OpenMind/OM1/blob/main/config/turtlebot4.json5) * [MCP Integration](https://github.com/OpenMind/OM1/blob/main/config/conversation_mcp.json5) # Conversation Using Cloud Endpoints for Voice Inputs and Text to Speech This section provides various examples for integrating and using multiple cloud-based AI endpoints, such as OpenAI, DeepSeek, and others, for voice input processing, text-to-speech (TTS) and emotion detection. Whether you need to convert spoken language into text (ASR) or generate natural-sounding speech from text, these examples will help you interact with different cloud providers seamlessly. ### Voice to Text Processing with OpenAI This example uses your `default` audio in (microphone) and your `default` audio output (speaker). Please test both your microphone and speaker in your system settings to make sure they are connected and working. On a Mac, the system may request permission to access your audio - Allow permissions. ```bash make run CONFIG=conversation ``` Especially on Linux, such as on Ubuntu 20.04 on the Nvidia Orin, audio support can be marginal. Expect some audio inputs and outputs to not work correctly, or to advertise incorrect hardware capabilities, such as USB microphones that report zero input channels etc. ### Enumerating your Audio You can enumerate available audio via the test script in `/system_hw_test`: ```bash python test_audio.py ``` ### Testing Audio You can provide test sentences to speak by adding the `MockInput` to the config file: ```bash { "type": "MockInput", "config": { "input_name": "Voice Input" } } ``` Then connect to the `ws` (`wscat -c ws://localhost:8765`) and type in the words you want the system to speak. This is useful to debug audio out issues and related settings such as chunk values. # Smart Toy Combining Inputs, an LLM, and Outputs to create a smart, engaging toy. ### LLM Model This example takes two inputs, emotional state and voice inputs, sends them to an LLM, and produces speech outputs and physical movements. The overall behavior of the system is configured in `/config/cubly.json5`. ### Emotion Detection with Cubly Run ```bash make run CONFIG=cubly ``` You should see your webcam light turn on and Cubly should speak to you from your default laptop speaker. You can see what is happening in the terminal logs. * There will be an initial delay for your system to download various packages and AI/ML models. * Arduino based movement generation only works if you actually have a suitable actuator connected to an Arduino, which is connected to your computer via a USB serial dongle. On Mac, you can determine the correct serial port name to use via `ls /dev/cu.usb*`. If you do not specify your computer's serial port, the example will provide logging data that simulates what it would send. * The **emotion estimation input** is provided by webcam data feeding into `cv2.CascadeClassifier(haarcascade_frontalface_default)`. See `/inputs/plugins/webcam_to_face_emotion`. * The **voice input** originates from the default microphone, which sends data to the cloud instance of Nvidia's RIVA. See `/inputs/plugins/asr`. * The **voice output** uses a cloud text to speech endpoint, which sends audio data to the default speaker. See `/actions/speak/connector/tts`. * The **Arduino serial movement actions** are sent to serial com port `COM1`, flowing to a connected Arduino, which can then generate servo commands. See `/actions/move_serial_arduino/connector/serial_arduino`. # OM1 Integration with different machines # Unitree G1 Humanoid Unitree G1 Humanoid ### Useful links and Reading C++ SDK Python SDK ### Basic Command Run ```bash make run CONFIG=unitree_g1_humanoid ``` #### Installation on Mac ```bash brew install portaudio cmake ``` Install cycloneDDS via . ```bash git clone https://github.com/eclipse-cyclonedds/cyclonedds -b releases/0.10.x cd cyclonedds && mkdir build install && cd build cmake -DBUILD_EXAMPLES=ON -DCMAKE_INSTALL_PREFIX=$HOME/Documents/GitHub/cyclonedds/install .. cmake --build . --target install ``` At this point, if you are connected to a robot and you run `./ddsperf sanity` (located in `$HOME/Documents/GitHub/cyclonedds/install/bin/` or whatever you chose) you should start to see data: ``` [8268] 2.005 rss:4.7MB vcsw:0 ivcsw:472 tev:0%+1% recv:0%+1% [8268] 3.005 rss:4.8MB vcsw:0 ivcsw:275 tev:0%+1% others:0%+1% ``` **List CycloneDDS Topics on Mac** ```bash export CYCLONEDDS_HOME=$HOME/Documents/GitHub/cyclonedds/install export CMAKE_PREFIX_PATH=$HOME/Documents/GitHub/cyclonedds/install # Then, set your wired Ethernet adapter to `192.168.123.99` and `255.255.255.0`, double check via `ifconfig`, and set the correct Ethernet adapter name in `NetworkInterface`: export CYCLONEDDS_URI=' true ' ``` Then, compile and run the listtopics example: ```bash cd $HOME/Documents/GitHub/cyclonedds/install/share/CycloneDDS/examples/listtopics cmake . cmake --build . ``` Now, when you run `./listtopics`, you should see an extensive data dump: ```bash alive: ea9d27e0:9769a902:84fcfb59:2b6083bc rt/lf/lowstate unitree_go::msg::dds_::LowState_ alive: 3b07fe34:41cac3a8:9f12e6e6:f719133e rt/api/motion_switcher/response unitree_api::msg::dds_::Response_ alive: eec56342:f93120c6:bab05432:a1b4a0f8 rt/api/motion_switcher/request unitree_api::msg::dds_::Request_ alive: 8e9380cf:5e6f9214:9d5e768f:dfd6d595 rt/utlidar/voxel_map sensor_msgs::msg::dds_::PointCloud2_ alive: fee410f9:eaab3e20:b1c6a349:ac7b8b8c rt/utlidar/voxel_map_compressed unitree_go::msg::dds_::VoxelMapCompressed_ alive: d0709b33:bfcd6551:4d85a309:74df1cbb rt/utlidar/height_map sensor_msgs::msg::dds_::PointCloud2_ alive: 9e531810:a229c451:3315fd9c:415edc24 rt/utlidar/range_map sensor_msgs::msg::dds_::PointCloud2_ alive: 0a6d5257:787977ea:5ac0156c:1b39ba46 rt/utlidar/range_info geometry_msgs::msg::dds_::PointStamped_ alive: 83ebb9e5:6d69cbf8:4b458542:dae1be14 rt/utlidar/height_map_array unitree_go::msg::dds_::HeightMap_ alive: 1bf83b4b:562d97ef:9df5443a:84971a8a rt/utlidar/map_state unitree_go::msg::dds_::VoxelHeightMapState_ alive: a71b9ea9:984d116d:a2400d39:0839ce01 rt/utlidar/grid_map sensor_msgs::msg::dds_::PointCloud2_ alive: 3db4a0f1:1e75176b:ed20434a:fe87a63f rt/utlidar/robot_odom nav_msgs::msg::dds_::Odometry_ alive: 036f7221:ad729459:9435a047:09a9934e rt/utlidar/cloud_deskewed sensor_msgs::msg::dds_::PointCloud2_ alive: aa29f45f:8ce692b7:22cfb924:75a5bc71 rt/utlidar/mapping_cmd std_msgs::msg::dds_::String_ alive: 33264264:c316c9f9:525f6b90:cb1eb2ec rt/wirelesscontroller unitree_go::msg::dds_::WirelessController_ alive: 9613c077:46dfce42:88f8e974:9c7c8717 rt/api/sport/request unitree_api::msg::dds_::Request_ alive: d994fe76:2755380f:1d1835bb:ddba9c93 rt/api/obstacles_avoid/request unitree_api::msg::dds_::Request_ ... ``` The Ethernet adapter you set above (such as `en0`) is the value you should provide to `/config/unitree_g1_humanoid.json`. Then add the optional Python CycloneDDS module to OM1: ```bash uv pip install -r pyproject.toml --extra dds ``` #### Installation on Linux You will need: * OM1 * uv * ffmpeg (for audio, otherwise the audio out will not work due to missing `ffprobe`) * v4l-utils (for video) * CycloneDDS (for DDS comms to the G1 motion client) ```bash sudo apt-get update sudo apt-get install ffmpeg v4l-utils ``` v4l-utils is also useful to debug video problems. **WARNING**: The camera system, if not correctly configured, has a tendency to bring down the entire USB bus. FIX: reboot the humanoid. Set the correct `CYCLONEDDS_HOME` env var. This is where the actual CycloneDDS is installed on your computer: ```bash export CYCLONEDDS_HOME="$HOME/unitree_ros2/cyclonedds_ws/install/cyclonedds" ``` If you do not do this correctly, installation of the Python CycloneDDS, a later step, will fail since it cannot find the correct libraries. Note: make absolutely sure `CYCLONEDDS_HOME` actually points to the CycloneDDS install /lib. This can be confusing, since if you install indirectly via `unitree_ros2`, then the location of the CycloneDDS libraries will be in slightly different location than if you install directly, via `git clone https://github.com/eclipse-cyclonedds/cyclonedds`. Then add the optional Python CycloneDDS module to OM1: ```bash uv pip install -r pyproject.toml --extra dds ``` > **Note:** on first invocation, the system sometimes cannot find the Unitree libraries. This should resolve by itself quickly. ### ORIN System Description Your development computer will (should) be at `192.168.123.99` The LIDAR is `192.168.123.120` The internal control computer (RockChip, aka the `operation and control computing unit`) is at `192.168.123.161` The internal development computer (Orin 16GB, aka the `development computing unit`) is at `192.168.123.164` Useful commands: ```bash sudo nmcli radio wifi on # Turn the wifi on/off sudo nmcli device wifi connect XXXXX password XXXXX # Join WiFi network sudo timedatectl set-ntp yes # Set time via NTP ``` #### ORIN Set default input and output Audio devices ```bash pactl list sources short # List input (microphone) devices pactl list sinks short # List output (speaker) devices pactl set-default-sink [SINK_NAME || SINK_ID] # Set default output device pactl set-default-source [SOURCE_NAME || SOURCE_ID] # Set default input device pactl set-sink-volume @DEFAULT_SINK@ 70% # Set default output volume ``` ### Control via Unitree Hand Controller Hang G1 on gantry\ Turn on (short press, long press)\ Wait for boot to complete\ When the G1 boots, it is in `damp` state\ Use the hand controller to command "L1+A" and "L1+UP"\\ The system is then ready to move using the `ai_sport` client. The system will respond to manual controller and SDK commands. * Press "L1+A" -> EMERGENCY DAMP / SINK TO FLOOR * Press "L1+UP" -> Stand firmly (aka "lock stand"). The arms will move slightly. The system is now in the "ready" state. * Lower the G1 to the ground (but do not unclip her yet). Stability not yet running - she will fall over if let go. * Press "R2+X" -> Start motion control. The arms will jump outwards and she will actively control her stability. * Press "Start" to switch back and forth between `stand` and `step in place`. Other actions: * "SELECT + Y" -> Wave Hand. Alternates sides. * "SELECT + A" -> Handshake. Hold for movement to complete. Wait 3s and press again to relax arm to initial state. * "SELECT + X" -> Turn around and wave hands. Use the joysticks to move forwards and backwards, and to rotate/turn ### Boot from Chair This is similar to boot from gantry, except, when you press "L1+UP", you have to help the humanoid stand up, while it straightens itself. For the sit-down procedure, back up the chair behind the robot, select "L1+LEFT", and help the humanoid settle back into the chair. ### Special DEBUG state **Avoid this mode** since it disables all high level motion since it turns off the `ai_sport` client. * Press L2+R2 -> Enter DEBUG STATE * Press L2+A -> Diagnostic Posture (Arms bent) * Press L2+B -> Relax arms, damping state To exit this mode, reboot the G1. ### Using the Internal Orin SSH to Orin via ```bash ssh unitree@192.168.123.164 ``` The default password is `123` but you should obviously change this. Result: ``` Welcome to Ubuntu 20.04.6 LTS (GNU/Linux 5.10.104-tegra aarch64) Last login: Thu Jan 1 08:30:46 1970 ros:foxy(1) noetic(2) ? ``` Select `foxy(1)`. ### Fixing the broken CycloneDDS installation on the Nvidia Orin The default installation of CycloneDDS on the G1 Orin is broken, since it does not support the newer `unitree_hg` IDL data format for the G1. Solution: remove the default CycloneDDS installation and reinstall following the [Unitree ROS2 installation instructions](https://github.com/unitreerobotics/unitree_ros2). The `unitree_hg` bug was fixed in early Dec. 2024 in this commit: unitreerobotics/unitree\_ros2\@b34fdf7. You will need to export suitable env variables and correct the setting in `.bashrc` and in `$HOME/unitree_ros2/setup.sh`. Add this to the `.bashrc`: ```bash export CYCLONEDDS_HOME="$HOME/unitree_ros2/cyclonedds_ws/install/cyclonedds" ``` Set `$HOME/unitree_ros2/setup.sh` to ```bash #!/bin/bash echo "Setup Unitree ROS2 Environment" source /opt/ros/foxy/setup.bash source $HOME/unitree_ros2/cyclonedds_ws/install/setup.bash export RMW_IMPLEMENTATION=rmw_cyclonedds_cpp export CYCLONEDDS_URI=' ' ``` Source the `setup.sh` via `source ~/unitree_ros2/setup.sh`. Finally, you should start to see data. ```bash version: - 0 - 0 mode_pr: 0 mode_machine: 4 tick: 1120016 imu_state: quaternion: - 0.9940093159675598 - 0.0074933432042598724 - -0.0023974007926881313 - -0.10901317745447159 gyroscope: - -0.00523598724976182 - -0.00523598724976182 - -0.0017453291220590472 accelerometer: - 0.009999999776482582 - -0.05000000074505806 - 9.829999923706055 rpy: - 0.015420285053551197 - -0.003132336074486375 - -0.21849146485328674 temperature: 78 motor_state: - mode: 1 q: -0.008177042007446289 dq: 0.0 ddq: 0.0 tau_est: -0.11186079680919647 temperature: - 29 - 28 vol: 49.0 sensor: - 0 - 0 motorstate: 0 reserve: - 0 - 1142 - 4 - 0 ``` ```bash mode_pr: 0 mode_machine: 4 motor_cmd: - mode: 1 q: 0.0 dq: 0.0 tau: 0.0 kp: 0.0 kd: 0.0 reserve: 0 ``` ```bash stamp: sec: 0 nanosec: 0 error_code: 0 imu_state: quaternion: - 0.9981062412261963 - 0.007613412104547024 - -0.0016289741033688188 - -0.06102222576737404 gyroscope: - 0.0 - 0.0 - 0.0 accelerometer: - 0.0 - 0.0 - 0.0 rpy: - 0.01539743971079588 - -0.002322605811059475 - -0.12214189022779465 temperature: 0 mode: 0 progress: 0.0 gait_type: 0 foot_raise_height: 0.0 position: - 0.0013194791972637177 - -0.018103253096342087 - 0.7247929573059082 body_height: 0.0 velocity: - 6.116795248090057e-07 - -1.2412459682309418e-06 - -4.238392648403533e-05 yaw_speed: 0.0017453291220590472 range_obstacle: - 0.0 ... foot_force: - 0 - 0 - 0 - 0 foot_position_body: - 0.0 ... foot_speed_body: - 0.0 ... ``` ```bash /EstimatorData /SymState /SymState_back /api/bashrunner/request /api/bashrunner/response /api/config/request /api/config/response /api/loco/request /api/loco/response /api/motion_switcher/request /api/motion_switcher/response /api/robot_state/request /api/robot_state/response /arm_sdk /audiosender /config_change_status /dex3/left/cmd /dex3/left/state /dex3/right/cmd /dex3/right/state /frontvideostream /gnss /lf/bmsstate /lf/dex3/left/state /lf/dex3/right/state /lf/lowstate /lf/lowstate_doubleimu /lf/mainboardstate /lf/odommodestate /loco_sdk /lowcmd /lowstate /lowstate_doubleimu /multiplestate /odommodestate /parameter_events /public_network_status /rosout /rtc/state /rtc_status /selftest /servicestate /servicestateactivate /videohub/inner /webrtcreq /webrtcres /wirelesscontroller ``` Here is what will be visible on an external development machine at `.99`, for example, using `./bin/listtopics`: ```bash rt/dex3/right/state unitree_hg::msg::dds_::HandState_ rt/lf/dex3/right/state unitree_hg::msg::dds_::HandState_ rt/dex3/right/cmd unitree_hg::msg::dds_::HandCmd_ rt/EstimatorData unitree_go::msg::dds_::EstimatorData_ rt/SymState_back unitree_go::msg::dds_::SymState_ rt/odommodestate unitree_go::msg::dds_::SportModeState_ rt/lf/odommodestate unitree_go::msg::dds_::SportModeState_ rt/lowstate unitree_hg::msg::dds_::LowState_ rt/SymState unitree_go::msg::dds_::SymState_ rt/api/bashrunner/response unitree_api::msg::dds_::Response_ rt/selftest std_msgs::msg::dds_::String_ rt/api/bashrunner/request unitree_api::msg::dds_::Request_ rt/lf/lowstate unitree_hg::msg::dds_::LowState_ rt/api/motion_switcher/response unitree_api::msg::dds_::Response_ rt/api/motion_switcher/request unitree_api::msg::dds_::Request_ rt/config_change_status unitree_go::msg::dds_::ConfigChangeStatus_ rt/api/config/response unitree_api::msg::dds_::Response_ rt/api/config/request unitree_api::msg::dds_::Request_ rt/api/robot_state/response unitree_api::msg::dds_::Response_ rt/api/robot_state/request unitree_api::msg::dds_::Request_ rt/servicestate std_msgs::msg::dds_::String_ rt/multiplestate std_msgs::msg::dds_::String_ rt/public_network_status std_msgs::msg::dds_::String_ rt/gnss std_msgs::msg::dds_::String_ rt/lf/bmsstate unitree_hg::msg::dds_::BmsState_ rt/lf/mainboardstate unitree_hg::msg::dds_::MainBoardState_ rt/webrtcreq std_msgs::msg::dds_::String_ rt/webrtcres std_msgs::msg::dds_::String_ rt/lowcmd unitree_hg::msg::dds_::LowCmd_ rt/lowstate_doubleimu unitree_hg_doubleimu::msg::dds_::doubleIMUState_ rt/lf/lowstate_doubleimu unitree_hg_doubleimu::msg::dds_::doubleIMUState_ rt/wirelesscontroller unitree_go::msg::dds_::WirelessController_ rt/frontvideostream unitree_go::msg::dds_::Go2FrontVideoData_ rt/audiosender unitree_go::msg::dds_::AudioData_ rt/servicestateactivate std_msgs::msg::dds_::String_ rt/rtc_status std_msgs::msg::dds_::String_ rt/videohub/inner std_msgs::msg::dds_::String_ rt/rtc/state std_msgs::msg::dds_::String_ rt/api/bashrunner/request unitree_api::msg::dds_::Request_ rt/lf/dex3/left/state unitree_hg::msg::dds_::HandState_ rt/dex3/left/state unitree_hg::msg::dds_::HandState_ rt/dex3/left/cmd unitree_hg::msg::dds_::HandCmd_ ``` ### Terminal based setup of Bluetooth audio devices This is only needed on the headless Orin, otherwise (e.g. on the Mac) just use the system settings. ```bash bluetoothctl list # show all paired devices scan on # search for nearby devices # once you have found the right device, you can then pair it # many devices also require 'trusting' them trust pair connect ``` # Unitree Go2 Quadruped Unitree Go2 EDU Quadruped (dog) OM1 can control a Unitree Go2 EDU out of the box. This has been tested on Nvidia Orin, Mac Mini, and current (silicon) Mac laptops. ### Step 1 - Establishing Ethernet and DDS Connectivity Connect the Unitree Go2 EDU to your development machine with an Ethernet cable. Open the network settings and find the network interface that is connected to the Go2 EDU. In the IPv4 settings, change the IPv4 mode to `manual`, set the address to `192.168.123.99`, and set the mask to `255.255.255.0`. After completion, click `apply` (or equivalent) and wait for the network to reconnect. Provide the name of the network adapter in the `"unitree_ethernet": "en0"` entry in the `unitree_go2.config` file. Then, install [`CycloneDDS`](https://index.ros.org/p/cyclonedds/). `CycloneDDS` works on Mac, Linux, and PC. Run: ```bash git clone https://github.com/eclipse-cyclonedds/cyclonedds -b releases/0.10.x cd cyclonedds && mkdir build install && cd build cmake .. -DCMAKE_INSTALL_PREFIX=../install -DBUILD_EXAMPLES=ON cmake --build . --target install ``` Next, set `CYCLONEDDS_HOME`, `CMAKE_PREFIX_PATH`, and `CYCLONEDDS_URI` to the correct values for your computer. Example settings for a typical Mac installation are provided below. You should add these paths to your environment via your `.zshrc` or equivalent. ```bash export CYCLONEDDS_HOME=$HOME/Documents/GitHub/cyclonedds/install export CMAKE_PREFIX_PATH=$HOME/Documents/GitHub/cyclonedds/install export CYCLONEDDS_URI=' true ' ``` Then, compile and run the `listtopics` example: ```bash cd $HOME/Documents/GitHub/cyclonedds/install/share/CycloneDDS/examples/listtopics cmake . cmake --build . ./listtopics ``` On Mac, you **might** need to `allow incoming connections` in the popup the first time you run `listtopics`. Running `listtopics` should result in an extensive data dump of available topics: ```bash alive: df4efe0e:812a86b1:647728be:c1f7a312 rt/lf/lowstate unitree_go::msg::dds_::LowState_ alive: c3a612c2:99b329c6:22510d3d:d385e1e6 rt/api/motion_switcher/response unitree_api::msg::dds_::Response_ alive: 78b1db7f:622f2dfe:fb4d9e8f:987083f0 rt/api/motion_switcher/request unitree_api::msg::dds_::Request_ alive: 84978827:0d460527:b4f054ac:e546468a rt/api/gpt/request unitree_api::msg::dds_::Request_ alive: 23d08ca2:bea974c8:d44a51c0:d2c3cf27 rt/api/gpt/response unitree_api::msg::dds_::Response_ alive: ac63fb3a:bb2b8a5b:c1a42ed4:bf470e91 rt/gptflowfeedback std_msgs::msg::dds_::String_ alive: fc5e351e:00f676bb:236ec7d8:da9f115b rt/api/sport/request unitree_api::msg::dds_::Request_ alive: b1be3e06:596bd40d:8ef51579:50245496 rt/api/sport/response unitree_api::msg::dds_::Response_ alive: da491277:72915ef6:c7407d32:fb2ce5fa rt/api/videohub/request unitree_api::msg::dds_::Request_ alive: 9072826b:ffcd904a:ba8a0092:792335d7 rt/api/videohub/response unitree_api::msg::dds_::Response_ alive: 077c7627:183247d6:2a698bc3:f8df4577 rt/utlidar/range_info geometry_msgs::msg::dds_::PointStamped_ alive: 188652eb:bf16b2a2:dbc4ac3a:af71d576 rt/lf/sportmodestate unitree_go::msg::dds_::SportModeState_ alive: 23dd31a3:b6828a1e:cb390d4f:1a9a1d87 rt/gpt_cmd std_msgs::msg::dds_::String_ alive: c53b329d:7aa96ef0:80c9d35d:d8459b42 rt/api/vui/request unitree_api::msg::dds_::Request_ alive: 87b1b32e:d9ec09ce:b928db03:eaf5f28f rt/api/vui/response unitree_api::msg::dds_::Response_ alive: 4d0e307c:c20328f4:0dd1efc5:7ca1e56c rt/mf/sportmodestate unitree_go::msg::dds_::SportModeState_ alive: 083d7026:d0ac857d:ad12c938:2b98f89a rt/utlidar/height_map_array unitree_go::msg::dds_::HeightMap_ alive: 06f05795:ebf8f837:fb736321:16cd723d rt/wirelesscontroller unitree_go::msg::dds_::WirelessController_ alive: 3f2bd6b9:22da59bf:ca7b2ec6:2ab25639 rt/api/obstacles_avoid/response unitree_api::msg::dds_::Response_ alive: 3392a31a:eec44934:a234ee4e:e9b0645d rt/api/obstacles_avoid/request unitree_api::msg::dds_::Request_ alive: 9c38174a:b9719d10:5ceaf5d6:58cad182 rt/api/config/request unitree_api::msg::dds_::Request_ alive: 243d1a9a:beb73e99:a7438b89:8881ba7a rt/api/config/response unitree_api::msg::dds_::Response_ alive: 1ffd3f85:9e4cbd10:b07286be:95b31d4c rt/api/sport_lease/response unitree_api::msg::dds_::Response_ alive: ab8cdb14:cb59d2c1:7dba035c:424e94c5 rt/api/sport_lease/request unitree_api::msg::dds_::Request_ alive: 232c059d:f9650b34:ee28aae8:da3ab7d6 rt/lowcmd unitree_go::msg::dds_::LowCmd_ alive: a9cf187b:009f8f56:bfdf14b4:c2525358 rt/sportmodestate unitree_go::msg::dds_::SportModeState_ alive: 282f8688:c4486d0f:ae222a71:f33dc3b9 rt/lowstate unitree_go::msg::dds_::LowState_ alive: e553c276:3cdb6d66:52898f8b:59b25a8c rt/config_change_status unitree_go::msg::dds_::ConfigChangeStatus_ alive: 699e3ec7:13005f67:4165667f:a84d4593 rt/webrtcreq std_msgs::msg::dds_::String_ alive: 921b611e:2649a718:a59ee5cd:22dc2309 rt/webrtcres std_msgs::msg::dds_::String_ alive: 4e0a5acf:15405a12:f6e929ea:5c320a0a rt/api/audiohub/request unitree_api::msg::dds_::Request_ alive: 873309ee:12ef5d7c:8cd557b7:00b1931c rt/api/audiohub/response unitree_api::msg::dds_::Response_ alive: e7a31796:64c222e8:a8f3e215:22e11e24 rt/rtc/state std_msgs::msg::dds_::String_ alive: 80ba3475:34843e73:c939481c:0db3198d rt/audiohub/player/state std_msgs::msg::dds_::String_ alive: 82fb9789:0b61fd9c:5a20c796:876944da rt/api/fourg_agent/response unitree_api::msg::dds_::Response_ alive: 63540cad:26e63d68:0b2a3b41:0a431b21 rt/api/fourg_agent/request unitree_api::msg::dds_::Request_ alive: 0582765e:bebc0d3c:31f2aa76:849d52d7 rt/public_network_status std_msgs::msg::dds_::String_ alive: 5a1e209f:77622587:dff56d06:100b934a rt/gnss std_msgs::msg::dds_::String_ alive: 5c4904ed:4cdcda1f:17498349:91cc9154 rt/api/uwbswitch/request unitree_api::msg::dds_::Request_ alive: 40ce3a5c:80f4f2e4:50679aee:073b34d7 rt/api/uwbswitch/response unitree_api::msg::dds_::Response_ alive: c2bd3891:436200a6:bcf7d9fe:67115b31 rt/api/bashrunner/response unitree_api::msg::dds_::Response_ alive: c0acff0c:48bc2d73:b9c0bc78:9a39ac86 rt/selftest std_msgs::msg::dds_::String_ alive: 22066ccf:337367fe:bfc8f45d:07cb2de6 rt/api/bashrunner/request unitree_api::msg::dds_::Request_ alive: e1235a04:2b4036af:97482629:7591bc73 rt/utlidar/cloud sensor_msgs::msg::dds_::PointCloud2_ alive: 7daf47c7:581b2844:c890ed12:95e30a2e rt/utlidar/cloud_deskewed sensor_msgs::msg::dds_::PointCloud2_ alive: 4a8d54dc:e5ee8a08:fe3219d9:07f76dcf rt/utlidar/lidar_state unitree_go::msg::dds_::LidarState_ alive: 44260812:9b9f6a04:af980abc:e73a77fe rt/utlidar/switch std_msgs::msg::dds_::String_ alive: 749c1f9a:4b0cec32:19b92633:17e62e1d rt/utlidar/robot_odom nav_msgs::msg::dds_::Odometry_ alive: 79ce8c20:07aae9f8:c523d89c:b110b187 rt/utlidar/robot_pose geometry_msgs::msg::dds_::PoseStamped_ alive: 2c16a46d:da94a86f:d52ef5cc:7051a9fa rt/utlidar/foot_position sensor_msgs::msg::dds_::PointCloud2_ alive: 4384ffd2:e20ddb0e:f7b76a2c:d2c6a30c rt/utlidar/imu sensor_msgs::msg::dds_::Imu_ alive: f3aab9a3:95247169:cfbf8263:c5a9b106 rt/utlidar/mapping_cmd std_msgs::msg::dds_::String_ alive: 22359611:515cc56c:deeb92c9:c94220cf rt/uslam/client_command std_msgs::msg::dds_::String_ alive: 3dcca3f4:78429e39:564ca64f:05a1ece3 rt/uslam/cloud_map sensor_msgs::msg::dds_::PointCloud2_ alive: 4cc72605:f2246ae7:b747729e:64efbe90 rt/uslam/server_log std_msgs::msg::dds_::String_ alive: ebdc8a4a:9e5c19b5:26543bcc:0fa10748 rt/utlidar/voxel_map sensor_msgs::msg::dds_::PointCloud2_ alive: d447621e:3988c1dc:f66702de:0686316c rt/utlidar/voxel_map_compressed unitree_go::msg::dds_::VoxelMapCompressed_ alive: 5e812cab:094b8728:1b5e3e01:e016ee20 rt/utlidar/height_map sensor_msgs::msg::dds_::PointCloud2_ alive: 2de95d6a:9bb1ad47:ffd2e68a:e630fc84 rt/utlidar/range_map sensor_msgs::msg::dds_::PointCloud2_ alive: 46a6be60:38d89c4b:2c15e85e:1de4d940 rt/utlidar/map_state unitree_go::msg::dds_::VoxelHeightMapState_ alive: 35ccc911:8434f2c6:dcde737d:9e2be105 rt/utlidar/grid_map sensor_msgs::msg::dds_::PointCloud2_ alive: 1fdd190f:1809f14a:cbcf54a8:83382be7 rt/uwbstate unitree_go::msg::dds_::UwbState_ alive: 7474c70d:75a1ec58:39efc2e5:9dde2533 rt/api/programming_actuator/response unitree_api::msg::dds_::Response_ alive: 42699445:5dcf6a55:4ed227af:6ba4ae91 rt/api/programming_actuator/request unitree_api::msg::dds_::Request_ alive: 47a98910:be44d5c4:55ef536f:30e24971 rt/wireless_controller unitree_go::msg::dds_::WirelessController_ alive: 1c3b1069:4aa20466:30a3036b:c6b59fa8 rt/frontvideostream unitree_go::msg::dds_::Go2FrontVideoData_ alive: b998bf1b:6930232f:a9a62e3e:b1785122 rt/videohub/inner std_msgs::msg::dds_::String_ alive: 42c7a686:b0e92d30:fe95d5c0:9af26c6f rt/api/robot_state/response unitree_api::msg::dds_::Response_ alive: 09866cdd:dd97fa27:978c004f:3624dace rt/api/robot_state/request unitree_api::msg::dds_::Request_ alive: fb0e8c0f:5f108434:e87e9cf7:7848477d rt/servicestate std_msgs::msg::dds_::String_ alive: 6e9b68c3:8cb0df52:98603f8c:b934b1f6 rt/multiplestate std_msgs::msg::dds_::String_ alive: 6f564b14:8d2a936a:e3ac501c:d0fa740f rt/audiosender unitree_go::msg::dds_::AudioData_ alive: ab2e56b3:0906d3af:9a0efbb8:2c0fbd2b rt/uwbswitch unitree_go::msg::dds_::UwbSwitch_ alive: edf34ede:ec71ed87:ff6d918b:fd6177f9 rt/servicestateactivate std_msgs::msg::dds_::String_ alive: da41bb3b:e14530da:3288e103:dcf8888a rt/wirelesscontroller_unprocessed unitree_go::msg::dds_::WirelessController_ alive: 8a575fb2:ad676bfd:a3b20683:ae69b9af rt/rtc_status std_msgs::msg::dds_::String_ alive: 05d38f9b:d5113e6e:14601dbc:42e800ca rt/api/bashrunner/request unitree_api::msg::dds_::Request_ alive: 82a94751:dd399ada:7d9e6a9f:d7a0176e rt/api/gas_sensor/request unitree_api::msg::dds_::Request_ alive: ba193460:48bd7d2f:116da37d:f30b6c42 rt/qt_command unitree_interfaces::msg::dds_::QtCommand_ alive: d4f5d6cb:b6399171:13f983fb:45b46d7a rt/qt_add_node unitree_interfaces::msg::dds_::QtNode_ alive: 6f31b8ea:188ccefd:0bfd96dc:3a37a02f rt/qt_add_edge unitree_interfaces::msg::dds_::QtEdge_ alive: d1df34ff:1eab5cca:dd6efa75:2d6822cb rt/arm_Command unitree_arm::msg::dds_::ArmString_ alive: ea56b746:45db6798:5bc61259:56a9ced8 rt/programming_actuator/command std_msgs::msg::dds_::String_ alive: bcfd7290:715c3062:f749cd88:65a00f50 rt/api/gas_sensor/response unitree_api::msg::dds_::Response_ alive: cbda9e6d:70ba0985:dde9d637:788a68c1 rt/query_result_node unitree_interfaces::msg::dds_::QtNode_ alive: 3e9b2d75:a0c37b00:84a6a836:0d551674 rt/query_result_edge unitree_interfaces::msg::dds_::QtEdge_ alive: caa2c5ce:1b35a561:838107c5:426dc33b rt/qt_notice std_msgs::msg::dds_::String_ alive: d04fbb81:2460ecfd:65992300:0488c9e0 rt/pctoimage_local unitree_interfaces::msg::dds_::PcToImage_ alive: 0a0535a3:698337d4:c18b0e88:881cfe6e rt/lio_sam_ros2/mapping/odometry nav_msgs::msg::dds_::Odometry_ alive: 38c5e268:9cf0ebc1:740c30c6:aeb63f0a rt/arm_Feedback unitree_arm::msg::dds_::ArmString_ alive: ed0192ea:695ebbbd:9c0704a9:44a56576 rt/gas_sensor std_msgs::msg::dds_::String_ alive: 56b92f32:a4984539:7d02a691:72b8ed71 rt/uslam/frontend/cloud_world_ds sensor_msgs::msg::dds_::PointCloud2_ alive: 11641fd3:41e4e795:eb3a132d:6ba0b093 rt/uslam/frontend/odom nav_msgs::msg::dds_::Odometry_ alive: c0263c37:58c215e0:a5cf9018:0255fb8e rt/uslam/localization/odom nav_msgs::msg::dds_::Odometry_ alive: a3bb4e5c:ca3c645a:58b4a065:a84059ab rt/uslam/navigation/global_path sensor_msgs::msg::dds_::PointCloud2_ alive: 1453e55f:a4d5babe:aec670aa:6c0a9bff rt/uslam/localization/cloud_world sensor_msgs::msg::dds_::PointCloud2_ alive: 1a6535ff:d7346fd8:87885024:9a136972 rt/programming_actuator/feedback std_msgs::msg::dds_::String_ ``` ### Step 2 - Add the Python CycloneDDS module Add the `dds` python module to your code base: `uv pip install -r pyproject.toml --extra dds`. ### Step 3 - Connect an Game Controller On Mac, if you are using an Xbox controller, **make sure that it is running the most recent firmware**, otherwise you will be able to pair but not connect to the controller. You will need a PC with `Xbox Accessories` to update the Xbox controller's firmware. Many Mac OS updates hose support for the Xbox controller, which is then fixed (after some delay) via an Xbox firmware update. On Mac, you will need to install `hidapi`: ```bash brew install hidapi ``` NOTE: There is a bug on Mac when installing packages with `brew` - some libraries cannot be found by `uv`. If you get errors such as `Unable to load any of the following libraries:libhidapi-hidraw.so`, set `export DYLD_FALLBACK_LIBRARY_PATH=$HOMEBREW_PREFIX/lib` in your `.zshenv` or equivalent. On Linux, install `hidapi` like this: ```bash sudo apt-get update sudo apt-get install python3-dev libusb-1.0-0-dev libudev-dev libhidapi-dev ``` ### Accessing Unitree Data For debugging, you can access Unitree data as follows: Front video feed: ```bash uv run system_hw_test/go2_camera_opencv.py en0 ``` Front video single image: ```bash uv run system_hw_test/go2_capture_image.py en0 ``` Lowstate system/joint data: ```bash uv run system_hw_test/go2_data_stream.py en0 ``` Lidar: ```bash uv run system_hw_test/go2_lidar.py en0 ``` > **Note**: the internal Go2 LIDAR is currently not used - a separate RPLIDAR mounted to the top of the dog's head is used instead. ### Minimal Quadruped Functionality In this configuration, the quadruped observes its environment, listens and speaks, but there is no AI-controlled movement. You can manually control the dog's movements with a game controller. Run ```bash make run CONFIG=unitree_go2_basic ``` Press: * A to stand up * B to sit down * The D-pad allows you to steer the quadruped. See the [Quadruped Configurations](/robotics/unitree_go2_quadruped_configurations) for additional configurations. ### Unitree Go2 EDU Common Problems *Channel factory init error*: If you see a `channel factory init error`, then you have not set the correct network interface adapter - the one you want to use is the network interface adapter *on your development machine - the computer you are currently sitting in front of* that is plugged into the Unitree quadruped (which has its own internal RockChip computer and network interface, which is *not* relevant to you right now). The Ethernet adapter - such as `eno0` or `en0` - needs to be set in the `"unitree_ethernet": "en0"` entry in the `unitree_go2.config` file. *The CycloneDDS library could not be located*: You did not install CycloneDDS (see above), or, you did not provide a path to the `/install`, via `export CYCLONEDDS_HOME=$HOME/Documents/GitHub/cyclonedds/install` or equivalent. *"nothing is working"* There are dozens of potential reasons "nothing is working". The first step is to test your ability to `ping` the quadruped motion control computer: ```bash ping 192.168.123.161 ``` Assuming you can `ping` the robot, then test the CycloneDDS middleware (see **STEP 1**). Once you see data flowing, then the rest of the system should work. # Raspberry Pi Getting Started with OM1 on Raspberry Pi ### Basics Make sure you have the following before installation: * Device: Raspberry Pi 4 or 5 (4GB RAM or more recommended) * OS: 64-bit Raspberry Pi OS or Debian-based 64-bit Linux distro * Storage: At least 16GB free * Network: Internet access Then, install OM1 on the Raspberry Pi from its command line, following the [standard instructions](/developing/1_get-started). ### Done You’re now running OM1 on your Raspberry Pi. Explore its capabilities and refer to the [OM1 GitHub repository](https://github.com/openmind/OM1) for advanced settings and next steps. # Tesla Dimo Bring your Tesla to Life We work with DIMO to bring your Tesla to life. This plugin allows you to read data from your Tesla and control it. ### Prerequisites #### Step 0: Install the DIMO App Before you can use the DIMO APIs, you need to install the DIMO app on your phone and connect it to your Tesla. #### Step 1: Get a Developer License To access the DIMO APIs, you must first obtain and configure a [Developer License](https://docs.dimo.org/developer-platform/developer-guide/developer-console#getting-a-license) from DIMO. Once you have the license, you will receive the following credentials: `Client ID`, `API key`, and `Redirect URI`. #### Step 2: Log in with DIMO Replace `` and `` with your own credentials, then open the following URL in your browser to log in with DIMO and authorize your application to access your Tesla data. You will obtain your `car ID` (`token_id`) on the permission page. ```dimo https://login.dimo.org/?clientId=&redirectUri=&permissionTemplateId=1&entryState=VEHICLE_MANAGER ``` #### Step 3: Set up the Tesla Virtual Key Go to the [Tesla Virtual Key](https://www.tesla.com/_ak/auth.drivedimo.com) page and follow the instructions to set up the virtual key for DIMO. If everything is set up correctly, you should be able to view your Tesla data in the DIMO app and control your Tesla via the DIMO API. > **Note:** If you encounter any issues, please refer to the [DIMO documentation](https://docs.dimo.org/developer-platform/developer-guide/dimo-developer-sdks/data-sdk). ### Basic Commands Run ```bash make run CONFIG=tesla ``` ### Configuration Provide the Dimo `client_id`, `domain`, `private_key` and `token_id` obtained from the previous steps in your configuration file (`/config.tesla.json5`): ```bash "client_id": "", "domain": "", "private_key": "", "token_id": 0 ``` # TurtleBot4 TurtleBot4 Basic Setup ### Establishing Basic Networking and Connectivity For many development tasks, you will need to `ssh` to the RPi on the TB4 from your laptop: ```bash ssh ubuntu@192.168.1.116 # use your TB4's IP address ``` The password is `turtlebot4`. Once on the command line, you can use `turtlebot4-setup` to configure ROS2, the WiFi, etc. ### Required Operating System Versions OM1 supports TurtleBot4 running ROS2 Humble (image 1.0.4) and Create3 with H.2.6. #### Required Software for the Raspberry Pi * Ubuntu 22.04.4 LTS (GNU/Linux 5.15.0-1073-raspi aarch64) - combined with TurtleBot4 Humble image * [turtlebot4\_standard\_humble\_1.0.4.img (2024-08-19 16:59 2.2G)](http://download.ros.org/downloads/turtlebot4/turtlebot4_standard_humble_1.0.4.zip) Note: choose TurtleBot4 lite image file if your TurtleBot4 is the lite version. #### Required Software for the Create3 [H.2.6 (Humble)](https://github.com/iRobotEducation/create3_docs/releases/download/H.2.6/Create3-H.2.6.swu) **Important: Make sure your Raspberry Pi 4 is running Ubuntu 22.04.4 LTS (Turtlebot4 Humble image) and that the Create 3 is running H.2.6/Create3-H.2.6.swu**. Among other possible problems, running Ubuntu 24 will create compatibility issues with the version of ROS2 and CycloneDDS on the Create3. ### Prepare the TurtleBot4 1. Flash the correct TurtleBot4 image to an SD card (if needed). 2. Upgrade/downgrade the Create3 firmware to H.2.6 (if needed). 3. Insert the SD card into the TurtleBot4 and power it on. 4. Set up your TurtleBot4 following the [Basic Setup](https://turtlebot.github.io/turtlebot4-user-manual/setup/basic.html#robot). ### Identity and API Keys **API\_KEY** Go to [portal](https://portal.openmind.com/) to get a free API key for the OM1 APIs. Enter this API key in the "api\_key" field in the `/config/turtlebot4.json5` file. You can also provide this API key via your `.env` - just enter it as: ```bash OM_API_KEY=om1_live_e4252f1cf005af... ``` **UNIVERSAL\_ROBOT\_ID (URID)** Go to "[Hello Robots, come join us](https://portal.openmind.com/robots)" to join a decentralized machine<>machine coordination and communication system (FABRIC). Enter machine metadata - currently an arbitrary string - and click "join". The system will provide a unique URID for your robot. The URIDs all share the same format: they begin with `OM`, then 12 alphanumeric characters (numerals and letters), adding up to 14 characters in total. They're not case sensitive. A unique URID allows multiple robots to communicate with one another, similar to how humans use different phone numbers to help them communicate and coordinate. Enter the URID in the "URID" field in the `/config/turtlebot4.json5` file. You can also provide the URID via your .env - just enter it as: ```bash URID=OM742d35Cc6634 # yours will be different! ``` > **Note:** for testing you can use any short string as the URID, as long as it's unique for each robot within a team of robots within the same local network. ### Configure the TurtleBot4 Once the TurtleBot4 is set up, configure it as follows: * Configure the RPi4 through `turtlebot4-setup` * Configure the Create3 through its web server * Install docker and configure the `docker-compose.yaml` file #### Configure the TurtleBot4 Internal RPi4 Make the following `ROS_DOMAIN_ID` changes to the TurtleBot4. Use the `turtlebot4-setup` tool to access `ROS Setup:Bash Setup` and set it to the following: ```bash ROBOT_NAMESPACE=/_your_robot_URID_/pi # example: ROBOT_NAMESPACE=/OM742d35Cc6634/pi ROS_DOMAIN_ID=0 RMW_IMPLEMENTATION=rmw_cyclonedds_cpp CYCLONEDDS_URI=[] # Empty ``` Click "Save", "Esc", and then "Apply Settings". The TurtleBot4 will reboot. #### Configure the TurtleBot4 Internal Create3 Once the reboot is complete (wait for chime, 1 min), access the Create3's App config page at its web server (e.g. `192.168.1.XXX:8080/ros-config`). Change the ROS\_DOMAIN\_ID to 1 and enter your robot's URID. The correct settings are: ``` ROS 2 Domain ID (default 0): 1 ROS 2 Namespace: /_your_robot_URID_/c3 # example: /OM742d35Cc6634/c3 RMW_IMPLEMENTATION: rmw_cyclonedds_cpp ``` Basically, you are using the "/c3" prefix to create a unique namespace for the c3. Click `Save` and *Restart Application*. Do not forget to click *Restart Application*, otherwise the changes will not be applied. Wait for chime (1 min) indicating Create3 reboot. #### Install Docker Finally, on the TurtleBot4's RPi, [install Docker](https://docs.docker.com/engine/install/ubuntu/) and run `sudo docker compose -f docker-compose.yaml up -d`. The `docker-compose` should be: ```bash services: zenoh-bridge-turtlebot4: image: openmindagi/turtlebridge container_name: zenoh-bridge-turtlebot4 network_mode: "host" restart: always # Ensures the container restarts on reboot ``` The TurtleBot4 will now be more stable, can discover other computers running Zenoh, send them Zenoh messages, and also, accept Zenoh messages and forward them to ROS2. ### OM1 Installation and Launch You can install and run OM1 * on your laptop, or * onboard the Raspberry Pi on the TurtleBot4 #### Running OM1 on the Internal RPi4 For fully autonomous use, install OM1 on the TurtleBot4's Raspberry Pi. On the RPi terminal command line, follow these [instructions](/developer-cookbook/om1-integration-with-different-machines/raspberrypi). When you see all the right topics listed in `ros2 topic list`, your TurtleBot4 is set up and you are ready to install OM1. Install `uv` - Python package manager: ```bash curl -LsSf https://astral.sh/uv/install.sh | sh ``` Install `portaudio`, `ffmpeg` and other dependencies: ```bash sudo apt install pulseaudio pulseaudio-utils ffmpeg portaudio19-dev ``` Then, connect a Logitech 270 Webcam (or equivalent). Unplug the standard TurtleBot4 Depth Camera. Connect a speaker to RPi with a 3.5 mm audio patch cable (or use bluetooth, depending on your pain threshold and patience with debugging bluetooth issues). Make sure [Identity and Keys](#identity-and-keys) are set correctly. To update the `.env` with your OM1 key, run `vim .env`, "i" to insert, paste in key, "ESC", ":", "wq" to write and exit. Finally, set Default Input and Output Audio devices. Use `pactl` to set your default microphone and speaker. If you get `pa_context_connect() failed: Connection refused`, then start the audio daemon manually via `pulseaudio --start -D`. ```bash pactl list sources short # List input (microphone) devices pactl list sinks short # List output (speaker) devices pactl set-default-sink [SINK_NAME || SINK_ID] # Set default output device pactl set-default-source [SOURCE_NAME || SOURCE_ID] # Set default input device pactl set-sink-volume @DEFAULT_SINK@ 100% # Set default output volume # for example, pactl set-default-source alsa_input.usb-046d_C270_HD_WEBCAM_2CD9A910-02.mono-fallback pactl set-default-sink alsa_output.platform-bcm2835_audio.stereo-fallback ``` * Launch OM1 ```bash make run CONFIG=turtlebot4_lidar_gps ``` On the RPi4, there will be a lengthy delay before OM1 runs the first time you invoke this command. #### Running OM1 on your Laptop Install OM1 and Zenoh on your laptop, following [Install OM1 on your laptop](/developing/1_get-started) and [Installing the Zenoh router](https://zenoh.io/docs/getting-started/installation/). Then, run OM1: ```bash make run CONFIG=turtlebot4 ``` Please make sure that your development machine (e.g. your laptop) is running the same version of the Zenoh bridge (e.g. 1.2.1) as the RPi otherwise you will get errors. When you run OM1 on your laptop, OM1 will use your laptop's microphone, speaker, and camera inputs/outputs, rather than the sensors on the TurtleBot4. #### Interacting with TurtleBot4 from a Remote Computer using Zenoh and the Command Line This is useful for debugging. Inside `system_hw_test`, there are several scripts for you to interact with the TurtleBot4. For all these scripts, provide the robot's URID, such as `OM742d35Cc6634` as an argument. **Steer the TurtleBot4 with your Laptop Keyboard** ```bash uv run turtlebot4_keyboard_movement.py --URID OM123435Cc1234 ``` * W - Move Forward * S - Move Backward * A - Turn Left * D - Turn Right **Read TurtleBot4 Laserscan Data** Provide the robot's URID, such as `OM742d35Cc6634` as an argument: ```bash uv run rptest.py --URID OM123435Cc1234 ``` **Read TurtleBot4 Battery Data** Provide the robot's URID, such as `OM742d35Cc6634` as an argument: ```bash uv run turtlebot4_battery.py --URID OM123435Cc1234 ``` ### Expected ROS2 Topics for a Correctly Configured System On the TurtleBot4 command line, run `ros2 topic list`. Topics with a `pi` prefix originate from the RPi, and topics with a `c3` prefix are from the Create3. If you do not see any non-prefixed topics, your Create3 is not talking correctly to the RPi. ```bash /OM742d35Cc6634/c3/battery_state /OM742d35Cc6634/c3/cmd_vel /OM742d35Cc6634/c3/dock_status /OM742d35Cc6634/c3/hazard_detection /OM742d35Cc6634/c3/imu /OM742d35Cc6634/c3/odom /OM742d35Cc6634/c3/robot_state/transition_event /OM742d35Cc6634/c3/static_transform/transition_event /OM742d35Cc6634/c3/tf /OM742d35Cc6634/c3/tf_static /OM742d35Cc6634/pi/battery_state /OM742d35Cc6634/pi/cmd_vel /OM742d35Cc6634/pi/diagnostics /OM742d35Cc6634/pi/diagnostics_agg /OM742d35Cc6634/pi/diagnostics_toplevel_state /OM742d35Cc6634/pi/dock_status /OM742d35Cc6634/pi/function_calls /OM742d35Cc6634/pi/hazard_detection /OM742d35Cc6634/pi/hmi/buttons /OM742d35Cc6634/pi/hmi/display /OM742d35Cc6634/pi/hmi/display/message /OM742d35Cc6634/pi/hmi/led /OM742d35Cc6634/pi/imu /OM742d35Cc6634/pi/interface_buttons /OM742d35Cc6634/pi/ip /OM742d35Cc6634/pi/joint_states /OM742d35Cc6634/pi/joy /OM742d35Cc6634/pi/joy/set_feedback /OM742d35Cc6634/pi/mouse /OM742d35Cc6634/pi/oakd/imu/data /OM742d35Cc6634/pi/oakd/rgb/preview/camera_info /OM742d35Cc6634/pi/oakd/rgb/preview/image_raw /OM742d35Cc6634/pi/oakd/rgb/preview/image_raw/compressed /OM742d35Cc6634/pi/oakd/rgb/preview/image_raw/compressedDepth /OM742d35Cc6634/pi/oakd/rgb/preview/image_raw/theora /OM742d35Cc6634/pi/robot_description /OM742d35Cc6634/pi/scan /OM742d35Cc6634/pi/tf /OM742d35Cc6634/pi/tf_static /OM742d35Cc6634/pi/wheel_status /diagnostics /parameter_events /rosout ``` ### Building the Docker Dual Bridge Images You can build your own dual bridge docker images using the provided `Dockerfile` (see `/system_hw_test/turtlebot_zenoh/Dockerfile`): ```bash docker build -t my-username/my-image . docker push my-username/my-image ``` Useful docker commands: ```bash sudo docker compose pull # pull latest image # Images sudo docker images sudo docker rmi IMAGE_ID --force # Containers sudo docker ps -a sudo docker attach CONTAINER_ID # to stream logs sudo docker exec -it CONTAINER_ID sh # to get a shell sudo docker kill CONTAINER_ID ``` ### Debugging Commands Webcam debugging: ```bash lsusb sudo apt install v4l-utils v4l2-ctl --list-devices v4l2-ctl -d /dev/video0 --stream-mmap --all ``` # UBTech Yanshee UbTech Yanshee Setup (Mini Humanoid) ## Hardware * A laptop/PC * A mobile phone * UbTech Yanshee (mini humanoid robot) ## Software * Yanshee mobile app * OM1 * [OM1 API key](https://portal.openmind.com/) ## Installation First, install the Yanshee mobile app on your phone. This will allow you to retrieve robot's IP address. Then, install OM1 on your laptop or PC, following the [standard instructions](/developing/1_get-started). ## Connecting to the Robot * Switch on your Bluetooth and location. * Go to the Yanshee mobile application and click on the icon on the top right corner. * Check the serial number at the back of your robot and connect to the same robot. * Make sure you are connected to the same network as your system is connected to. * Once the connection has been set up, check to make sure you are able to execute the functionalities available on the application. ## Retrieve the Robot's IP Address Option 1- Open the menu on the top left corner of the application. Go to Setup -> Robot information. Note down the IP address for future reference. Option 2- You'll need: * HDMI cable * External monitor * USB mouse * USB keyboard Setup Steps: * Connect the HDMI cable to the robot's HDMI port. * Plug the mouse and keyboard into the USB ports on the right side of the robot's chest. * Ensure the robot's Raspberry Pi is on the same network as your computer. To find the IP address: * Open the terminal. * Run: ```bash ifconfig ``` The IP address will be listed under the `inet` field. ## Steps Set the OM1 API key and robot IP address in the config file (`config/ubtech_yanshee.json5`). ```bash "api_key": "om1_live_..." "robot_ip": "" ``` Then, run the agent using `make run CONFIG=ubtech_yanshee`. You will now be able to make the robot take actions like walk forward, walk backward, do a push up, give you a hug, turn right, turn left, bow, crouch and more via voice commands. You can also have a conversation with it and ask about its surroundings. # Brainpack Introduction Introduction to the BrainPack ### Overview From research to real-world autonomy, the **OM1 BrainPack** is a plug-and-play module that brings full autonomy to your robots. The BrainPack is designed to be mounted directly onto a robot to bring together mapping, object recognition, remote control, and self-charging capabilities — giving humanoids and quadrupeds what they need to navigate, remember, and act with purpose. > The BrainPack makes your robot smarter — a system that **learns, moves, and builds with you.** ![](/files/1tZR1MRZedJ3bTJy8BeZ) ### Key Features | Feature | Description | | ------------------------- | ------------------------------------------------------------- | | **NVIDIA Thor** | Provides powerful GPU compute for AI inference and navigation | | **JetPack 7.0** | Latest NVIDIA software stack with optimized ML libraries | | **ROS2 Support** | Compatible with current and future versions of ROS2 | | **Robot Agnostic** | Control different robot form factors (quadrupeds, humanoids) | | **Connectivity** | Ethernet, USB, and multiple power options | | **Open Reference Design** | Build your own with published specifications | ### Hardware Specifications Each BrainPack comes with: * NVIDIA Thor compute module * Integrated speakers for TTS output * Multiple connectivity options (Ethernet, USB) * Flexible power input options * Mounting hardware for Unitree robots ### Supported Robots | Robot | Support Level | | ----------- | ----------------- | | Unitree Go2 | ✅ Fully Supported | | Unitree G1 | ✅ Fully Supported | | LimX Tron | ✅ Fully Supported | ### Next Steps The BrainPack is open-source and you can refer to the guidelines to build your own [here](https://github.com/OpenMind/brainpack). # Overview Full autonomy architecture and introduction This section describes the full autonomy architecture and deployment model for OM1 with BrainPack. OM1 is a modular robotics intelligence platform that connects perception, language, and motion into a single runtime. In full autonomy mode, the robot operates independently — navigating its environment, responding to speech, streaming video, and making decisions — without requiring constant human input. All services run as containerized processes on the BrainPack, communicating over well-defined interfaces. #### Platform Support | Platform | Support Level | Notes | | ------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------ | | **NVIDIA AGX Orin** | Limited | Suitable for deployments that do not require heavy ML inference workloads | | **NVIDIA Thor** | Full | Recommended for ML-heavy autonomy workloads. Leverages GPU and DLA across navigation, vision, and audio processing | #### Robot Support * Unitree Go2 * Unitree G1 * LimX Tron *** ### Architecture Overview The full autonomy stack is built from modular, containerized services that communicate through well-defined interfaces. Each service has a single responsibility and can be updated, restarted, or replaced independently without affecting the rest of the system. ![](/files/ClW0asLHbUko31SHQwFX) At a high level, sensor data flows from hardware into the ROS2 SDK, which publishes it as structured topics. OM1 consumes those topics alongside user input, runs them through the LLM, and emits action commands back to the robot. The video processor handles media as a parallel pipeline, and the avatar renders robot state on the display throughout. *** ### Open Source Components #### OM1 (`om1`) OM1 is the central intelligence of the system. It acts as the orchestration layer between the robot's hardware, its sensors, and the language model — translating perception and user intent into physical action. At runtime, OM1 maintains a continuous loop: it listens for speech via ASR, passes the transcription (along with relevant context and system state) to the configured LLM, receives a response, and dispatches the appropriate action — whether that is speaking a reply, issuing a movement command, or triggering a downstream service. This loop runs in real time, keeping the robot responsive to its environment and the people around it. | Feature | Description | | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Basic Movement** | Translates high-level motion intents (move forward, turn left, stop) into low-level velocity commands sent to the robot's motor controller. Movement is coordinated with the navigation stack when the ROS2 SDK is available, or executed directly when operating in open-loop mode. | | **ASR** | Streams audio from the onboard microphone through a speech recognition pipeline that produces timestamped text transcriptions. These are fed into the LLM context as user input, enabling continuous voice-driven interaction. | | **TTS** | Takes text output from the LLM and synthesizes it into audio, which is played through the robot's speaker. The synthesis pipeline is tuned for low latency so responses feel conversational rather than delayed. | | **LLM Integration** | Manages the prompt lifecycle: assembles context (system prompt, conversation history, sensor summaries, task state), sends it to the configured language model, and routes the structured response back to the appropriate output channel (speech, movement, logging). Multiple LLM providers and models are supported through a plugin interface. | #### OM1 Avatar (`om1-avatar`) The OM1 Avatar is the frontend interface layer displayed on the BrainPack screen. It gives the robot a visual identity and surfaces system state to anyone nearby, making the robot's inner workings legible without requiring a separate device or dashboard. The avatar renders in real time and reacts to what the robot is doing — speaking, listening, navigating, or idle — so observers can read the robot's current state at a glance. It is built on React and communicates with the OM1 backend over a local websocket connection. | Feature | Description | | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | | **React-based UI** | Built as a responsive web application served locally on the BrainPack, accessible on the device's display. | | **Real-time Avatar Rendering** | Animates a visual avatar that reflects the robot's current activity and emotional state, updated continuously as the robot operates. | | **System Status Visualization** | Displays key system metrics and service health so operators can quickly spot issues without SSH-ing into the device. | | **User Interaction** | Accepts touch or button input directly on the BrainPack screen, enabling local control and conversation without an external controller. | *** ### Premium Components These components are accessible through the **Enterprise Plan**. #### OM1 ROS2 SDK (`om1-ros2-sdk`) The OM1 ROS2 SDK is the robotics middleware layer that connects OM1's high-level decisions to the robot's physical hardware. It handles everything from raw sensor ingestion to autonomous navigation, remote control, and simulation — all built on the ROS2 (Robot Operating System 2) framework, which provides standardized topic-based messaging between hardware and software components. Without the ROS2 SDK, OM1 can speak and reason but cannot navigate, map, or perceive its spatial environment. The SDK is what turns a conversational robot into an autonomous one. **Features** * **Sensor drivers** - Low-level drivers for onboard hardware sensors: * *Intel RealSense D435* — Provides an RGB color stream plus a calibrated depth map, giving the robot the ability to perceive the 3D structure of its surroundings. * *RPLidar* — Emits a 360° 2D laser scan of the environment, used as the primary input for mapping and localisation. Both sensors publish their data as typed ROS2 topics consumed by the orchestrator and navigation pipeline. * **Remote robot control** — Exposes a network interface for sending motion commands and operational instructions to the robot from a remote system. This enables tele-operation, remote supervision, and integration with external control software without physical access to the robot. * **Remote audio** — Enables bidirectional audio communication with the robot over the network. Operators can listen to what the robot hears and speak to it remotely, supporting use cases like remote supervision, guided operation, and off-site interaction. * **SLAM (Simultaneous Localization and Mapping)** — Builds a live map of the environment as the robot moves, while simultaneously estimating the robot's position within that map. The system fuses LiDAR scans and depth data to construct and update a spatial model of the environment in real time, enabling the robot to navigate areas it has never explicitly been programmed for. * **Full simulation support** — The complete ROS2 SDK stack — sensors, SLAM, navigation, and control — can run inside a simulator (such as Gazebo) without physical hardware. This enables developers to test navigation algorithms, tune parameters, and validate new features in a reproducible environment before deploying to a physical robot. * **Multi-Robot Support** - Compatible with Unitree Go2, Unitree G1, and LimX Tron robots. * **Auto Charging** - When the robot's battery falls below a threshold, the system initiates an autonomous return-to-dock sequence: 1. The Nav2 stack navigates the robot to the general vicinity of the charging station using the stored map 2. The robot switches to precision docking mode and activates its onboard cameras to detect AprilTag markers mounted on or near the dock 3. Visual servoing aligns the robot incrementally by tracking the AprilTag's pose in camera space 4. The robot approaches and physically docks, aligning its charging contacts with the pad's contact points > **Note:** Currently supported on **Go2 only**. * **Navigation & Localisation** - Integration with Nav2 for autonomous navigation. We have a custom localisation pipeline that the robot uses to determine its position and plan paths through its environment.Process incoming LaserScan message to determine feasible paths.Publish feasible paths and visualization markers. | Component | Description | | ----------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Visual Place Recognition (VPR)** | Uses camera images to estimate which general area of the map the robot is in | | **Correlative Scan Matching (CSM)** | Identifies distinctive geometric landmarks in sensor data and aligns them against a reference map | | **Nav2 AMCL** | A probabilistic particle filter that converges on the most likely position as new sensor data arrives. Robust in dynamic environments where the map may have changed. | > Refer to [Hybrid Localisation](/full-autonomy-guidelines/localization) for an in-depth explanation of the system. * **Obstacle Avoidance** - The obstacle avoidance system generates candidate paths by projecting straight-line segments from the robot's origin across a configurable range of headings and distances. Each candidate path is evaluated by fusing data from multiple sensor sources — RPLidar 360° laser scans, Intel RealSense depth images, and hazard point clouds, to determine which paths are clear of obstacles. Path segment geometry is precomputed once to eliminate redundant calculations at runtime. Feasible paths are then published as ROS2 topics alongside RViz visualization markers, giving operators real-time visibility into the robot's path selection decisions. **Components** The `om1-ros2-sdk` is composed of four internal services: | Service | Role | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `om1_sensor` | Manages low-level sensor drivers and continuously publishes raw sensor data (depth, RGB, LiDAR scans) to ROS2 topics | | `orchestrator` | Consumes sensor topics to run SLAM and navigation; manages map storage and path planning; exposes a REST API for external control and status queries | | `watchdog` | Monitors the health of sensor topics and the `om1_sensor` process; automatically restarts `om1_sensor` if data stops arriving or quality degrades | | `zenoh_bridge` | Acts as a protocol bridge between the OM1 core runtime and the ROS2 ecosystem, translating between Zenoh pub/sub messages and ROS2 topics in both directions | *** #### OM1 Video Processor (`om1-video-processor`) The OM1 Video Processor is a dedicated media processing pipeline that runs entirely on the robot's edge device. It handles real-time face anonymisation, audio cleanup, and AV streaming — all without sending raw video or audio off-device. CUDA acceleration via NVIDIA TensorRT ensures each stage meets real-time latency requirements even on embedded hardware. | Feature | Description | | ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Face Detection** | Each incoming video frame is scanned by the SCRFD model, a lightweight face detection architecture optimised with TensorRT for real-time inference at full frame rate. | | **Face Blurring** | Expands detected bounding boxes, applies feathered masks, and uses strong Gaussian blur to make identity unrecoverable. Raw frames never leave the device. | | **Audio Noise Cancellation** | Filters microphone input in real time to suppress ambient noise (fan hum, footsteps, crowd noise) before ASR processing or remote streaming. | | **Audio Streaming** | Captures processed audio and streams it to an RTSP server as a continuous audio track for external consumers. | | **Video Streaming** | Captures processed, blurred video output and streams it to the RTSP server as a synchronized video track. | > **What is RTSP?** RTSP (Real Time Streaming Protocol) is a network control protocol for managing multimedia streaming sessions. Rather than transporting media itself, it manages the session — establishing the stream, synchronising audio and video tracks, and providing controls (play, pause, seek) to the consumer. The video processor uses RTSP so that any compatible media player or monitoring system can consume the robot's live feed without custom integration work. *** #### Person Following (`person-following`) The Person Following service enables the robot to autonomously detect, track, and follow a designated person through its environment. It combines continuous visual detection with spatial reasoning and scene understanding, allowing the robot to stay close to a person while navigating around obstacles in its path. This service is designed for use cases such as personal assistance, guided tours, and supervised autonomy — anywhere the robot needs to stay with a person rather than navigate to a fixed destination. | Feature | Description | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Body Detection** | Detects and tracks a person's full body using body pose and silhouette (not just face). Robust to turned faces, crouching, or partial occlusion. Maintains persistent identity for the target person even as others move through the scene. | | **Distance Detection** | Uses depth data from the Intel RealSense D435 to estimate 3D distance between the robot and tracked person. Motion controller adjusts speed and heading to maintain comfortable following gap. | | **Video Description** | Generates real-time natural language descriptions of what the robot sees using a vision-language model. Enables verbal narration of observations and provides foundation for complex reasoning behaviours. | *** ### OM1-OTA All services in the full autonomy stack are delivered as OTA-managed containers. The `ota_agent` and `ota_updater` services handle the full lifecycle of every container on the device, ensuring the robot stays up to date without requiring manual intervention. #### `ota_agent` The main OTA (over-the-air) lifecycle manager. It is responsible for the complete update cycle across all containerized services on the robot: * Connects to the container registry and pulls new images when updates are available * Starts, stops, and restarts service containers in the correct dependency order * Applies version upgrades to application images without requiring a full system restart * Reports service health and update status back to the management plane #### `ota_updater` A self-update companion for `ota_agent`. Because `ota_agent` manages all other containers, it cannot update itself — `ota_updater` exists specifically to handle that case: * Monitors for new versions of `ota_agent` and applies updates when available * Ensures that the update agent itself never becomes outdated or incompatible with the services it manages * Acts as the last line of the update chain, keeping the entire update infrastructure current *** Your robot is now ready to accompany you, assist with tasks, explore new environments, and learn alongside you. To access the premium features through API endpoints refer the documentation [here](/full-autonomy-guidelines/api_endpoints). # Hybrid Localisation Introducing Hybrid Localization ### What is Robot Localization? Every mobile robot must answer one fundamental question before it can do anything useful: **"Where am I?"** The process of determining a robot's position and orientation within a map of its environment is called **localization**. Without it, the robot cannot plan a path to a destination, avoid obstacles along the way, or know when it has arrived. Localization is the invisible foundation that makes autonomous navigation possible. Think of it like waking up in a hotel room in a city you have visited before. You know what the city looks like (you have a map). You can see the room around you (your sensors are working). But you do not know which hotel you are in, which floor, or which direction you are facing. You need to figure out your position on the map before you can navigate to the breakfast buffet. ### Why is Localization Hard? Localization sounds straightforward, but in practice it involves several genuinely difficult problems: | Challenge | Description | | ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Global localization (cold start)** | The robot is powered on at an unknown location in a known map. It must determine its position from scratch, without any prior information. This is the hardest variant. | | **Continuous tracking** | Once the robot knows where it is, it must keep track of its position as it moves. Sensors drift over time — wheels slip, gyroscopes accumulate error, laser scans get noisy. | | **Environmental change** | The real world changes. Furniture gets moved, doors open and close, boxes appear in hallways. The map may not perfectly match what sensors see today. | | **Symmetric environments** | Long corridors, identical meeting rooms, and repeating architectural features look the same from multiple positions. The laser scanner sees identical geometry at different locations. | | **Kidnapped robot problem** | The robot is physically picked up and placed somewhere else (or its localization catastrophically fails). It must detect that it is lost and re-localize automatically. | > No single localization technique handles all of these well. That observation is what led to this hybrid system. ### What This System Does The hybrid localization system combines three complementary technologies into one architecture: | Technology | Description | | ----------------------------------- | --------------------------------------------------------------------------------------------- | | **Visual Place Recognition (VPR)** | Uses camera images to estimate which general area of the map the robot is in | | **Correlative Scan Matching (CSM)** | Uses LiDAR to brute-force search for the exact position, guided by the VPR hint | | **Nav2 AMCL** | The standard ROS 2 particle filter for smooth, continuous position tracking during navigation | Together, these three systems handle every failure mode listed above: * **VPR** breaks symmetry * **CSM** provides instant global localization * **AMCL** provides robust continuous tracking A health monitor watches AMCL's output and triggers automatic recovery when needed. **No human intervention is required at any point.** ### Architecture The system is implemented as three ROS 2 nodes working together, orchestrated by a state machine in the Hybrid Localization Manager. This node does not publish any TF transforms — Nav2 AMCL is the sole authority for the `map→odom` transform. The hybrid node acts as a quality gate: it finds initial poses, validates them, feeds them to AMCL, and monitors AMCL's output for failures. #### The Three Nodes | Node | Phase | Description | | ------------------------------- | ---------- | ----------------------------------------------------------------------------------------------------------- | | **VPR Capture Node** | Mapping | Captures camera snapshots during SLAM, computes visual embeddings, stores `(embedding, pose)` pairs to disk | | **VPR Query Node** | Navigation | Matches current camera view against the database at 2 Hz, publishes rough location hints | | **Hybrid Localization Manager** | Navigation | Main orchestrator containing CSM, candidate ranking, AMCL seeding, and health monitoring | **Node 1: VPR Capture Node (Mapping Phase)** This node runs while the robot is building a map using SLAM. As the robot explores its environment, the capture node periodically takes camera snapshots, computes a compact visual fingerprint (called an embedding) for each image, and stores it alongside the robot's current map-frame position. The result is a database of `(embedding, pose)` pairs saved to disk alongside the map. **Capture triggers:** * Movement > **10 cm** from last capture * Rotation > **\~11°** from last capture This ensures good spatial coverage without wasting storage on redundant images. **Node 2: VPR Query Node (Navigation Phase)** This node runs during autonomous navigation at **2 Hz** (every 0.5 seconds). It: 1. Takes the latest camera image 2. Computes its embedding 3. Compares against every saved embedding in the database 4. If best match exceeds similarity threshold (**0.65**), publishes the corresponding pose as a location hint The hint has high uncertainty (**2 meter standard deviation**) because VPR provides a neighborhood estimate, not a precise position. The query node also publishes a "ready" signal after its first query attempt. **Node 3: Hybrid Localization Manager (Navigation Phase)** This is the main orchestrator. It subscribes to: | Topic | Purpose | | ------------------- | ------------------------------- | | `/scan` | Laser scanner data for CSM | | `/map` | Occupancy grid from map\_server | | `/visual_pose_hint` | VPR location hints | | `/amcl_pose` | AMCL pose output for monitoring | It contains the correlative scan matcher for global localization, the candidate ranking logic, the AMCL seeding mechanism, and the health monitoring system. Everything is coordinated through a **five-state state machine**. #### The Five States | State | Description | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `WAITING_FOR_MAP` | Startup state. Waits for the occupancy grid map from `map_server`. On receipt, preprocesses the map (crop, gradient field, free space extraction) and transitions to `GLOBAL_LOCALIZING`. | | `GLOBAL_LOCALIZING` | Collects VPR camera hints for up to 5 seconds. Computes robust consensus from multiple hints. Runs VPR-guided correlative scan matching. Ranks up to 5 candidate poses. Selects the best and transitions to `SEEDING_AMCL`. | | `SEEDING_AMCL` | Publishes the discovered pose to `/initialpose_validated` (which AMCL subscribes to). Re-publishes every 2 seconds until AMCL responds. Transitions to `MONITORING` when AMCL publishes its first pose. Times out after 30 seconds and falls back to `RECOVERING`. | | `MONITORING` | Steady-state. Watches every AMCL pose update through three checks: pose jump detection, scan match cross-validation, and covariance monitoring. Triggers `RECOVERING` if sustained failures are detected. | | `RECOVERING` | Triggered when monitoring detects persistent failure. Clears old VPR hints, collects fresh camera data, and re-runs the full global localization pipeline. Functionally identical to `GLOBAL_LOCALIZING` but entered from a failed state. | #### The /initialpose interception pattern A critical architectural detail: Nav2 AMCL does not subscribe to the standard /initialpose topic in this system. Instead, it subscribes to /initialpose\_validated. The hybrid node intercepts all /initialpose messages (including those from RViz’s "2D Pose Estimate" button), evaluates them via scan matching, and only forwards them to /initialpose\_validated if they pass quality validation (score above 90%). This means every pose that reaches AMCL has been validated. A careless RViz click or a malfunctioning external node cannot inject a bad pose and destroy a perfectly good localization. During the early states (before monitoring begins), external poses are forwarded without validation since there is no existing good localization to protect. ### Complete Startup Sequence To tie everything together, here is the exact sequence of events from power-on to autonomous navigation: | Time | Event | | ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **t = 0s** | `map_server` publishes the occupancy grid. The hybrid node receives it, crops to the occupied region plus margin, builds the gradient field, extracts free space cells. Transitions to `GLOBAL_LOCALIZING`. Starts the VPR warmup delay (5 seconds). | | **t = 0.5s** | VPR Query Node matches the current camera view against the database. Publishes the first `/visual_pose_hint` with the saved pose of the best matching image. Publishes `/vpr_query_ready = true`. | | **t = 1–4s** | VPR continues publishing hints at 2 Hz. The hybrid node accumulates these into a list. Once 3 or more hints are collected and the ready signal has been received, it computes the VPR consensus (median, outlier removal, inlier average). | | **t = 2–4s** | The hybrid node runs `perform_global_localization`: generates \~160,000 VPR-biased candidate poses, evaluates all of them against the gradient map via Numba, selects the top 5 spatially distinct candidates, refines each via iterative hill-climbing, ranks them with VPR proximity bonus. | | **t = 3–5s** | Best candidate published to `/initialpose_validated`. The hybrid node transitions to `SEEDING_AMCL`. Starts re-publish timer (every 2 seconds). | | **t = 3–6s** | Nav2 AMCL receives `/initialpose_validated`. Initializes particle cloud around the given pose. Publishes first `/amcl_pose`. The hybrid node detects this, cancels re-publish timer, transitions to `MONITORING`. Starts 5-second settle timer. | | **t = 8–11s** | Settle time elapses. AMCL particles have converged. The hybrid node begins active health monitoring (jump detection, scan match validation, covariance checks). The robot is now localized and ready for autonomous navigation. | > **Total time from power-on to navigation-ready: typically 8–11 seconds**, fully automatic, no human intervention. Compared to Nav2 AMCL alone (which requires a manual 2D Pose Estimate click and 10–30 seconds of wandering), this is a significant operational improvement. ### Comparison with Previous Approaches The following table summarizes how the hybrid system addresses each localization challenge compared to using AMCL or the correlative scan matcher alone: | Challenge | AMCL Alone | CSM Alone | Hybrid System | | ---------------------------------- | ----------------------------------------------------------- | --------------------------------------------- | ---------------------------------------------------------------------- | | **Global localization on startup** | Manual RViz click or slow global service (10–30s wandering) | Instant brute-force search (2–3s) | VPR-guided CSM search in under 5 seconds, fully automatic | | **Continuous tracking** | Excellent: smooth, low-CPU particle filter | Poor: expensive per-scan, prone to jumps | AMCL handles tracking; CSM only runs on-demand | | **Environmental changes** | Moderate with tuning (`z_rand`, beam skip) | Poor: gradient map is static | AMCL handles moderate changes; recovery handles major changes | | **Symmetric environments** | Vulnerable: particles converge to wrong location | Vulnerable: identical geometry scores equally | VPR breaks symmetry; jump validation catches errors | | **Kidnapped robot** | Poor: slow random particle injection | Good: can re-localize instantly | Health monitor detects failure and triggers VPR-guided re-localization | | **Bad external pose (RViz)** | Accepted blindly: can destroy good localization | N/A | Scan match validation gate: rejected if quality < 90% | ### Hardware Context This system was designed and tested on the **Unitree Go2** quadruped robot with the following sensor configuration: | Sensor | Type | Purpose | | ------------------- | ---------------------- | ----------------------------------------------------------------------------------------------------------- | | **Primary LiDAR** | RPLidar S2L (2D, 360°) | Mounted on top. Used for mapping, localization (CSM + AMCL), and obstacle detection in Nav2 costmap. | | **Secondary LiDAR** | 4D LiDAR | Mounted on nose. Used only for ground obstacle detection via STVL costmap layer. Not used for localization. | | **Camera** | Any ROS 2 compatible | Used for Visual Place Recognition. Any camera publishing `sensor_msgs/Image` is compatible. | #### Platform-Specific Challenges The quadruped platform presents unique localization challenges that make the hybrid architecture particularly valuable: | Challenge | Impact | How Hybrid Helps | | --------------------------- | --------------------------------------------- | --------------------------------------- | | Body pitch/roll during gait | Laser scan distortion | Multi-modal validation (LiDAR + camera) | | Leg odometry noise | Different characteristics than wheel odometry | Covariance monitoring detects drift | | Sit/stand transitions | Odometry jumps | Pose jump detection triggers recovery | # NVIDIA Thor Getting Started with OM1 on Nvidia Thor Dev Kit ### Basics Make sure you have the following before installation: * Device: 945-14070-0080-000 | Jetson AGX Thor Developer Kit * USB Memory Stick Install the latest Dev Kit base image \[following this guide] (). Helpful hint: If you are on a mac, you can quickly wipe the USB drive for use with Balena Etcher with these commands: ```bash diskutil list diskutil unmountDisk /dev/disk4 # customize this, obviously diskutil eraseDisk FAT32 SANDISK /dev/disk4 ``` We typically use the `Monitor-attached` flow. We have successfully used `jetsoninstaller-0.2.0-r38.2-2025-08-22-01-33-29-arm64.iso`. Sometimes the installation hangs right after `freeing initrd memory` at t = 4s, but just restart the Thor and try again; it should work eventually. You will be left with the Thor running Ubuntu 24.04.3 LTS. ### JetPack SDK Once you have installed Ubuntu, [install JetPack](https://docs.nvidia.com/jetson/agx-thor-devkit/user-guide/latest/setup_jetpack.html). ### System Basics ```bash sudo apt-get update # browser sudo apt install chromium-browser # github desktop # directly download from github. Use the most recent release for arm64, such as # https://github.com/shiftkey/desktop/releases/download/release-3.4.13-linux1/GitHubDesktop-linux-arm64-3.4.13-linux1.deb # install by double-clicking the .deb # sublime wget -qO - https://download.sublimetext.com/sublimehq-pub.gpg | sudo tee /etc/apt/keyrings/sublimehq-pub.asc > /dev/null echo -e 'Types: deb\nURIs: https://download.sublimetext.com/\nSuites: apt/stable/\nSigned-By: /etc/apt/keyrings/sublimehq-pub.asc' | sudo tee /etc/apt/sources.list.d/sublime-text.sources sudo apt-get update sudo apt-get install sublime-text # camera device info v4l2-ctl --listdevices # to test cameras sudo apt install guvcview # FYI the cheese webcam app is broken on Thor ``` You can run the usual Ubuntu software updater without hosing the install. ### System Info * **GPU**: You can get GPU stats and other basic information via the built in command `nvidia-smi`. The command should show CUDA version 13. * **tegastats**: The `tegrastats` command (via the top right corner Nvidia dropdown) gives you information about system temperatures, frequencies, and power consumption in a terminal. * **Jetson Power GUI** gives you a visual overview of CPU, GPU, Thermal, Power, and fans. ### OM1 Then, install OM1 on the Nvidia Thor from its command line, following the [standard instructions](/developing/1_get-started). ### RealSense Depth Camera * Install `cmake`: ```bash sudo apt install cmake ``` Download the `development` branch of [librealsense](https://github.com/IntelRealSense/librealsense/tree/development). Follow [Building from Source using Native Backend](https://github.com/IntelRealSense/librealsense/blob/development/doc/installation_jetson.md#building-from-source-using-native-backend). * Patch `./scripts/patch-realsense-ubuntu-L4T.sh` Add `38.2.2` to the list of supported versions: ```bash Line 100 "38.2") -> "38.2" | "38.2.2") ``` * Make sure your local CUDA is working; test with the [Nvidia examples](https://github.com/NVIDIA/cuda-samples): ```bash python3 run_tests.py --output ./test --dir ./build/Samples --config test_args.json ``` * Add needed env vars to your `.bashrc`: ```bash echo "export PATH=/usr/local/cuda/bin:$PATH" >> ~/.bashrc echo "export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH" >> ~/.bashrc echo "export CUDACXX=/usr/local/cuda-13.0/bin/nvcc" >> ~/.bashrc source ~/.bashrc ``` * Install OpenGL Utility Library (GLU) ```bash sudo apt-get install libglu1-mesa-dev ``` * Change the `cmake` command relative to "building-from-source-using-native-backend" Specifically, `-DCMAKE_CUDA_ARCHITECTURES=native` seems to be needed. The resulting command is: ```bash cmake .. -DBUILD_EXAMPLES=true -DCMAKE_BUILD_TYPE=release -DFORCE_RSUSB_BACKEND=false -DCMAKE_CUDA_ARCHITECTURES=native -DBUILD_WITH_CUDA=true && make -j$(($(nproc)-1)) && sudo make install ``` Finally, plug in your RealSense. When you now run `realsense-viewer`, it *should* just work. **Warning** The RealSense camera requires the full bandwidth of an entire USB 3.2 hub. This means that on the Thor, depending on where you plug in the RealSense, the adjacent USB port will stop working (practically speaking). So e.g. you cannot use one USB 3.2 A port for your mouse, and the other for the DepthSense. We suggest adding a hub to one of the USB Type-C ports, and then using that hub for your mouse, keyboard, microphones, speakers, and other cameras, etc, and then the RealSense can use/block both USB 3.2 Type A ports. ### Robot Deployments For robot deployments, we recommend modules such as the [Auvidea Jetson T5000 module](https://auvidea.eu/press-release-nvidia-jetson-thor-t5000-launch/) since it directly accepts 24V to 48V, unlike the dev kit's more restricted 9–28V DC Micro-fit power connector. ### Done You are now running OM1 on your Nvidia Thor. Explore its capabilities and refer to the [OM1 GitHub repository](https://github.com/openmind/OM1) for advanced settings and next steps. # Premium Features ### Open Source Access OM1 and Simulation environment are fully open source and publicly accessible. You can explore, use, and contribute to the codebase freely. ### Premium Features All other features beyond the core framework are available exclusively through our **Enterprise Plan**. This includes: * Advanced robotics integrations * Extended simulation capabilities * SLAM map generation and Navigation * Face detection and Anonymization * Person following * Obstacle avoidance * Remote control for the robots * Auto charging * Custom feature development ### Simulation Limitations The open source simulation environment does not include the following features: * **Navigation** - Autonomous path planning and movement * **SLAM** - Simultaneous Localization and Mapping * **Obstacle Avoidance** These capabilities are available exclusively with the Builder Plan and above. ### Students & Researchers We offer special access for students and researchers. To apply: 1. Fill out our application form *(coming soon)* 2. Include a Statement of Purpose (SOP) describing your goals and intended use 3. Our team will review your application and get back to you Based on your use case and feasibility, we may offer: * **Free Trial** - Limited-time access to premium features (7 days) * **Enterprise Plan** ### Getting Access To unlock premium features, contact our team to learn more about the Enterprise Plan: * **Email**: [Support Team](mailto:support@openmind.com?subject=Inquiry\&body=Hello>) * **Website**: [OpenMind Portal](https://portal.openmind.com) Our team will work with you to understand your needs and provide access to the full suite of premium capabilities. # API Endpoints to Interact with Premium Features REST API documentation for OM1 ROS2 SDK ### Overview The OM1 ROS2 SDK provides two REST APIs for remote control and monitoring of the robot: | API | Port | Description | | -------------------- | ------ | -------------------------------------------------------------------------------- | | **Orchestrator API** | `5000` | System orchestration and high-level control (SLAM, Nav2, charging, patrol, maps) | | **Nav2 API** | `5001` | Direct navigation control and real-time monitoring (pose, goals, localization) | > **Note:** These APIs are part of the **Premium Features** available through the Enterprise Plan. See [Premium Features](/full-autonomy-guidelines/premium_features) for details. *** ### Orchestrator API (Port 5000) The Orchestrator API manages the robot's operational state, including SLAM, navigation, charging, patrol, and map management. All endpoints return JSON responses. #### Quick Reference | Method | Endpoint | Description | | ------ | -------------------------- | ----------------------------------- | | `GET` | `/status` | Get status of all running processes | | `POST` | `/start/base_control` | Start base control process | | `POST` | `/stop/base_control` | Stop base control process | | `POST` | `/start/slam` | Start SLAM mapping | | `POST` | `/stop/slam` | Stop SLAM mapping | | `POST` | `/start/nav2` | Start Nav2 navigation stack | | `POST` | `/stop/nav2` | Stop Nav2 navigation stack | | `POST` | `/charging/dock` | Start autonomous docking (Go2 only) | | `POST` | `/charging/stop` | Stop docking process | | `GET` | `/charging/status` | Get charging status | | `POST` | `/patrol/start` | Start patrol (Go2 only) | | `POST` | `/patrol/stop` | Stop patrol | | `POST` | `/patrol/pause` | Pause patrol | | `POST` | `/patrol/resume` | Resume patrol | | `POST` | `/maps/save` | Save current SLAM map | | `GET` | `/maps/list` | List all saved maps | | `POST` | `/maps/delete` | Delete a saved map | | `POST` | `/maps/locations/add` | Add a location to a map | | `GET` | `/maps/locations/list` | List all saved locations | | `POST` | `/maps/locations/add/slam` | Save current position as location | *** #### System Status **GET /status** Returns the operational status of all robot processes. Use this to check which services are currently active before starting or stopping processes. **Response:** ```json { "status": "success", "message": "{\"base_control_running\": true, \"slam_running\": false, \"nav2_running\": true, \"patrol_running\": false}" } ``` | Field | Type | Description | | ---------------------- | ------- | --------------------------------------- | | `base_control_running` | boolean | Whether base motor control is active | | `slam_running` | boolean | Whether SLAM mapping is in progress | | `nav2_running` | boolean | Whether Nav2 navigation stack is active | | `patrol_running` | boolean | Whether autonomous patrol is running | *** #### Base Control Base control manages the low-level motor commands that allow the robot to move. It must be running for any movement operations. **POST /start/base\_control** Start the base control process. This initializes the robot's motor controller and enables movement commands. **Request body (optional):** ```json { "launch_file": "base_control_launch.py" } ``` | Parameter | Type | Required | Description | | ------------- | ------ | -------- | ------------------------------------------------------ | | `launch_file` | string | No | Custom launch file (default: `base_control_launch.py`) | **Response:** ```json { "status": "success", "message": "Base control started" } ``` **Error conditions:** | Status | Condition | | ------ | ------------------------------- | | `400` | SLAM or Nav2 is already running | **POST /stop/base\_control** Stop the base control process. This disables all motor commands. **Response:** ```json { "status": "success", "message": "Base control stopped" } ``` *** #### SLAM SLAM (Simultaneous Localization and Mapping) allows the robot to build a map of its environment while tracking its position within that map. **POST /start/slam** Start the SLAM process for mapping. The robot will begin building an occupancy grid map as it moves through the environment. **Request body (optional):** ```json { "launch_file": "slam_launch.py", "map_yaml": "/path/to/existing/map.yaml" } ``` | Parameter | Type | Required | Description | | ------------- | ------ | -------- | --------------------------------------------- | | `launch_file` | string | No | Custom SLAM launch file | | `map_yaml` | string | No | Path to existing map to continue mapping from | **Response:** ```json { "status": "success", "message": "SLAM started" } ``` **Error conditions:** | Status | Condition | | ------ | ----------------------------------------------------- | | `400` | Nav2 is running (must stop Nav2 before starting SLAM) | **POST /stop/slam** Stop the SLAM process. Remember to save the map using `/maps/save` before stopping if you want to keep the map. **Response:** ```json { "status": "success", "message": "SLAM stopped" } ``` *** #### Navigation (Nav2) Nav2 is the ROS2 navigation stack that enables autonomous navigation using a pre-built map. It handles path planning, obstacle avoidance, and localization. **POST /start/nav2** Start the Nav2 navigation stack with a saved map. The robot will localize itself within the map and be ready to receive navigation goals. **Request body:** ```json { "map_name": "office_floor1", "launch_file": "nav2_launch.py" } ``` | Parameter | Type | Required | Description | | ------------- | ------ | -------- | ------------------------------------------- | | `map_name` | string | **Yes** | Name of the saved map to use for navigation | | `launch_file` | string | No | Custom Nav2 launch file | **Response:** ```json { "status": "success", "message": "Nav2 started" } ``` **Error conditions:** | Status | Condition | | ------ | ----------------------------------------------------- | | `400` | SLAM is running (must stop SLAM before starting Nav2) | | `400` | `map_name` is not provided | **POST /stop/nav2** Stop the Nav2 navigation stack. This will cancel any active navigation goals. **Response:** ```json { "status": "success", "message": "Nav2 stopped" } ``` *** #### Charging (Go2 Only) The charging endpoints control the autonomous docking and charging process. The robot uses AprilTag visual servoing to precisely align with the charging dock. > **Note:** These endpoints are only available on the Unitree Go2 robot. **POST /charging/dock** Start the autonomous docking sequence. The robot will navigate to the charging station and align itself with the dock contacts. **Prerequisites:** * Nav2 must be running * Robot must not already be charging **Request body (optional):** ```json { "launch_file": "go2_charge_launch.py" } ``` | Parameter | Type | Required | Description | | ------------- | ------ | -------- | --------------------------- | | `launch_file` | string | No | Custom charging launch file | **Response:** ```json { "status": "success", "message": "Charging dock process started" } ``` **Error conditions:** | Status | Condition | | ------ | ------------------------------------------- | | `400` | Robot type is not Go2 | | `400` | Nav2 is not running | | `400` | Already charging or dock process is running | **POST /charging/stop** Stop the charging dock process. Use this to abort docking or undock the robot. **Response:** ```json { "status": "success", "message": "Charging dock process stopped" } ``` **GET /charging/status** Get the current charging and docking status. Use this to monitor battery level and charging state. **Response:** ```json { "is_charging": false, "battery_percentage": 85, "dock_process_running": false } ``` | Field | Type | Description | | ---------------------- | ------- | ------------------------------------------- | | `is_charging` | boolean | Whether the robot is currently charging | | `battery_percentage` | integer | Current battery level (0-100) | | `dock_process_running` | boolean | Whether the docking sequence is in progress | *** #### Patrol (Go2 Only) The patrol endpoints control autonomous patrol behavior. The robot will navigate between predefined waypoints, monitoring for activity. If autocharging is enabled, and battery drops below a particular level, the robot will go to the docking station, charge and continue patrol. > **Note:** These endpoints are only available on the Unitree Go2 robot. **POST /patrol/start** Start the autonomous patrol process. The robot will begin navigating between saved patrol waypoints. **Prerequisites:** * Nav2 must be running * Patrol waypoints must be configured **Request body (optional):** ```json { "launch_file": "go2_patrol_launch.py" } ``` | Parameter | Type | Required | Description | | ------------- | ------ | -------- | ------------------ | | `launch_file` | string | No | Patrol launch file | **Response:** ```json { "status": "success", "message": "Patrol started" } ``` **Error conditions:** | Status | Condition | | ------ | ------------------------- | | `400` | Robot type is not Go2 | | `400` | Nav2 is not running | | `400` | Patrol is already running | **POST /patrol/stop** Stop the patrol process. The robot will stop at its current position. **Response:** ```json { "status": "success", "message": "Patrol stopped" } ``` **POST /patrol/pause** Pause the currently running patrol. The robot will hold its position until resumed. **Response:** ```json { "status": "success", "message": "Patrol pause command sent" } ``` **POST /patrol/resume** Resume the paused patrol from where it left off. **Response:** ```json { "status": "success", "message": "Patrol resume command sent" } ``` *** #### Map Management Map management endpoints allow you to save, list, and delete SLAM-generated maps. Maps are stored on the robot and can be loaded later for navigation. **POST /maps/save** Save the current SLAM map to a file. Call this before stopping SLAM to preserve the map. **Request body:** ```json { "map_name": "office_floor1", "map_directory": "/custom/path/to/maps" } ``` | Parameter | Type | Required | Description | | --------------- | ------ | -------- | ----------------------------------------------------- | | `map_name` | string | **Yes** | Name for the saved map (no `/`, `\`, `..`, or spaces) | | `map_directory` | string | No | Custom directory path for map storage | **Response:** ```json { "status": "success", "message": "Map saved successfully", "map_path": "/path/to/maps/office_floor1" } ``` **Error conditions:** | Status | Condition | | ------ | ---------------------------------------------------- | | `400` | SLAM is not running | | `400` | `map_name` is missing or contains invalid characters | **GET /maps/list** List all saved maps with their metadata. **Response:** ```json { "maps": [ { "name": "office_floor1", "path": "/path/to/maps/office_floor1", "created": "2026-04-21T10:30:00" } ] } ``` | Field | Type | Description | | --------- | ------ | ---------------------------------- | | `name` | string | Map identifier | | `path` | string | Full path to map files | | `created` | string | ISO 8601 timestamp of map creation | **POST /maps/delete** Delete a saved map. **Request body:** ```json { "map_name": "office_floor1" } ``` | Parameter | Type | Required | Description | | ---------- | ------ | -------- | ------------------------- | | `map_name` | string | **Yes** | Name of the map to delete | **Response:** ```json { "status": "success", "message": "Map deleted successfully" } ``` **Error conditions:** | Status | Condition | | ------ | ------------------ | | `404` | Map does not exist | *** #### Location Management Location management endpoints allow you to save and manage named waypoints within maps. These locations can be used as navigation goals. **POST /maps/locations/add** Add a named location to a map's location list with explicit pose data. **Request body:** ```json { "map_name": "office_floor1", "location": { "name": "reception", "description": "Front reception desk", "timestamp": "2026-04-21T10:30:00", "pose": { "position": {"x": 1.0, "y": 2.0, "z": 0.0}, "orientation": {"x": 0.0, "y": 0.0, "z": 0.0, "w": 1.0} } } } ``` | Parameter | Type | Required | Description | | --------------------------- | ------ | -------- | -------------------------------------- | | `map_name` | string | **Yes** | Name of the map to add the location to | | `location.name` | string | **Yes** | Unique name for this location | | `location.description` | string | No | Human-readable description | | `location.pose.position` | object | **Yes** | x, y, z coordinates in map frame | | `location.pose.orientation` | object | **Yes** | Quaternion (x, y, z, w) orientation | **Response:** ```json { "status": "success", "message": "Location added successfully" } ``` **GET /maps/locations/list** List all saved locations across all maps. Returns locations grouped by map name. **Response:** ```json { "status": "success", "message": "{\"office_floor1\": [{\"name\": \"reception\", \"description\": \"Front desk\", ...}]}" } ``` **POST /maps/locations/add/slam** Save the robot's current position as a named location during SLAM. This is the easiest way to add waypoints — just drive the robot to a location and call this endpoint. **Request body:** ```json { "map_name": "office_floor1", "label": "conference_room", "description": "Main conference room entrance" } ``` | Parameter | Type | Required | Description | | ------------- | ------ | -------- | -------------------------------------- | | `map_name` | string | **Yes** | Name of the map to add the location to | | `label` | string | **Yes** | Unique name for this location | | `description` | string | No | Human-readable description | **Response:** ```json { "status": "success", "message": "Location 'conference_room' saved successfully", "location": { "name": "conference_room", "description": "Main conference room entrance", "timestamp": "2026-04-21T10:30:00", "pose": { "position": {"x": 3.5, "y": 1.2, "z": 0.0}, "orientation": {"x": 0.0, "y": 0.0, "z": 0.707, "w": 0.707} } } } ``` **Error conditions:** | Status | Condition | | ------ | ------------------------------------------- | | `400` | SLAM is not running | | `500` | Unable to get robot position from map frame | *** ### Nav2 API (Port 5001) The Nav2 API provides direct access to navigation control and real-time localization data. Use this API for sending navigation goals and monitoring the robot's position and navigation status. #### Quick Reference | Method | Endpoint | Description | | ------ | -------------------- | ------------------------------ | | `GET` | `/api/status` | Check API status | | `GET` | `/api/pose` | Get current robot pose | | `POST` | `/api/move_to_pose` | Send robot to a specific pose | | `GET` | `/api/amcl_variance` | Get localization uncertainty | | `GET` | `/api/nav2_status` | Get navigation goal status | | `GET` | `/api/map` | Get current occupancy grid map | *** #### GET /api/status Returns the API health status. Use this to verify the Nav2 API is running and accessible. **Response:** ```json { "status": "OK", "message": "Go2 API is running" } ``` *** #### GET /api/pose Returns the current robot pose in the map frame with covariance matrix. The covariance indicates localization uncertainty. **Response:** ```json { "position": {"x": 0.0, "y": 0.0, "z": 0.0}, "orientation": {"x": 0.0, "y": 0.0, "z": 0.0, "w": 1.0}, "covariance": [0.0, ...] } ``` | Field | Type | Description | | ------------- | ------ | -------------------------------------------- | | `position` | object | x, y, z coordinates in map frame (meters) | | `orientation` | object | Quaternion (x, y, z, w) representing heading | | `covariance` | array | 36-element covariance matrix (6x6 flattened) | *** #### POST /api/move\_to\_pose Send the robot to a specific pose in the map frame. This creates a navigation goal and returns immediately — use `/api/nav2_status` to monitor progress. **Request body:** ```json { "position": {"x": 1.0, "y": 2.0, "z": 0.0}, "orientation": {"x": 0.0, "y": 0.0, "z": 0.0, "w": 1.0} } ``` | Parameter | Type | Required | Description | | ------------- | ------ | -------- | --------------------------------------- | | `position` | object | **Yes** | Target x, y, z coordinates in map frame | | `orientation` | object | **Yes** | Target orientation as quaternion | **Response:** ```json { "status": "success", "message": "Moving to specified pose" } ``` *** #### GET /api/amcl\_variance Returns the AMCL localization uncertainty estimates. Lower values indicate more confident localization. **Response:** ```json { "x_uncertainty": 0.1, "y_uncertainty": 0.1, "yaw_uncertainty": 5.0 } ``` | Field | Type | Description | | ----------------- | ----- | ---------------------------------- | | `x_uncertainty` | float | Position uncertainty in x (meters) | | `y_uncertainty` | float | Position uncertainty in y (meters) | | `yaw_uncertainty` | float | Heading uncertainty (degrees) | *** #### GET /api/nav2\_status Returns the status of all active navigation goals. Use this to monitor navigation progress after calling `/api/move_to_pose`. **Response:** ```json { "nav2_status": [ { "goal_id": "abc123...", "status": "EXECUTING", "timestamp": {"sec": 1234567890, "nanosec": 123456789} } ] } ``` | Field | Type | Description | | ----------- | ------ | ----------------------------------------- | | `goal_id` | string | Unique identifier for the navigation goal | | `status` | string | Current status of the goal | | `timestamp` | object | Time when status was recorded | **Status Values:** | Status | Description | | ----------- | ------------------------------------------- | | `UNKNOWN` | Goal status is unknown | | `ACCEPTED` | Goal has been accepted but not yet started | | `EXECUTING` | Robot is actively navigating to goal | | `CANCELING` | Goal cancellation in progress | | `SUCCEEDED` | Robot reached the goal successfully | | `CANCELED` | Goal was canceled | | `ABORTED` | Navigation failed (obstacle, timeout, etc.) | *** #### GET /api/map Returns the current occupancy grid map. The map is represented as a 2D array where each cell indicates occupancy probability. **Response:** ```json { "map_metadata": { "map_load_time": 1234567890.123456789, "resolution": 0.05, "width": 384, "height": 384, "origin": { "position": {"x": -10.0, "y": -10.0, "z": 0.0}, "orientation": {"x": 0.0, "y": 0.0, "z": 0.0, "w": 1.0} } }, "data": [0, 0, 0, ...] } ``` | Field | Type | Description | | ------------------------- | ------- | -------------------------------------------------------------- | | `map_metadata.resolution` | float | Size of each cell in meters | | `map_metadata.width` | integer | Map width in cells | | `map_metadata.height` | integer | Map height in cells | | `map_metadata.origin` | object | Pose of the map origin (bottom-left corner) | | `data` | array | Occupancy values: `-1` = unknown, `0` = free, `100` = occupied | *** # Modes Introduction to the OM1 modes ### What Are Modes? Modes define the primary behavioral state and functional context of the OM1 system. Each mode adjusts how OM1 perceives its environment, processes user inputs, and prioritizes tasks. By switching modes, OM1 can dynamically transition between social interaction, exploration, patrol, or autonomous operation based on user intent or system triggers. Modes can be user-selected (via voice commands or UI). ### Supported Modes for Unitree Go2 ![](/files/el0RUT7rQLy5qptbXu6H) 1. Welcome mode - Initial greeting and user information gathering * Face detection and face anonymization. * The robot greets you and remembers you. 2. Slam mode - Autonomous navigation and mapping mode * Enables OM1 to explore and map its surroundings using its sensors. * Builds and updates internal maps for navigation and spatial awareness. * Typically used during setup, or when mapping new areas 3. Guard mode - Patrol and security monitoring mode In Guard Mode, OM1 performs scheduled or continuous patrols within a defined area. It uses onboard sensors and AI models to detect unusual activity, monitor for movement, or respond to security alerts. Designed for reliability and alertness, Guard Mode operates with high autonomy but can notify human operators when needed. Key Functions: * OM1 performs patrol routines within a defined area. * Monitors for motion or unusual activity during patrols. * Reports its status and logs key events during operation. 4. Conversation mode - Focused conversation and social interaction mode * OM1 engages in direct communication with the user. * Focused on natural dialogue and maintaining user attention. 5. Navigation mode - Autonomous navigation mode * OM1 moves between defined points within a mapped area. * Uses existing maps (from Slam Mode) for pathfinding. * Avoids obstacles and ensures safe movement to the target location. ### Introducing Lifecycle Each operational mode in OM1 follows a defined lifecycle, representing the complete process from entry to exit of that mode. A mode lifecycle ensures predictable behavior, safe transitions, and consistent data handling across all system states. For more details, see [Lifecycle](/modes-and-lifecycle/lifecycle). # Mode Selection Different ways to switch between modes Once your Go2 is setup to run in full autonomy, you can get started with exploring different modes offered via OM1 and explore the functionalities. There are multiple ways to do this: 1. Context Aware 2. Time based 3. Input triggered 4. Manual trigger (via portal) ### Context Aware The system supports context-aware transition rules that enable automatic mode switching based on operational state and task completion for a particular mode. Once the robot is powered on, it autonomously progresses through predefined operational modes without requiring user commands. * Upon startup, the robot enters Welcome Mode, where it greets the user and performs facial capture for identification purposes. * Facial data is processed in compliance with privacy-preserving mechanisms. Face detection and anonymisation takes place on the edge device. * After successful initialization and user recognition, the robot automatically transitions to SLAM (Simultaneous Localization and Mapping) Mode. * The robot maps the surrounding environment. Location labels are generated and stored for future navigation tasks. * Once the SLAM process is completed successfully, the robot transitions to Navigation Mode. The robot can now navigate autonomously within the mapped area. * Guard Mode is excluded from context aware transitions and must be explicitly activated by the user via voice commands or through the OpenMind portal, as required. Example config to setup context\_aware transition type for transitioning into navigation mode from slam mode. ``` { from_mode: "slam", to_mode: "navigation", transition_type: "context_aware", context_conditions: { exploration_done: true }, priority: 3, cooldown_seconds: 5.0, } ``` ``` { from_mode: "mode_1", to_mode: "mode_2", transition_type: "context_aware", context_conditions: { owner_identified: true, temperature: 70 }, priority: 2, cooldown_seconds: 5.0, } ``` ### Time based Time-based transitions enable the robot to automatically switch operational modes after a predefined period of elapsed time. These transitions are designed to ensure safe and secure operation without requiring continuous user input. Below is an example of configuring a time-based transition. Once configured, the system automatically transitions from the `from_mode` to the specified `to_mode` after the defined `timeout_seconds` has elapsed (300 seconds in this example). If an intervention occurs during this interval, such as user interaction or another eligible transition being triggered, the system evaluates all applicable transition rules. The transition associated with the highest priority is selected and executed. Example config to setup time based transition type for transitioning into guard mode from conversation mode. ``` { "from_mode": "conversation", "to_mode": "guard", "transition_type": "time_based", "trigger_keywords": ["guard", "security", "patrol", "keep watch"], "priority": 2, "timeout_seconds": 300.0 // Switch to guard mode after 5 minutes of inactivity } ``` ### Input Triggered (Voice Commands) 1. Configure your API key in `~/.bashrc` file and start your machine in full autonomy mode. 2. Start talking to your robot dog and ask it to switch to a particular mode. **Example:** > The robot says: "Hi, I'm your friendly robot dog, how may I help you?" > > The user can then request the robot to switch to a particular mode by saying: "Switch to \[desired mode]." | Desired Mode | Trigger Keywords | | ---------------- | --------------------------------------------------------------- | | **Welcome** | `reset`, `start over`, `welcome mode`, `restart`, `initialize` | | **Conversation** | `talk`, `chat`, `conversation`, `tell me`, `ask you`, `discuss` | | **SLAM** | `explore`, `map`, `navigate`, `look around`, `slam`, `wander` | | **Navigation** | `navigate`, `navigation`, `go to`, `take me to`, `show me` | | **Guard** | `guard`, `security`, `patrol`, `keep watch` | ### Manual Trigger (via Portal) 1. Configure your API key in `~/.bashrc` file and start your machine in full autonomy mode. 2. Login to your OM1 portal and head over to **Machine Teleops** on the left navigation bar. ![](/files/odSeyxvxDavR2wK2iOQ8) 3. Once connected, you'll see your machine listed as **Online** at the top of the screen. ![](/files/NrT2y8ClhaRVIYfeVtHr) 4. Scroll down to access the **Mode Selection** section. From here, choose the mode you want your robot to switch to. ![](/files/GghcIBy3oDgkorbJm6m3) 5. In **SLAM Mode**, you can manually guide the robot through its environment to generate a map. As you move, you can label specific areas and have the robot remember them. The resulting map should appear as follows: ![](/files/wzUBjMycMl1u5WAlSAO3) 6. Once the map is saved, switch to **Navigation Mode** to make the robot move autonomously between locations. Use the dropdown menu to select a destination. ![](/files/7Q6LGJO2sxfvaX491wWL) 7. You can also monitor three live camera streams directly from the portal. ![](/files/V4JFREBvcy0sTktIdlmQ) These steps and exploration methods provide a structured approach to understanding and managing OM1’s modes. # Transition Rules Introduction to transition rules Transition rules define how and when the robot switches between operational modes. Each rule specifies a source mode, a target mode, a trigger mechanism, and an execution priority. The system continuously evaluates these rules and executes the most appropriate transition based on current inputs, elapsed time, and contextual conditions. Transition rules are defined as an array of rule objects under the transition\_rules configuration. ### Rule Schema Each transition rule is represented as an object with required and optional fields. #### Required Fields | Field | Type | Required | Description | | -------------------- | --------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | `from_mode` | `string` | Yes | The operational mode in which this transition rule is evaluated | | `to_mode` | `string` | Yes | The target mode to transition into when the rule is triggered | | `transition_type` | `string` | Yes | Defines how the transition is triggered. Supported values: `input_triggered`, `time_based`, `context_aware` | | `priority` | `integer` | Yes | Determines rule precedence when multiple transition rules are eligible (higher value takes precedence) | | `trigger_keywords` | `array` | No | List of keywords or phrases that can trigger the transition via user input; applicable to `input_triggered` and `time_based` transitions | | `cooldown_seconds` | `number` | No | Minimum time in seconds before the same transition rule can be triggered again, preventing rapid or repeated transitions | | `context_conditions` | `object` | No | Contextual conditions that must be satisfied for a `context_aware` transition to be eligible | Supported Operators in `context_conditions` | Operator | Value Type | Description | | ---------- | ------------------- | ------------------------------------------------------------------------- | | `min` | `number` | Minimum allowed numeric value | | `max` | `number` | Maximum allowed numeric value | | `contains` | `string` or `array` | Checks whether a string contains a substring or an array contains a value | | `one_of` | `array` | Evaluates to true if the context value matches any value in the array | | `not` | `any` | Negates the specified condition | #### Priority Resolution When multiple transition rules are eligible: * Rules are sorted by priority (higher values take precedence) * The highest-priority rule is selected # Lifecycle Introducing Lifecycle ### Overview A lifecycle defines the operational boundaries of a mode, from its activation to its completion or transition to another mode. It is responsible for initializing, managing, and safely terminating a mode's operation, ensuring predictable transitions, consistent state handling, and controlled execution of mode-specific logic. A mode represents a functional state of the robot, such as Guard, SLAM, or Navigation, and each mode runs within a lifecycle that determines how and when it starts, executes, and ends. ![](/files/LGNPWiW14bAedV3FH7KM) ### Conceptual Model | Concept | Description | | ------------- | ------------------------------------------------------------------------------------ | | **Lifecycle** | Defines the start, execution, and end of a mode | | **Stage** | Represents a logical phase within the lifecycle (e.g., startup, entry, active, exit) | | **Hook** | A programmable event point that executes specific actions at key stages | | **Mode** | The operational context governed by the lifecycle | > **In short:** Lifecycle controls the flow → Stages define phases → Hooks perform actions → Mode defines behavior. ### Stages and Hooks Each lifecycle is composed of several stages. Hooks are executed at specific points in these stages to perform initialization, cleanup, or handling tasks. | Stage | Hook | Description | | ------------ | ------------- | --------------------------------------------------------------------------------------------- | | **Startup** | `ON_STARTUP` | Executes system-level initialization before any mode begins | | **Entry** | `ON_ENTRY` | Runs when entering a mode; prepares mode-specific resources and context | | **Active** | — | Main operational phase; mode logic runs continuously (no fixed hook) | | **Exit** | `ON_EXIT` | Executes cleanup, saves state, and prepares for the next transition | | **Timeout** | `ON_TIMEOUT` | Handles cases where a mode exceeds its defined duration or fails to complete expected actions | | **Shutdown** | `ON_SHUTDOWN` | Performs final cleanup and safe termination of the lifecycle management system | #### Startup Stage Triggered by `ON_STARTUP`. Executes system-level initialization before any mode begins. #### Entry Stage Triggered by `ON_ENTRY`. Runs when entering a mode; prepares mode-specific resources and context. #### Active Stage Represents the main operational phase of the mode. No fixed hook; mode logic runs continuously during this stage. #### Exit Stage Triggered by `ON_EXIT`. Executes cleanup, saves state, and prepares for the next transition. #### Timeout Stage Triggered by `ON_TIMEOUT`. Handles cases where a mode exceeds its defined duration or fails to complete expected actions. #### Shutdown Stage Triggered by `ON_SHUTDOWN`. Performs final cleanup and safe termination of the lifecycle management system. # Gazebo Quadruped Simulation and Control ### System Requirements | Component | Minimum | Good/Recommended | Ideal | | --------- | -------------------------------------------------- | -------------------------------------------- | ------------------------------------------------------------ | | **CPU** | Intel i7 (10th gen) or AMD Ryzen 7-8 cores minimum | Intel i9 (12th gen+) or AMD Ryzen 9-12 cores | AMD Ryzen 9 7950X or Intel i9-13900K 16+ cores (24+ threads) | | **RAM** | 16 GB | 32 GB | 64 GB | | **GPU** | NVIDIA GTX 1660 Ti 6 GB VRAM | NVIDIA RTX 3070 or RTX 4060 Ti 8-12 GB VRAM | NVIDIA RTX 4080/4090 16+ GB VRAM with CUDA 11.8+ | | **OS** | Ubuntu 22.04 | Ubuntu 22.04 | Ubuntu 22.04 | It's ideal to have at least 128 GB SSD storage for the setup to run smoothly. ### Simulation Instructions To get started with **Gazebo** and **Unitree SDK**, please install cyclonedds and **ROS2 Humble** first. You can find the installation steps [here](/core-concepts/middleware). To install compilers and other tools to build ROS packages, run ```bash sudo apt install ros-dev-tools ``` Install the following additional dependencies ```bash sudo apt install ros-humble-rmw-cyclonedds-cpp sudo apt install ros-humble-rosidl-generator-dds-idl ``` Set up your environment by sourcing the following file. ```bash source /opt/ros/humble/setup.bash ``` If you don't have uv installed, use the following command to install it on your system. ```bash curl -LsSf https://astral.sh/uv/install.sh | sh ``` Check if you have rosdep installed by running `rosdep` or `rosdep --version`. If it is not installed, run the following: ```bash sudo apt-get update sudo apt-get install python3-rosdep ``` Once you've successfully completed above steps, follow the following steps to start the gazebo simulation, generate SLAM map of the surrounding and start navigation. Step 1: Clone the [OM1-sim](https://github.com/OpenMind/OM1-sim) repository: ```bash git clone https://github.com/OpenMind/OM1-sim.git ``` Step 2: Initialize rosdep by setting up the source list (this is only needed once per machine) > **Note:** Run 'sudo rosdep init' only if rosdep has not been initialized on this machine before. If it has, skip this line and start from 'rosdep update'. ```bash cd OM1-sim sudo rosdep init rosdep update rosdep install --from-paths . --ignore-src -r -y ``` It automatically installs all system dependencies needed by the ROS packages in your current directory. Step 3: Build all the packages: ```bash colcon build ``` Once the build is successful, create a virtual environment and install the dependencies ```bash uv venv --python 3.10 source .venv/bin/activate uv pip install . ``` Now you should be able to launch the **Gazebo Simulator**. Step 4: Open a terminal and run the following commands. You'll now be able to see the Gazebo and RViZ windows launch on your system. ```bash source install/setup.bash ros2 launch go2_gazebo_sim go2_launch.py ``` Step 5: Run Zenoh Ros2 Bridge To run the Zenoh bridge for the Unitree Go2, you need to have the Zenoh ROS 2 bridge installed. You can find the installation instructions in the [Zenoh ROS 2 Bridge documentation](https://github.com/eclipse-zenoh/zenoh-plugin-ros2dds) After installing the Zenoh ROS 2 bridge, you can run it with the following command: ```bash zenoh-bridge-ros2dds -c ./zenoh/zenoh_bridge_config.json5 ``` Step 6: Start OM1 Refer to the [Installation Guide](/developing/1_get-started) for detailed instructions. Setup your API key in `.bashrc` file and run your simulation agent: Get your API key from the [portal](https://portal.openmind.com), and add it to `bashrc` ```bash vi ~/.bashrc ``` ```bash export OM_API_KEY="" ``` Now, run the simulation agent ```bash CONFIG=unitree_go2_autonomy USE_SIM=true make dev ``` Step 7: Teleoperate the robot in simulation You can also use teleoperation to control the robot through your keyboard. Open `OM1-sim` directory in a new terminal and run the following commands ```bash source install/setup.bash ros2 run teleop_twist_keyboard teleop_twist_keyboard ``` Use the keyboard controls displayed in the terminal to move the robot: ``` i - Move forward , - Move backward j - Turn left l - Turn right k - Stop U/O/M/> - Move diagonally ``` > **Note**: > > 1. Auto charging feature is not supported with Gazebo but it will be launched soon. Stay tuned! > 2. SLAM Map generation and Navigation are only offered as part of Premium features and would require an active subscription to Enterprise Plan on [OpenMind Portal](https://portal.openmind.com). # Isaac Sim Simulation and Control with OM1 ### System Requirements | Component | Minimum | Good | Ideal | | --------- | --------------------------------------------- | --------------------------------------------- | ------------------------------------------------------------------------------- | | OS | Ubuntu 20.04 / 22.04 | Ubuntu 20.04 / 22.04 | Ubuntu 20.04 / 22.04 | | CPU |

Intel Core i7 (7th Gen)
AMD Ryzen 5

|

Intel Core i7 (9th Gen)
AMD Ryzen 7

|

Intel Core i9, X-series or higher
AMD Ryzen 9, Threadripper or higher

| | Cores | 4 | 8 | 16 | | RAM | 32 GB | 64 GB | 64 GB | | Storage | 50 GB SSD | 500 GB SSD | 1 TB NVMe SSD | | GPU | GeForce RTX 3070 | GeForce RTX 4080 | RTX Ada 6000 | | VRAM | 8 GB | 16 GB | 48 GB | ### Features * Isaac Sim: Realistic physics simulation of the Unitree Go2. * Navigation Stack (Nav2): Fully configured navigation stack for autonomous movement. * SLAM: Mapping capabilities using slam\_toolbox. * LiDAR Support: Simulation of Velodyne VLP-16 and Unitree 4D LiDAR. ### Simulation Instructions To get started with **Isaac Sim** and **Unitree SDK**, please install cyclonedds and **ROS2 Humble** first. You can find the installation steps [here](/core-concepts/middleware). To install compilers and other tools to build ROS packages, run ```bash sudo apt install ros-dev-tools ``` Install the following additional dependencies ```bash sudo apt install ros-humble-rmw-cyclonedds-cpp sudo apt install ros-humble-rosidl-generator-dds-idl ``` Set up your environment by sourcing the following file. ```bash source /opt/ros/humble/setup.bash ``` If you don't have uv installed, use the following command to install it on your system. ```bash curl -LsSf https://astral.sh/uv/install.sh | sh ``` Check if you have rosdep installed by running `rosdep` or `rosdep --version`. If it is not installed, run the following: ```bash sudo apt-get update sudo apt-get install python3-rosdep ``` Once you've successfully completed above steps, follow the following steps to start the Isaac Sim simulation with OM1. Step 1: Clone the [OM1-ros2-sdk](https://github.com/OpenMind/OM1-ros2-sdk) repository: ```bash git clone https://github.com/OpenMind/OM1-ros2-sdk.git ``` Step 2: Initialize rosdep by setting up the source list (this is only needed once per machine) > **Note:** Run 'sudo rosdep init' only if rosdep has not been initialized on this machine before. If it has, skip this line and start from 'rosdep update'. ```bash cd OM1-ros2-sdk sudo rosdep init rosdep update rosdep install --from-paths . --ignore-src -r -y ``` It automatically installs all system dependencies needed by the ROS packages in your current directory. Step 3: Build all the packages: ```bash colcon build ``` Once the build is successful, create a virtual environment and install the dependencies ```bash uv venv --python 3.10 source .venv/bin/activate uv pip install . ``` Step 4: Install Isaac Sim Open a new terminal window and switch to `unitree/isaac_sim` directory within `OM1-ros2-sdk` ```bash cd OM1-ros2-sdk/unitree/isaac_sim uv venv --python 3.11 --seed env_isaacsim source env_isaacsim/bin/activate # note that here we are installing IsaacSim 5.1 pip install "isaacsim[all,extscache]==5.1.0" --extra-index-url https://pypi.nvidia.com # install the following or another CUDA-enabled PyTorch build that matches your system architecture pip install -U torch==2.7.0 torchvision==0.22.0 --index-url https://download.pytorch.org/whl/cu128 # after installation, run the following to test successful installation isaacsim ``` If the previous command ran with no issues, Isaac Sim was installed successfully. Step 5: Now, let's get our system ready to run OM1 with Isaac Sim. Open a terminal and run the following commands. To run the script, export the following > **Note**: a trained policy is required, which should contain the policy.pt, env.yaml, and deploy.yaml files ```bash export ROS_DISTRO=humble export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:PATH_TO_VENV/env_isaaclab/lib/python3.11/site-packages/isaacsim/exts/isaacsim.ros2.bridge/humble/lib ``` > **Note:** Make sure to replace PATH\_TO\_VENV with the actual path to your virtual environment. We use `env_isaacsim` (Python 3.11) for Isaac Sim and `.venv` (Python 3.10) for other services like orchestrator due to compatibility requirements. We support Isaac Sim for Unitree Go2 and G1. To run the simulation for Go2, run ```bash source env_isaacsim/bin/activate python3 run.py # using default policy python3 run.py --policy_dir YOUR_POLICY_DIR # using your own policy ``` To run the simulation for G1, run ```bash source env_isaacsim/bin/activate python3 run.py --robot_type g1 # using default policy python3 run.py --robot_type g1 --policy_dir YOUR_POLICY_DIR # using your own policy ``` You'll now be able to see Isaac Sim running on your system. ![](/files/W3sa2RB6TtSH0OSU2xdd) > **Note**: You can skio Step 6 to Step 8 if you don't have Enterprise Plan Subscription. SLAM and Navigation are only supported as Premium features. Step 6: Open a new terminal , switch to base directory `OM1-ros2-sdk` and run: ```bash source install/setup.bash ros2 launch go2_sdk sensor_launch.py use_sim:=true ``` This will bring up the `om/path` topic, enabling OM1 to understand the surrounding environment. Step 7: Open a new terminal and run: ```bash export PYTHONPATH=$PYTHONPATH:$(pwd)/.venv/lib/python3.10/site-packages source install/setup.bash ros2 launch orchestrator orchestrator_launch.py use_sim:=true ``` This will bring up the `orchestrator`, to consume data collected by `om1_sensor` for SLAM and Navigation. Step 8: Run Zenoh Ros2 Bridge To run the Zenoh bridge for the Unitree Go2, you need to have the Zenoh ROS 2 bridge installed. You can find the installation instructions [here](/core-concepts/middleware/zenoh-bridge). After installing the Zenoh ROS 2 bridge, you can run it with the following command: ```bash zenoh-bridge-ros2dds -c ./zenoh/zenoh_bridge_config.json5 ``` Step 9: Start OM1 Refer to the [Installation Guide](/developing/1_get-started) for detailed instructions. Setup your API key in `.bashrc` file and run your simulation agent: Get your API key from the [portal](https://portal.openmind.com), and add it to `bashrc` ```bash vi ~/.bashrc ``` ```bash export OM_API_KEY="" ``` Now, run the simulation agent ```bash CONFIG=unitree_go2_autonomy USE_SIM=true make dev ``` > **Note**: Update your agent name depending on robot type. Congratulations! You have launched Isaac Sim with OM1 successfully. #### Control Methods When keyboard control is enabled (default), use the following keys: | Key | Action | | ----------------- | ------------- | | `↑` or `Numpad 8` | Move forward | | `↓` or `Numpad 2` | Move backward | | `←` or `Numpad 4` | Strafe left | | `→` or `Numpad 6` | Strafe right | | `N` or `Numpad 7` | Rotate left | | `M` or `Numpad 9` | Rotate right | > **Note**: We don't have auto charging feature supported with Isaac Sim but it will be launched soon. Stay tuned! # Cloud Isaac Sim Learn how to run cloud Isaac Sim integrated with OM1 ### Cloud Isaac Sim Developer Walkthrough Cloud Isaac Sim enables you to run robot simulations on managed cloud infrastructure, fully integrated with OM1. One of the biggest challenges in robotics development is that the robot isn't always available when you need it — someone else may be using it, you may be working remotely, or you may just want to validate an idea before deploying to real hardware. The Cloud Simulator lets you go from zero setup to an autonomous robot in a few minutes, entirely from your browser. This guide has two parts: * [**Part 1 — Autonomy in the Portal**](#part-1-autonomy-in-the-portal): launch a simulated robot, build a map with SLAM, create an autonomous patrol, monitor it remotely, and configure automatic charging — no code required. * [**Part 2 — Connecting OM1**](#part-2-connecting-om1): run the OM1 runtime against your cloud simulator instance, either in the cloud or from your local machine. ### Prerequisites * OpenMind Portal account on **Builder plan** or higher (required for Cloud Simulator access) * API key (found in your portal) * OM1 codebase built (`make build`) — only needed for [Part 2](#part-2-connecting-om1) ### Cost & Billing Cloud Simulator usage is billed in OMCU (OpenMind Compute Units). Billing begins as soon as an instance is **allocated** and stops only when the instance is deleted. Ensure your account has sufficient balance before launching. A **Builder plan** or higher is required to access the Cloud Simulator. Check your OMCU balance and plan in the [OpenMind Portal](https://portal.openmind.com) dashboard before starting. ### Part 1 — Autonomy in the Portal Everything in this part runs from the browser. You'll launch a simulated robot, give it an understanding of its surroundings with SLAM, set up an autonomous patrol, watch it operate remotely, and configure it to charge itself — all without physical hardware or writing any code. #### Step 1: Launch a Simulator Session 1. Log in to the [OpenMind Portal](https://portal.openmind.com) 2. Navigate to **Cloud Simulator** from the sidebar 3. Select the instance type, robot model and choose the environment you'd like to work in, then launch the simulator **Instance Types** Choose based on your simulation workload: | Instance Type | vCPUs | RAM | Best For | Price (Per Hour) | | --------------- | ----- | ----- | ------------------------------------ | ---------------- | | **Standard** | 8 | 32 GB | Development & testing | 4800 OMCU | | **Performance** | 16 | 64 GB | Heavy compute, multi-robot scenarios | 7200 OMCU | ![](/files/fTe62g7g1hdOP7iL3LgF) **Supported Robots** * Unitree Go2 * Unitree G1 * LimX Tron * Deep Robotics M20 Pro **Available Environments** * Warehouse * Warehouse Lite * Apartment **Launch Time** The instance goes through the following stages before it is ready: 1. Allocating Instance 2. Load Robot Configuration 3. Launching Simulator 4. Render Environment 5. Finalizing Simulator Setup > **Note**: Expect **10-15 minutes** for your instance to fully initialize. Once you initiate the launch, the system begins setting up your cloud environment. ![](/files/n0pwjB2eN1rdmLwdkYkS) ![](/files/fFGJ442RKfQlCfqJ6OgJ) The instance is ready when the status changes to **Running**. > **Note**: If the requested GPU is not available, you will see the error below. Wait a few minutes and try again, or switch to a different instance type. ![](/files/5oC0pyTYSPEdrkw7ZEQ0) Open your running session from the portal dashboard. ![](/files/kf2BOh9zmbNySdkHfzIg) The simulator view reflects the robot you selected when launching the instance: Unitree Go2: ![](/files/Udb68vYTBbIALxCsBjGs) Unitree G1: ![](/files/l1oo6L0eVyosDb5JZqYK) LimX Tron: ![](/files/wRKhXqvz2A8lMZ1nxyfM) Deep Robotics M20 Pro: ![](/files/lmTt9BJiQVV3Zy6qc73O) #### Step 2: Explore & Teleoperate Before building autonomy, confirm everything is working as expected. From the portal you have access to the **live camera feed**, **robot status**, and **manual teleoperation controls**. Drive the robot around to verify that it's connected and responding correctly. This is also a good way to familiarize yourself with the environment before creating autonomous behaviors. > **Tip**: You can also drive the robot with an **Xbox controller**. Pair the controller to your computer over Bluetooth and use it to teleoperate the robot in the simulator. #### Step 3: Build a Map with SLAM Next, give the robot an understanding of its surroundings. Open the **Map view** tab and start **SLAM**, which lets the robot build a map while it explores the environment. As the robot moves, it continuously observes its surroundings and constructs the map in real time. SLAM produces a live **3D point cloud** of the space, colored by height, that you can rotate and zoom to inspect: ![](/files/yrjZEgEDhOnjT0QN9RV6) The point cloud is also flattened into a **2D navigation map** — an occupancy grid showing walls and obstacles. This is the navigation-ready map used for autonomous tasks like patrols and navigation. Once **Navigation** mode is active, the map's **Set Goal** and **Localize** tools become available, and the robot's live position is shown on the map: ![](/files/RWcFCRO3mX073s1ktWPm) #### Step 4: Create an Autonomous Patrol Instead of manually driving the robot every time, open the **Route Planner** tab and create a **patrol route** by placing waypoints throughout the environment. 1. Click **+ New Route** to start a fresh route. 2. Toggle **Add Waypoints** and click points across the map to lay out the path. Use smooth turns so the robot can navigate naturally. **Undo Last** removes the most recent waypoint, and the node/edge count updates as you build. You can also **Import** or **Export** a route to reuse it later. 3. When you're happy with the path, click **Deploy**. The robot then takes over and begins following the route autonomously, navigating between each waypoint while continuously localizing itself within the map. ![](/files/E7qLupNSsNSf6HzYSZXU) #### Step 5: Monitor the Patrol While the robot carries out its patrol, monitor everything directly from the portal — the live camera stream, robot status, and patrol progress. This makes it easy to remotely verify that everything is operating as expected without being physically present. #### Step 6: Configure Automatic Charging Autonomous robots also need to manage their battery. Instead of waiting for an operator to intervene, configure a **battery threshold** that automatically sends the robot back to its charging station. Set the minimum battery level. Once the battery drops below that threshold, the robot automatically: 1. Pauses its patrol 2. Returns to the docking station 3. Re-localizes if needed 4. Docks and charges Once charged, it's ready to continue operating. This enables long-running deployments with minimal manual intervention. > **Note**: Automatic charging is currently supported on **Unitree Go2 only**. #### Step 7: Automatic Localization If the robot starts up again or loses localization, it can automatically determine its position on the existing map before continuing its mission. This removes another manual step from the deployment process and helps keep operations running smoothly. #### Cleaning Up When you're finished with your simulation: 1. Return to the Cloud Simulator dashboard 2. Click **Delete Instance** ![](/files/Pj7S9K7lz7QitrannvAh) 3. Confirm the deletion — this stops billing and frees cloud resources ### Part 2 — Connecting OM1 Part 1 runs the robot's autonomy entirely from the portal. If you want to drive the simulator with the **OM1 runtime** — for example to test your own agent behaviors, inputs, and actions — connect OM1 to a running instance using one of the two options below. #### Option A: Cloud VS Code Access a full VS Code environment running in the cloud with OM1 pre-configured. ![](/files/NPL6x8VoIZBdyNjZPWHK) From here, you can: * Edit and run OM1 code directly in the cloud * Execute `make run` commands without local hardware * Test and debug your robot behaviors #### Option B: Local Environment Run OM1 on your local machine and connect to the cloud simulator: 1. Copy your **API Key** from the portal and ensure `OM_API_KEY` is set in your environment or `.env` file. 2. Open `config/unitree_go2_autonomy.json5` in your local OM1 repo. This config targets a Unitree Go2 in the cloud simulator with voice input, VLM, and autonomous movement. You can adjust the `system_prompt_base`, robot inputs, and actions to match your use case. 3. Run the config: ```bash CONFIG=unitree_go2_autonomy USE_SIM=true make dev ``` ### What's Next? * Explore the [unitree\_go2\_modes config](https://github.com/OpenMind/OM1/blob/main/config/unitree_go2_modes.json5) to try multi-mode behaviors (including SLAM and Patrol) in the cloud sim * Read the [Configuration Guide](/core-concepts/concepts/3_configuration) to modify inputs, actions, and prompts in your config # Troubleshooting Guidelines ### Gazebo Specific Issues #### colcon build fails **Problem:** Build process exits with errors. **Solution:** 1. Ensure you're not running `colcon build` inside a virtual environment 2. Deactivate any active environments (including conda) 3. Open a fresh terminal and retry the build #### Gazebo and RViz fail to launch **Problem:** Simulation environment doesn't start properly. **Solution:** * Verify CycloneDDS is configured correctly on your system * Check that ROS 2 middleware is properly initialized #### Robot stops moving unexpectedly **Problem:** Robot is unresponsive to movement commands. **Solution:** * Manually reposition the robot in Gazebo using translate or rotate mode * Alternatively, use RViz to send a 2D Nav Goal pose to the robot #### Orchestrator throws errors **Problem:** API communication or initialization fails. **Solution:** 1. Verify environment variables are set in your `~/.bashrc`: ```bash export OM_API_KEY= export OM_API_KEY_ID= ``` 2. Confirm your virtual environment is active with Python 3.10: ```bash python --version which python ``` **Problem:** Packages not found. **Solution:** 1. Confirm your virtual environment is active with Python 3.10: ```bash python --version which python ``` 2. Confirm you installed the dependencies using `uv pip install`, during the setup. If you still face issues, try deleting and creating a new virtual environment. Make sure to export `PYTHONPATH` to correct location. ### Audio Issues #### Robot doesn't respond to voice commands **Problem:** Audio input/output is not working. **Solution:** * Check system audio settings * Verify the correct microphone is selected as input * Verify the correct speaker is selected as output * Test audio with a simple recording to confirm functionality # API Reference Welcome to the OpenMind API Reference OpenMind integrates with multiple LLM providers to offer a diverse range of features. This API reference provides details on endpoints, parameters, and responses, enabling efficient interaction with the OpenMind API. ### API Keys OpenMind requires an API key to authenticate requests. You can obtain an API key by signing up for an account on the [OpenMind portal](https://portal.openmind.com). The API key must be included in the `Authorization` or `x-api-key` header of each request, used to authenticate your requests and track usage quotas. **Keep your API key confidential**. Never share it with others or expose it in client-side code, such as in browsers or apps. Remember to include your API key in the `Authorization` or `x-api-key` header of each request. For example: ```bash x-api-key: YOUR_API_KEY # or, Authorization: Bearer YOUR_API_KEY ``` For websocket connections, include the API key in the query string. For example: `wss://api.openmind.com?api_key=`. ### API Pricing Access our API, scale usage as needed, and stay in control of costs. For detailed API pricing, refer [here](https://github.com/OpenMind/OM1/blob/main/docs/api-reference/api_pricing/README.md) * High-speed requests * Cutting-edge large models * Integrated modules for multiple robots For developer walkthrough and support reach out to: #### LLM Models **OpenAI** | Model Name | Input Price (per 1M tokens) | Output Price (per 1M tokens) | | ------------ | --------------------------- | ---------------------------- | | gpt-4o | 42500 OMCU | 170000 OMCU | | gpt-4o-mini | 7000 OMCU | 28000 OMCU | | gpt-4.1 | 35000 OMCU | 140000 OMCU | | gpt-4.1-mini | 7000 OMCU | 28000 OMCU | | gpt-4.1-nano | 2000 OMCU | 8000 OMCU | | gpt-5 | 25000 OMCU | 200000 OMCU | | gpt-5-mini | 4500 OMCU | 36000 OMCU | | gpt-5-nano | 500 OMCU | 4000 OMCU | | gpt-5.1 | 25000 OMCU | 200000 OMCU | | gpt-5.2 | 35000 OMCU | 280000 OMCU | **Gemini** | Service | Input Price (per 1M tokens) | Output Price (per 1M tokens) | | ---------------------- | --------------------------- | ---------------------------- | | Gemini 3.5 Flash | 12500 OMCU | 90000 OMCU | | Gemini 3.1 Pro Preview | 40000 OMCU | 180000 OMCU | | Gemini 3.1 Flash Lite | 2500 OMCU | 15000 OMCU | | Gemini 2.5 Flash | 3000 OMCU | 25000 OMCU | | Gemini 2.5 Flash Lite | 1000 OMCU | 4000 OMCU | | Gemini 2.5 Pro | 25000 OMCU | 150000 OMCU | **DeepSeek** | Service | Input Price (per 1M tokens) | Output Price (per 1M tokens) | | ------------- | --------------------------- | ---------------------------- | | DeepSeek Chat | 1400 OMCU | 2800 OMCU | **X.AI Grok** | Service | Input Price (per 1M tokens) | Output Price (per 1M tokens) | | ------------- | --------------------------- | ---------------------------- | | grok-2-latest | 20000 OMCU | 100000 OMCU | | grok-3-beta | 30000 OMCU | 150000 OMCU | | grok-4-latest | 30000 OMCU | 150000 OMCU | | grok-4 | 30000 OMCU | 150000 OMCU | **Near AI** | Service | Input Price (per 1M tokens) | Output Price (per 1M tokens) | | -------------------------------- | --------------------------- | ---------------------------- | | Qwen/Qwen3-30B-A3B-Instruct-2507 | 1500 OMCU | 5500 OMCU | | deepseek-ai/DeepSeek-V3.1 | 10500 OMCU | 31000 OMCU | | openai/gpt-oss-120b | 1500 OMCU | 5500 OMCU | | openai/gpt-5.2 | 18000 OMCU | 155000 OMCU | | zai-org/GLM-4.7 | 8500 OMCU | 33000 OMCU | | anthropic/claude-sonnet-4-5 | 30000 OMCU | 155000 OMCU | | google/gemini-3-pro | 12500 OMCU | 150000 OMCU | **Open Router** | Service | Input Price (per 1M tokens) | Output Price (per 1M tokens) | | --------------------------------- | --------------------------- | ---------------------------- | | deepseek/deepseek-v3.2 | 2500 OMCU | 3800 OMCU | | anthropic/claude-sonnet-4.5 | 30000 OMCU | 150000 OMCU | | anthropic/claude-opus-4.5 | 50000 OMCU | 250000 OMCU | | anthropic/claude-haiku-4.5 | 10000 OMCU | 50000 OMCU | | moonshotai/kimi-k2.5 | 4500 OMCU | 25000 OMCU | | minimax/minimax-m2.1 | 2700 OMCU | 9500 OMCU | | z-ai/glm-4.7 | 4000 OMCU | 15000 OMCU | | x-ai/grok-4-fast | 2000 OMCU | 5000 OMCU | | meta-llama/llama-3.3-70b-instruct | 9000 OMCU | 9000 OMCU | > **Note:** For free local inference, [Ollama](https://ollama.ai) supports models like llama3.2, mistral, and phi3 with no API costs. #### TTS Models (Text to Speech) | Service | Price (per 1M characters) | | ----------- | ------------------------- | | Eleven Labs | 30k OMCU | | Riva | 10K OMCU | We will support more models in the future. Contact us if you have any questions or need a custom solution. #### ASR Models (Speech to Text) | Service | Price (per 1 minute) | | -------------- | -------------------- | | Google ASR | 50 OMCU | | ElevenLabs ASR | 35 OMCU | # Account & Key Management Managing your OpenMind Account and API Keys Your OpenMind account and API keys are used for authentication and authorization in your applications. You can manage your account and keys via the [OpenMind portal](https://portal.openmind.com) or directly via an API. The API provides access to the full set of operations (generate, delete, account balance, and key listing). **Base URL:** `https://api.openmind.com/api/core` **Authentication:** All endpoints require a JWT token generated with [Clerk](https://clerk.com/). Include the token in the `Authorization` header as a Bearer token. ### Endpoints Overview | Method | Endpoint | Description | | ------ | ------------------ | -------------------------- | | POST | `/api_keys/create` | Create a new API key | | POST | `/api_keys/delete` | Delete an existing API key | | GET | `/account/balance` | Get your account balance | | GET | `/api_keys` | List all API keys | ### Create API Key Create a new API key for your account. Each plan has a limit on the number of API keys you can create. **Endpoint:** `POST /api_keys/create` #### Request ```bash curl -X POST https://api.openmind.com/api/core/api_keys/create \ -H "Authorization: Bearer YOUR_JWT_TOKEN" \ -H "Content-Type: application/json" ``` #### Response **Success (200 OK):** ```json { "message": "API key created successfully", "api_key": "om1_live_a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6q7r8s9t0u1v2w3x4y5z6", "api_key_info": { "id": "a1b2c3d4e5f6g7h8", "user_id": "user_2abc123xyz", "name": "api-key-1738713600000", "prefix": "om1_live_", "total_cost": 0, "deleted": false, "created_at": "2026-02-04T10:00:00Z", "updated_at": "2026-02-04T10:00:00Z" } } ``` **Error Responses:** ```json // 401 Unauthorized - Missing or invalid JWT token { "error": "User not found" } // 403 Forbidden - API key limit reached { "error": "API key limit reached. Your starter plan allows up to 5 API keys. Please delete an existing key or upgrade your plan." } // 500 Internal Server Error { "error": "Failed to create API key" } ``` > **Note:** The returned `api_key` is only shown once. Store it securely as you won't be able to retrieve it again. ### Delete API Key Mark an API key as deleted and invalidate it. This action cannot be undone. **Endpoint:** `POST /api_keys/delete` #### Request ```bash curl -X POST https://api.openmind.com/api/core/api_keys/delete \ -H "Authorization: Bearer YOUR_JWT_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "id": "a1b2c3d4e5f6g7h8" }' ``` #### Request Body | Field | Type | Required | Description | | ----- | ------ | -------- | ------------------------------- | | `id` | string | Yes | The ID of the API key to delete | #### Response **Success (200 OK):** ```json { "message": "API key deleted successfully" } ``` **Error Responses:** ```json // 400 Bad Request - Missing or invalid ID { "error": "No data provided or invalid format" } { "error": "API key ID is required" } // 401 Unauthorized - Missing or invalid JWT token { "error": "User not found" } // 404 Not Found - API key doesn't exist or doesn't belong to user { "error": "API key not found" } // 500 Internal Server Error { "error": "Failed to delete API key" } ``` ### Get Account Balance Retrieve your current OMCU (OpenMind Compute Unit) balance and subscription details. **Endpoint:** `GET /account/balance` #### Request ```bash curl -X GET https://api.openmind.com/api/core/account/balance \ -H "Authorization: Bearer YOUR_JWT_TOKEN" ``` #### Response **Success (200 OK):** ```json { "plan": "pro", "omcu_balance": 150000, "monthly_unused_omcu": 50000, "monthly_total_omcu": 100000, "current_period_end": "2026-03-04T10:00:00Z", "cancel_at_period_end": false } ``` #### Response Fields | Field | Type | Description | | ---------------------- | -------------- | ----------------------------------------------------------------------------- | | `plan` | string | Your current subscription plan (e.g., "free", "starter", "pro", "enterprise") | | `omcu_balance` | integer | Total available OMCU credits (monthly + unused prepaid) | | `monthly_unused_omcu` | integer | Unused OMCU credits from your monthly subscription allowance | | `monthly_total_omcu` | integer | Total OMCU credits allocated for the current billing period | | `current_period_end` | string \| null | ISO 8601 timestamp of when the current billing period ends | | `cancel_at_period_end` | boolean | Whether the subscription will be cancelled at the end of the current period | **Error Responses:** ```json // 401 Unauthorized - Missing or invalid JWT token { "error": "User not found" } // 404 Not Found - User account not found { "error": "User not found" } ``` > **Note:** Note the following about your account balance: > > * `omcu_balance` represents your total available credits, including both monthly subscription credits and any prepaid/unused credits from previous periods > * Monthly credits reset at the start of each billing period > * Unused prepaid credits do not expire and carry over between billing periods ### List API Keys Retrieve a list of all active API keys for your account. **Endpoint:** `GET /api_keys` #### Request ```bash curl -X GET https://api.openmind.com/api/core/api_keys \ -H "Authorization: Bearer YOUR_JWT_TOKEN" ``` #### Response **Success (200 OK):** ```json { "api_keys": [ { "id": "a1b2c3d4e5f6g7h8", "user_id": "user_2abc123xyz", "name": "api-key-1738713600000", "prefix": "om1_live_", "total_cost": 12500, "deleted": false, "created_at": "2026-01-15T10:00:00Z", "updated_at": "2026-02-04T10:00:00Z" }, { "id": "z9y8x7w6v5u4t3s2", "user_id": "user_2abc123xyz", "name": "api-key-1738800000000", "prefix": "om1_live_", "total_cost": 8750, "deleted": false, "created_at": "2026-02-01T15:30:00Z", "updated_at": "2026-02-04T10:00:00Z" } ] } ``` #### Response Fields Each API key object contains: | Field | Type | Description | | ------------ | ------- | --------------------------------------------------------------- | | `id` | string | Unique identifier for the API key | | `user_id` | string | The user ID this key belongs to | | `name` | string | Auto-generated name for the key (format: "api-key-{timestamp}") | | `prefix` | string | The prefix of the API key ("om1\_live\_" or "om1\_test\_") | | `total_cost` | integer | Total OMCU credits consumed by this API key | | `deleted` | boolean | Whether the key is deleted (always false in this response) | | `created_at` | string | ISO 8601 timestamp of key creation | | `updated_at` | string | ISO 8601 timestamp of last update | **Error Responses:** ```json // 401 Unauthorized - Missing or invalid JWT token { "error": "User not found" } // 500 Internal Server Error { "error": "Failed to fetch API keys" } ``` > **Note:** Note the following about the API keys listed: > > * Only active (non-deleted) API keys are returned. > * The actual secret portion of the API key is never returned in this endpoint. > * The `hashed_key` field is stored in the database but not exposed in the API response for security. ### Authentication All endpoints require authentication using a JWT token issued by Clerk. Include the token in the Authorization header: ```bash Authorization: Bearer YOUR_JWT_TOKEN ``` #### Getting Your JWT Token You can obtain your JWT token through: 1. **OpenMind Portal:** Log in at [portal.openmind.com](https://portal.openmind.com) and copy your session token 2. **Clerk SDK:** Use the Clerk client library to authenticate and retrieve the session token programmatically #### Example with Token ```bash # Set your token as an environment variable export OPENMIND_JWT_TOKEN="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..." # Use it in your requests curl -X GET https://api.openmind.com/api/core/account/balance \ -H "Authorization: Bearer $OPENMIND_JWT_TOKEN" ``` ### Error Handling All endpoints follow consistent error response patterns: #### HTTP Status Codes | Code | Description | | ---- | --------------------------------------------------------- | | 200 | Success | | 400 | Bad Request - Invalid input parameters | | 401 | Unauthorized - Missing or invalid JWT token | | 403 | Forbidden - Action not allowed (e.g., rate limit reached) | | 404 | Not Found - Resource doesn't exist | | 500 | Internal Server Error - Server-side error | #### Error Response Format ```json { "error": "Descriptive error message" } ``` ### Rate Limits API key operations may be subject to rate limits depending on your subscription plan. If you exceed the rate limit, you'll receive a `429 Too Many Requests` response. # Google ASR Google Speech Recognition (ASR) API Reference The Google ASR API provides real-time speech-to-text transcription using Google Cloud Speech-to-Text. This WebSocket-based endpoint enables low-latency streaming recognition for live audio processing. **Base URL:** `wss://api.openmind.com` **Authentication:** Requires an OpenMind API key passed as a query parameter. > **Quick Start:** New integrations should use the V2 endpoint (`/api/core/google/asr`) for access to the Chirp 3 model and voice activity detection. V1 remains available for backward compatibility. ### API Versions The Google ASR service offers two API versions: #### V2 (Recommended) - Chirp 3 Model * **Endpoints:** `/api/core/google/asr`, `/api/core/google/asr/v2` * **Model:** Google's latest Chirp 3 speech recognition model * **Features:** * Enhanced accuracy with state-of-the-art Chirp 3 model * Voice activity detection events (`speech_start`, `speech_end`, `end_of_utterance`) * Configurable voice activity timeouts * Multi-language support in a single request * Better handling of accents and noisy environments * **Use when:** You need the highest accuracy and advanced features like voice activity detection > **About Chirp 3:** Google's Chirp 3 is a universal speech model trained on millions of hours of audio data, providing superior accuracy across 100+ languages and excellent performance in challenging acoustic conditions. #### V1 (Legacy) - Standard Model * **Endpoint:** `/api/core/google/asr/v1` * **Model:** Google Cloud Speech-to-Text v1 standard model * **Features:** * Standard speech recognition capabilities * Alternative language code support * Proven stability * **Use when:** You need compatibility with existing v1 implementations or prefer the standard model > **Recommendation:** Use v2 endpoints for new integrations to take advantage of the Chirp 3 model's improved accuracy and advanced features. ### Endpoints Overview | Protocol | Endpoint | Version | Description | | --------- | ------------------------- | ------- | ---------------------------------------------------------- | | WebSocket | `/api/core/google/asr` | V2 | Real-time speech recognition with Chirp 3 model (default) | | WebSocket | `/api/core/google/asr/v2` | V2 | Real-time speech recognition with Chirp 3 model (explicit) | | WebSocket | `/api/core/google/asr/v1` | V1 | Real-time speech recognition with standard model (legacy) | > **Note:** All endpoints also support the `/api/core/v1/` prefix for API versioning (e.g., `/api/core/v1/google/asr`). ### WebSocket Connection Establish a persistent WebSocket connection for streaming audio data and receiving real-time transcription results. **V2 Endpoint (Recommended):** `wss://api.openmind.com/api/core/google/asr?api_key=YOUR_API_KEY` **V1 Endpoint (Legacy):** `wss://api.openmind.com/api/core/google/asr/v1?api_key=YOUR_API_KEY` #### Connection Parameters | Parameter | Type | Required | Description | | --------- | ------ | -------- | ------------------------------------------------------------------------------------------------ | | `api_key` | string | Yes | Your OpenMind API key for authentication | | `model` | string | No | Override the speech recognition model (e.g. `chirp_3`, `latest_long`). Defaults to the chrip\_3. | #### Connection Example ```bash # V2 - Using wscat (install with: npm install -g wscat) wscat -c "wss://api.openmind.com/api/core/google/asr?api_key=om1_live_your_api_key" # V1 - Legacy endpoint wscat -c "wss://api.openmind.com/api/core/google/asr/v1?api_key=om1_live_your_api_key" ``` #### Connection Response Upon successful connection, you'll receive a confirmation message: **V2:** ```json { "type": "connection", "message": "Connected to ASR v2 service", "clientId": "1738713600000-a1b2c3d4e5f6g7h8" } ``` **V1:** ```json { "type": "connection", "message": "Connected to ASR v1 service", "clientId": "1738713600000-a1b2c3d4e5f6g7h8" } ``` #### Connection Errors **401 Unauthorized - Missing API Key:** ```json { "error": "Missing API key. Please connect with ?api_key=YOUR_API_KEY" } ``` **401 Unauthorized - Invalid API Key:** ```json { "error": "Invalid API key: [error details]" } ``` ### Sending Audio Data #### Message Format Send audio data as JSON messages over the WebSocket connection: ```json { "audio": "base64_encoded_audio_data", "rate": 16000, "language_code": "en-US" } ``` #### Message Fields | Field | Type | Required | Default | Description | | --------------- | ------- | -------- | --------- | --------------------------------------------------------------- | | `audio` | string | Yes | - | Base64-encoded audio data (LINEAR16 format) | | `rate` | integer | No | `16000` | Audio sample rate in Hz | | `language_code` | string | No | `"en-US"` | Language code for recognition (e.g., "en-US", "es-ES", "fr-FR") | > **Note:** Note the following when sending audio data: > > * The `rate` and `language_code` parameters only need to be sent with the first message. Subsequent messages can contain only the `audio` field. > * Audio must be LINEAR16 PCM encoded > * Maximum streaming duration is 4 minutes (240 seconds) per session ### Receiving Transcription Results #### Response Format **Transcription Result:** ```json { "asr_reply": "hello world", "clientId": "1738713600000-a1b2c3d4e5f6g7h8" } ``` **Error Message:** ```json { "type": "error", "message": "Error description", "clientId": "1738713600000-a1b2c3d4e5f6g7h8" } ``` #### Response Fields | Field | Type | Description | | ----------- | ------ | ------------------------------------------------------------------------------------------ | | `asr_reply` | string | Final transcription result for the audio segment | | `clientId` | string | Unique identifier for the WebSocket session | | `type` | string | Message type ("connection", "error", "speech\_start", "speech\_end", "end\_of\_utterance") | | `message` | string | Human-readable message for connection or error events | ### V2 Voice Activity Events V2 endpoints provide real-time voice activity detection events to help your application respond to speech activity: #### Event Types **Speech Activity Started:** ```json { "type": "speech_start", "message": "Speech activity detected", "clientId": "1738713600000-a1b2c3d4e5f6g7h8" } ``` **Speech Activity Ended:** ```json { "type": "speech_end", "message": "Speech activity ended", "clientId": "1738713600000-a1b2c3d4e5f6g7h8" } ``` **End of Utterance:** ```json { "type": "end_of_utterance", "message": "End of utterance", "clientId": "1738713600000-a1b2c3d4e5f6g7h8" } ``` #### Voice Activity Use Cases * **UI Feedback:** Show visual indicators when the user is speaking * **Turn-taking:** Detect when the user has finished speaking to trigger responses * **Recording Management:** Start/stop recording based on speech presence * **Conversation Flow:** Implement natural dialogue timing in voice assistants > **Note:** Voice activity events are only available in V2 endpoints. V1 endpoints return transcription results only. ### Audio Specifications #### Supported Audio Format * **Encoding:** LINEAR16 (16-bit PCM) * **Sample Rate:** 16000 Hz (recommended) or custom rate specified in first message * **Channels:** Mono (1 channel) * **Sample Width:** 2 bytes (16-bit) #### Calculating Audio Length Audio duration is calculated as: ``` duration_seconds = audio_bytes / (sample_rate × sample_width × channels) ``` For 16000 Hz mono LINEAR16: ``` duration_seconds = audio_bytes / (16000 × 2 × 1) = audio_bytes / 32000 ``` ### Usage Examples #### Python Example (V2 with Voice Activity) ```python import asyncio import websockets import base64 import json import pyaudio API_KEY = "om1_live_your_api_key" # V2 endpoint (recommended) WS_URL = f"wss://api.openmind.com/api/core/google/asr?api_key={API_KEY}" # Or use V1 endpoint: WS_URL = f"wss://api.openmind.com/api/core/google/asr/v1?api_key={API_KEY}" # Audio configuration RATE = 16000 CHUNK = 1024 FORMAT = pyaudio.paInt16 CHANNELS = 1 async def stream_audio(): """Stream audio from microphone to Google ASR with V2 features.""" audio = pyaudio.PyAudio() # Open audio stream stream = audio.open( format=FORMAT, channels=CHANNELS, rate=RATE, input=True, frames_per_buffer=CHUNK ) async with websockets.connect(WS_URL) as websocket: # Receive connection confirmation connection_msg = await websocket.recv() print(f"Connected: {connection_msg}") # Send first message with configuration first_audio = stream.read(CHUNK) first_message = { "audio": base64.b64encode(first_audio).decode('utf-8'), "rate": RATE, "language_code": "en-US" } await websocket.send(json.dumps(first_message)) # Start receiving task async def receive_transcriptions(): async for message in websocket: data = json.loads(message) # Handle transcription results if "asr_reply" in data: print(f"Transcript: {data['asr_reply']}") # Handle V2 voice activity events elif data.get("type") == "speech_start": print("🎤 Speech detected") elif data.get("type") == "speech_end": print("🔇 Speech ended") elif data.get("type") == "end_of_utterance": print("✅ Utterance complete") # Handle errors elif data.get("type") == "error": print(f"Error: {data['message']}") receive_task = asyncio.create_task(receive_transcriptions()) # Stream audio try: while True: audio_data = stream.read(CHUNK) message = { "audio": base64.b64encode(audio_data).decode('utf-8') } await websocket.send(json.dumps(message)) await asyncio.sleep(0.01) except KeyboardInterrupt: print("Stopping...") finally: stream.stop_stream() stream.close() audio.terminate() receive_task.cancel() # Run the streaming client asyncio.run(stream_audio()) ``` #### Python Example (V1 - Simple Transcription) ```python import asyncio import websockets import base64 import json import pyaudio API_KEY = "om1_live_your_api_key" WS_URL = f"wss://api.openmind.com/api/core/google/asr/v1?api_key={API_KEY}" # Audio configuration RATE = 16000 CHUNK = 1024 FORMAT = pyaudio.paInt16 CHANNELS = 1 async def stream_audio(): """Stream audio from microphone to Google ASR V1.""" audio = pyaudio.PyAudio() # Open audio stream stream = audio.open( format=FORMAT, channels=CHANNELS, rate=RATE, input=True, frames_per_buffer=CHUNK ) async with websockets.connect(WS_URL) as websocket: # Receive connection confirmation connection_msg = await websocket.recv() print(f"Connected: {connection_msg}") # Send first message with configuration first_audio = stream.read(CHUNK) first_message = { "audio": base64.b64encode(first_audio).decode('utf-8'), "rate": RATE, "language_code": "en-US" } await websocket.send(json.dumps(first_message)) # Start receiving task async def receive_transcriptions(): async for message in websocket: data = json.loads(message) if "asr_reply" in data: print(f"Transcript: {data['asr_reply']}") elif "type" in data and data["type"] == "error": print(f"Error: {data['message']}") receive_task = asyncio.create_task(receive_transcriptions()) # Stream audio try: while True: audio_data = stream.read(CHUNK) message = { "audio": base64.b64encode(audio_data).decode('utf-8') } await websocket.send(json.dumps(message)) await asyncio.sleep(0.01) except KeyboardInterrupt: print("Stopping...") finally: stream.stop_stream() stream.close() audio.terminate() receive_task.cancel() # Run the streaming client asyncio.run(stream_audio()) ``` #### JavaScript/Node.js Example ```javascript const WebSocket = require('ws'); const fs = require('fs'); const API_KEY = 'om1_live_your_api_key'; // V2 endpoint (recommended) - includes voice activity events const WS_URL = `wss://api.openmind.com/api/core/google/asr?api_key=${API_KEY}`; // Or use V1: const WS_URL = `wss://api.openmind.com/api/core/google/asr/v1?api_key=${API_KEY}`; // Connect to WebSocket const ws = new WebSocket(WS_URL); ws.on('open', () => { console.log('Connected to Google ASR'); // Read audio file and send in chunks const audioFile = fs.readFileSync('audio.raw'); // LINEAR16 PCM audio const chunkSize = 4096; let offset = 0; // Send first chunk with configuration const firstChunk = audioFile.slice(0, chunkSize); ws.send(JSON.stringify({ audio: firstChunk.toString('base64'), rate: 16000, language_code: 'en-US' })); offset += chunkSize; // Send remaining chunks const interval = setInterval(() => { if (offset >= audioFile.length) { clearInterval(interval); return; } const chunk = audioFile.slice(offset, offset + chunkSize); ws.send(JSON.stringify({ audio: chunk.toString('base64') })); offset += chunkSize; }, 100); }); ws.on('message', (data) => { const response = JSON.parse(data); if (response.type === 'connection') { console.log(`Client ID: ${response.clientId}`); } else if (response.asr_reply) { console.log(`Transcript: ${response.asr_reply}`); } // V2 voice activity events else if (response.type === 'speech_start') { console.log('🎤 Speech detected'); } else if (response.type === 'speech_end') { console.log('🔇 Speech ended'); } else if (response.type === 'end_of_utterance') { console.log('✅ Utterance complete'); } // Errors else if (response.type === 'error') { console.error(`Error: ${response.message}`); } }); ws.on('error', (error) => { console.error('WebSocket error:', error); }); ws.on('close', () => { console.log('Disconnected from Google ASR'); }); ``` #### Using wscat (Command Line) ```bash # Install wscat npm install -g wscat # Connect to V2 endpoint (recommended - with voice activity events) wscat -c "wss://api.openmind.com/api/core/google/asr?api_key=om1_live_your_api_key" # Connect to V1 endpoint (legacy) wscat -c "wss://api.openmind.com/api/core/google/asr/v1?api_key=om1_live_your_api_key" # Send a message (paste into the terminal after connection) {"audio":"UklGRiQAAABXQVZFZm10IBAAAAABAAEAQB8AAAB9AAACABAAZGF0YQAAAAA=","rate":16000,"language_code":"en-US"} ``` #### Recording Audio for Testing **Using SoX (Sound eXchange):** ```bash # Install SoX # macOS: brew install sox # Ubuntu: sudo apt-get install sox # Record audio in correct format sox -d -r 16000 -c 1 -b 16 -e signed-integer -t raw audio.raw # Or record as WAV sox -d -r 16000 -c 1 -b 16 audio.wav ``` **Using FFmpeg:** ```bash # Convert existing audio to correct format ffmpeg -i input.mp3 -ar 16000 -ac 1 -f s16le audio.raw # Record from microphone ffmpeg -f avfoundation -i ":0" -ar 16000 -ac 1 -f s16le audio.raw ``` ### Language Support The ASR service supports multiple languages. Specify the language code in the first message: | Language | Code | | ----------------------- | -------- | | English (US) | `en-US` | | English (UK) | `en-GB` | | Spanish (Spain) | `es-ES` | | Spanish (Latin America) | `es-419` | | French | `fr-FR` | | German | `de-DE` | | Italian | `it-IT` | | Portuguese (Brazil) | `pt-BR` | | Japanese | `ja-JP` | | Korean | `ko-KR` | | Chinese (Mandarin) | `zh-CN` | > **Note:** For a complete list of supported languages, refer to the [Google Cloud Speech-to-Text documentation](https://cloud.google.com/speech-to-text/docs/languages). ### Error Handling #### Common Errors **Invalid Message Format:** ```json { "type": "error", "message": "Invalid message format: [details]", "clientId": "1738713600000-a1b2c3d4e5f6g7h8" } ``` **Missing Audio Field:** ```json { "type": "error", "message": "Invalid message format: 'audio' field missing", "clientId": "1738713600000-a1b2c3d4e5f6g7h8" } ``` **Audio Decoding Error:** ```json { "type": "error", "message": "Failed to decode audio: [details]", "clientId": "1738713600000-a1b2c3d4e5f6g7h8" } ``` **Speech Recognition Error:** ```json { "type": "error", "message": "Speech recognition error: [details]", "clientId": "1738713600000-a1b2c3d4e5f6g7h8" } ``` #### Handling Connection Loss The WebSocket connection may close due to: * Network interruptions * 4-minute streaming limit reached * Client disconnect * Server errors Implement reconnection logic in your client: ```python async def connect_with_retry(max_retries=3): for attempt in range(max_retries): try: async with websockets.connect(WS_URL) as websocket: await stream_audio(websocket) except Exception as e: print(f"Connection attempt {attempt + 1} failed: {e}") if attempt < max_retries - 1: await asyncio.sleep(2 ** attempt) # Exponential backoff else: raise ``` ### Session Management #### Streaming Limit Each recognition session has a maximum duration of **4 minutes (240 seconds)** for both V1 and V2. After this time: * The current stream will automatically restart * A new recognition session will begin * Audio processing continues seamlessly * V2 users may receive a `"type": "info"` message indicating session restart #### Session Cleanup When the WebSocket connection closes: * All buffered audio is processed * Final transcriptions are sent * Usage tracking is recorded * Resources are cleaned up #### Client Identification Each connection receives a unique `clientId` in the format: ``` {timestamp}-{random_hex} ``` This ID is included in all server responses for tracking and debugging purposes. ### Cost Calculation Speech recognition costs are calculated based on the total audio duration processed: ``` cost_in_omcu = audio_duration_seconds × per_second_rate ``` Usage is tracked and billed to the API key provided in the connection URL. > **Note:** Note the following about cost calculation: > > * Audio length is calculated automatically from the data sent > * Only successfully processed audio is billed > * Usage details are available in your OpenMind dashboard ### Best Practices #### Audio Quality * Use high-quality audio input (clear speech, minimal background noise) * Maintain consistent audio levels * Use the recommended 16000 Hz sample rate for optimal recognition * Send audio in consistent chunk sizes (1024-4096 bytes recommended) #### Network Optimization * Implement exponential backoff for reconnection attempts * Buffer audio locally during temporary connection issues * Monitor WebSocket connection health * Handle network interruptions gracefully #### Error Handling * Always validate the API key before establishing connections * Check for error messages in server responses * Implement retry logic for transient failures * Log client IDs for debugging and support requests #### Performance Tips * Send audio chunks at regular intervals (every 50-100ms) * Avoid sending very large or very small chunks * Don't accumulate audio before sending - stream in real-time * Process transcription results asynchronously #### Security * Never hardcode API keys in client-side code * Use environment variables for API key storage * Rotate API keys regularly * Monitor API key usage for suspicious activity ### Troubleshooting #### No Transcription Results * Verify audio format is LINEAR16 PCM * Check sample rate matches the `rate` parameter * Ensure audio contains clear speech * Verify language code matches the spoken language #### Connection Issues * Confirm API key is valid and active * Check WebSocket support in your environment * Verify network allows WebSocket connections * Test connection with wscat first #### Poor Recognition Quality * Increase audio quality/bitrate * Reduce background noise * Speak clearly and at normal pace * Try adjusting the language model if available #### Buffer Full Warnings If you see "Audio stream buffer full" in logs: * Reduce the rate of audio sending * Increase chunk send interval * Check for network congestion * Verify client is reading responses ### Example: Complete Integration Here's a complete example integrating microphone input, WebSocket streaming, and real-time display with V2 voice activity events: ```python import asyncio import websockets import json import base64 import pyaudio from typing import Callable class GoogleASRClient: """Complete Google ASR WebSocket client with V2 voice activity support.""" def __init__(self, api_key: str, language: str = "en-US", use_v2: bool = True): self.api_key = api_key self.language = language # Choose endpoint version if use_v2: self.ws_url = f"wss://api.openmind.com/api/core/google/asr?api_key={api_key}" print("Using V2 endpoint with Chirp 3 model and voice activity detection") else: self.ws_url = f"wss://api.openmind.com/api/core/google/asr/v1?api_key={api_key}" print("Using V1 endpoint with standard model") self.use_v2 = use_v2 self.client_id = None # Audio config self.rate = 16000 self.chunk = 1024 self.format = pyaudio.paInt16 self.channels = 1 self.transcript_callback = None self.speech_start_callback = None self.speech_end_callback = None self.utterance_end_callback = None def on_transcript(self, callback: Callable[[str], None]): """Register callback for transcription results.""" self.transcript_callback = callback return self def on_speech_start(self, callback: Callable[[], None]): """Register callback for speech start events (V2 only).""" self.speech_start_callback = callback return self def on_speech_end(self, callback: Callable[[], None]): """Register callback for speech end events (V2 only).""" self.speech_end_callback = callback return self def on_utterance_end(self, callback: Callable[[], None]): """Register callback for end of utterance events (V2 only).""" self.utterance_end_callback = callback return self async def start(self): """Start streaming audio and receiving transcriptions.""" audio = pyaudio.PyAudio() stream = audio.open( format=self.format, channels=self.channels, rate=self.rate, input=True, frames_per_buffer=self.chunk ) try: async with websockets.connect(self.ws_url) as ws: # Handle connection conn_msg = json.loads(await ws.recv()) self.client_id = conn_msg.get('clientId') print(f"Connected with ID: {self.client_id}") print(f"Service: {conn_msg.get('message')}") # Send first message with config first_audio = stream.read(self.chunk) await ws.send(json.dumps({ "audio": base64.b64encode(first_audio).decode(), "rate": self.rate, "language_code": self.language })) # Create tasks for sending and receiving send_task = asyncio.create_task(self._send_audio(ws, stream)) recv_task = asyncio.create_task(self._receive_transcripts(ws)) # Wait for tasks await asyncio.gather(send_task, recv_task) finally: stream.stop_stream() stream.close() audio.terminate() async def _send_audio(self, ws, stream): """Send audio chunks to the WebSocket.""" try: while True: audio_data = stream.read(self.chunk, exception_on_overflow=False) message = { "audio": base64.b64encode(audio_data).decode() } await ws.send(json.dumps(message)) await asyncio.sleep(0.05) # 50ms between chunks except Exception as e: print(f"Send error: {e}") async def _receive_transcripts(self, ws): """Receive and process transcription results.""" try: async for message in ws: data = json.loads(message) # Transcription result if "asr_reply" in data and self.transcript_callback: self.transcript_callback(data["asr_reply"]) # V2 voice activity events elif self.use_v2 and data.get("type") == "speech_start": if self.speech_start_callback: self.speech_start_callback() elif self.use_v2 and data.get("type") == "speech_end": if self.speech_end_callback: self.speech_end_callback() elif self.use_v2 and data.get("type") == "end_of_utterance": if self.utterance_end_callback: self.utterance_end_callback() # Errors elif data.get("type") == "error": print(f"Error: {data.get('message')}") except Exception as e: print(f"Receive error: {e}") # Usage Example async def main(): # Use V2 with voice activity events client = GoogleASRClient( api_key="om1_live_your_api_key", language="en-US", use_v2=True # Set to False for V1 ) # Register callbacks client.on_transcript(lambda text: print(f"📝 Transcript: {text}")) # V2-specific callbacks if client.use_v2: client.on_speech_start(lambda: print("🎤 Speech started")) client.on_speech_end(lambda: print("🔇 Speech ended")) client.on_utterance_end(lambda: print("✅ Utterance complete")) # Start streaming print("Starting ASR stream... Press Ctrl+C to stop") await client.start() if __name__ == "__main__": asyncio.run(main()) ``` ### Additional Resources * [Google Cloud Speech-to-Text Documentation](https://cloud.google.com/speech-to-text/docs) * [Google Cloud Speech-to-Text v2 Documentation](https://cloud.google.com/speech-to-text/v2/docs) * [Chirp 3 Model Overview](https://docs.cloud.google.com/speech-to-text/docs/models/chirp-3) * [Supported Languages](https://cloud.google.com/speech-to-text/docs/languages) * [Audio Encoding Best Practices](https://cloud.google.com/speech-to-text/docs/encoding) # ElevenLabs ASR ElevenLabs Automatic Speech Recognition (ASR) API Reference The ElevenLabs ASR API provides real-time speech-to-text transcription using ElevenLabs' Scribe v2 model. This WebSocket-based endpoint enables low-latency streaming recognition for live audio processing with voice activity detection and partial transcript delivery. **Base URL:** `wss://api.openmind.com` **Authentication:** Requires an OpenMind API key passed as a query parameter. ### Endpoint Overview | Protocol | Endpoint | Description | | --------- | -------------------------- | ------------------------------------------------------ | | WebSocket | `/api/core/elevenlabs/asr` | Real-time speech recognition with ElevenLabs Scribe v2 | > **Note:** The endpoint also supports the `/api/core/v1/` prefix for API versioning (e.g., `/api/core/v1/elevenlabs/asr`). ### WebSocket Connection Establish a persistent WebSocket connection for streaming audio data and receiving real-time transcription results. **Endpoint:** `wss://api.openmind.com/api/core/elevenlabs/asr?api_key=YOUR_API_KEY` #### Connection Parameters | Parameter | Type | Required | Description | | --------- | ------ | -------- | ---------------------------------------- | | `api_key` | string | Yes | Your OpenMind API key for authentication | #### Connection Example ```bash # Using wscat (install with: npm install -g wscat) wscat -c "wss://api.openmind.com/api/core/elevenlabs/asr?api_key=om1_live_your_api_key" ``` #### Connection Response Upon successful connection, you'll receive a confirmation message: ```json { "type": "connection", "message": "Connected to ElevenLabs ASR service", "clientId": "1738713600000-a1b2c3d4e5f6g7h8" } ``` #### Connection Errors **401 Unauthorized - Missing API Key:** ```json { "error": "Missing API key. Please connect with ?api_key=YOUR_API_KEY" } ``` **401 Unauthorized - Invalid API Key:** ```json { "error": "Invalid API key: [error details]" } ``` ### Sending Audio Data #### Message Format Send audio data as JSON messages over the WebSocket connection: ```json { "audio": "base64_encoded_audio_data", "rate": 16000, "language_code": "en" } ``` #### Message Fields | Field | Type | Required | Default | Description | | --------------- | ------- | -------- | -------- | ---------------------------------------------------------------------------------------------------------- | | `audio` | string | Yes | - | Base64-encoded raw PCM audio data | | `rate` | integer | No | `16000` | Audio sample rate in Hz. Supported values: `8000`, `16000`, `22050`, `44100` | | `language_code` | string | No | `"auto"` | BCP-47 language code (e.g., `"en"`, `"es"`, `"fr"`). Use `"auto"` or omit for automatic language detection | > **Note:** The `rate` and `language_code` parameters only need to be sent with the first message. Subsequent messages can contain only the `audio` field. #### Audio Format Mapping The sample rate determines the PCM format sent to ElevenLabs: | Sample Rate (Hz) | PCM Format | | ---------------- | --------------------- | | 8000 | `pcm_8000` | | 16000 | `pcm_16000` (default) | | 22050 | `pcm_22050` | | 44100 | `pcm_44100` | > **Recommendation:** Use 16000 Hz for the best balance of quality and bandwidth. ### Receiving Transcription Results The service delivers two types of transcription events as results become available. #### Partial Transcript Intermediate, in-progress transcription result emitted as the user speaks: ```json { "type": "partial", "asr_reply": "hello wor", "clientId": "1738713600000-a1b2c3d4e5f6g7h8", "time": 1738713600123 } ``` #### Committed Transcript Final, committed transcription result for a completed utterance: ```json { "asr_reply": "hello world", "clientId": "1738713600000-a1b2c3d4e5f6g7h8", "time": 1738713600456 } ``` #### Response Fields | Field | Type | Description | | ----------- | ------- | ------------------------------------------------------------------------------------------------ | | `asr_reply` | string | Transcription text (partial or final) | | `clientId` | string | Unique identifier for the WebSocket session | | `type` | string | Message type: `"connection"`, `"partial"`, `"error"`, `"info"` (absent on committed transcripts) | | `message` | string | Human-readable message for connection, info, or error events | | `time` | integer | Unix timestamp in milliseconds when the result was produced | #### Info Messages The server may send informational messages during operation, such as when a recognition session is automatically restarted: ```json { "type": "info", "message": "Recognition session restarted", "clientId": "1738713600000-a1b2c3d4e5f6g7h8" } ``` #### Error Messages ```json { "type": "error", "message": "Speech recognition error: [details]", "clientId": "1738713600000-a1b2c3d4e5f6g7h8" } ``` ### Session Limits | Limit | Value | Description | | ---------------------- | ------------ | ------------------------------------------------------------------------------------------- | | Max streaming duration | 5 minutes | Each internal ElevenLabs session is capped at 5 minutes; the session automatically restarts | | Silence timeout | Configurable | The connection closes after a period of silence with no detected speech | When the 5-minute streaming limit is reached, the session is seamlessly restarted and an `"info"` message is sent to the client. Non-recoverable errors will close the WebSocket. ### Audio Specifications #### Supported Audio Format * **Encoding:** Raw PCM (signed 16-bit little-endian) * **Sample Rate:** 8000, 16000 (recommended), 22050, or 44100 Hz * **Channels:** Mono (1 channel) #### Calculating Audio Length Audio duration is calculated as: $$\text{duration (s)} = \frac{\text{audio bytes}}{\text{sample rate} \times 2 \times 1}$$ For 16000 Hz mono 16-bit PCM: $$\text{duration (s)} = \frac{\text{audio bytes}}{32000}$$ ### Usage Examples #### Python Example ```python import asyncio import websockets import base64 import json import pyaudio API_KEY = "om1_live_your_api_key" WS_URL = f"wss://api.openmind.com/api/core/elevenlabs/asr?api_key={API_KEY}" RATE = 16000 CHUNK = 1024 FORMAT = pyaudio.paInt16 CHANNELS = 1 async def stream_audio(): audio = pyaudio.PyAudio() stream = audio.open( format=FORMAT, channels=CHANNELS, rate=RATE, input=True, frames_per_buffer=CHUNK ) async with websockets.connect(WS_URL) as websocket: connection_msg = await websocket.recv() print(f"Connected: {connection_msg}") first_audio = stream.read(CHUNK) await websocket.send(json.dumps({ "audio": base64.b64encode(first_audio).decode("utf-8"), "rate": RATE, "language_code": "en" })) async def receive(): async for message in websocket: data = json.loads(message) if data.get("type") == "partial": print(f"[partial] {data.get('asr_reply', '')}", end="\r") elif "asr_reply" in data and data.get("type") != "partial": print(f"\n[final] {data['asr_reply']}") elif data.get("type") == "error": print(f"\n[error] {data.get('message')}") async def send(): while True: chunk = stream.read(CHUNK, exception_on_overflow=False) await websocket.send(json.dumps({ "audio": base64.b64encode(chunk).decode("utf-8") })) await asyncio.gather(receive(), send()) asyncio.run(stream_audio()) ``` #### JavaScript/Node.js Example ```javascript const WebSocket = require('ws'); const fs = require('fs'); const API_KEY = 'om1_live_your_api_key'; const WS_URL = `wss://api.openmind.com/api/core/elevenlabs/asr?api_key=${API_KEY}`; const ws = new WebSocket(WS_URL); ws.on('open', () => { console.log('Connected to ElevenLabs ASR'); const audioFile = fs.readFileSync('audio.raw'); // raw 16-bit PCM const chunkSize = 4096; let offset = 0; ws.send(JSON.stringify({ audio: audioFile.slice(0, chunkSize).toString('base64'), rate: 16000, language_code: 'en' })); offset += chunkSize; const interval = setInterval(() => { if (offset >= audioFile.length) { clearInterval(interval); return; } ws.send(JSON.stringify({ audio: audioFile.slice(offset, offset + chunkSize).toString('base64') })); offset += chunkSize; }, 100); }); ws.on('message', (data) => { const response = JSON.parse(data); if (response.type === 'connection') { console.log(`Client ID: ${response.clientId}`); } else if (response.type === 'partial') { process.stdout.write(`\r[partial] ${response.asr_reply}`); } else if (response.asr_reply) { console.log(`\n[final] ${response.asr_reply}`); } else if (response.type === 'error') { console.error(`[error] ${response.message}`); } }); ws.on('error', (err) => console.error('WebSocket error:', err)); ws.on('close', () => console.log('Disconnected')); ``` #### Using wscat (Command Line) ```bash # Install wscat npm install -g wscat # Connect wscat -c "wss://api.openmind.com/api/core/elevenlabs/asr?api_key=om1_live_your_api_key" # Send first message (paste after connection is established) {"audio":"","rate":16000,"language_code":"en"} ``` #### Recording Audio for Testing **Using SoX:** ```bash # macOS: brew install sox # Ubuntu: sudo apt-get install sox # Record raw PCM at 16 kHz mono sox -d -r 16000 -c 1 -b 16 -e signed-integer -t raw audio.raw ``` **Using FFmpeg:** ```bash # Convert existing audio file to correct format ffmpeg -i input.mp3 -ar 16000 -ac 1 -f s16le audio.raw # Record from microphone (macOS) ffmpeg -f avfoundation -i ":0" -ar 16000 -ac 1 -f s16le audio.raw ``` ### Language Support Pass a BCP-47 language code in the first message to pin recognition to a specific language. Omit the field or use `"auto"` to let ElevenLabs detect the language automatically. | Language | Code | | ------------------ | ---- | | English | `en` | | Spanish | `es` | | French | `fr` | | German | `de` | | Italian | `it` | | Portuguese | `pt` | | Japanese | `ja` | | Korean | `ko` | | Chinese (Mandarin) | `zh` | | Dutch | `nl` | | Polish | `pl` | | Russian | `ru` | > **Note:** For a complete list of supported languages, refer to the [ElevenLabs Speech-to-Text documentation](https://elevenlabs.io/docs/speech-to-text). # ElevenLabs TTS ElevenLabs Text to Speech (TTS) The ElevenLabs TTS API converts text into natural-sounding speech using ElevenLabs' advanced text-to-speech models. This endpoint provides high-quality voice synthesis with customizable voice selection, speech speed, and output formats. **Base URL:** `https://api.openmind.com` **Authentication:** OpenMind API key is required. Include the key in the `x-api-key` or `Authorization` header. ### Endpoints Overview | Method | Endpoint | Description | | ------ | ------------------------------ | ---------------------------------------------- | | POST | `/elevenlabs/tts` | Generate speech from text using ElevenLabs TTS | | POST | `/elevenlabs/tts/audio/speech` | Stream speech from text using ElevenLabs TTS | ### Generate Speech Convert text to speech using the ElevenLabs TTS engine with customizable voice and output options. **Endpoint:** `POST /elevenlabs/tts` #### Request ```bash curl -X POST https://api.openmind.com/elevenlabs/tts \ -H "Content-Type: application/json" \ -H "x-api-key: YOUR_API_KEY" \ -d '{ "input": "Hello, this is a test of the ElevenLabs text to speech API." }' ``` #### Request Body | Field | Type | Required | Default | Description | | -------------------- | ---------------- | -------- | ---------------------- | -------------------------------------------- | | `input` | string | Yes | - | The text to convert to speech | | `voice` | string or object | No | `JBFqnCBsd6RMkjVDRZzb` | ElevenLabs voice ID (string) or voice object | | `model` | string | No | `eleven_flash_v2_5` | ElevenLabs model ID to use for synthesis | | `response_format` | string | No | `mp3_44100_128` | Audio output format specification | | `speed` | float | No | `1.0` | Speech speed multiplier (0.5 - 2.0) | | `elevenlabs_api_key` | string | No | - | Optional ElevenLabs API key override | #### Response **Success (200 OK):** ```json { "text": "Hello, this is a test of the ElevenLabs text to speech API.", "response": "SUQzBAAAAAAAI1RTU0UAAAAPAAADTGF2ZjU4Ljc2LjEwMAAAAAAAAAAAAAAA//tQAAAAAAAAAAAA...", "format": "mp3_44100_128" } ``` #### Response Fields | Field | Type | Description | | ---------- | ------ | ----------------------------------------------------------- | | `text` | string | The original input text | | `response` | string | Base64-encoded audio data ready for decoding and playback | | `format` | string | Audio format of the returned data (e.g., "mp3\_44100\_128") | **Error Responses:** ```json // 400 Bad Request - Missing or invalid input { "error": "Missing or invalid JSON in request" } // 503 Service Unavailable - API key not configured { "error": "ElevenLabs API key not configured" } // 503 Service Unavailable - Connection failure { "error": "Failed to connect to ElevenLabs server" } // 500 Internal Server Error { "error": "Failed to read response body" } ``` > **Note:** The returned audio is base64-encoded. You must decode it before playback or saving to a file. ### Stream Speech Convert text to speech and stream the audio directly. This endpoint is ideal for real-time applications where low latency is critical. **Endpoint:** `POST /elevenlabs/tts/audio/speech` #### Request The request body parameters are identical to the `/elevenlabs/tts` endpoint. #### Response **Success (200 OK):** The response is a binary stream of the audio file. **Headers:** * `Content-Type`: `audio/mpeg` (depending on requested format) **Error Responses:** See Error Responses for `/elevenlabs/tts`. ### Usage Examples #### Basic Text-to-Speech Convert simple text to speech using default settings: ```bash curl -X POST https://api.openmind.com/elevenlabs/tts \ -H "Content-Type: application/json" \ -H "x-api-key: YOUR_API_KEY" \ -d '{ "input": "Welcome to OpenMind AGI. This is a demonstration of text to speech conversion." }' ``` #### Custom Voice and Speed Use a specific voice with faster speech rate: ```bash curl -X POST https://api.openmind.com/elevenlabs/tts \ -H "Content-Type: application/json" \ -H "x-api-key: YOUR_API_KEY" \ -d '{ "input": "This speech is faster than normal and uses a custom voice.", "voice": "JBFqnCBsd6RMkjVDRZzb", "speed": 1.3 }' ``` #### Full Configuration Customize all available parameters: ```bash curl -X POST https://api.openmind.com/elevenlabs/tts \ -H "Content-Type: application/json" \ -H "x-api-key: YOUR_API_KEY" \ -d '{ "input": "Fully customized text to speech with all parameters specified.", "voice": "your_voice_id", "model": "eleven_flash_v2_5", "response_format": "mp3_44100_128", "speed": 0.9, "elevenlabs_api_key": "your_elevenlabs_api_key" }' ``` #### Save Audio to File Generate speech and save directly to an MP3 file: ```bash curl -X POST https://api.openmind.com/elevenlabs/tts \ -H "Content-Type: application/json" \ -H "x-api-key: YOUR_API_KEY" \ -d '{ "input": "This audio will be saved to a file on your local machine." }' | jq -r '.response' | base64 -d > output.mp3 ``` #### With Environment Variables Store your configuration in environment variables for easier management: ```bash # Set environment variables export TTS_VOICE_ID="JBFqnCBsd6RMkjVDRZzb" export TTS_SPEED="1.1" # Use in request curl -X POST https://api.openmind.com/elevenlabs/tts \ -H "Content-Type: application/json" \ -H "x-api-key: YOUR_API_KEY" \ -d "{ \"input\": \"Using environment variables for configuration.\", \"voice\": \"$TTS_VOICE_ID\", \"speed\": $TTS_SPEED }" ``` #### Stream to File Stream the audio directly to a file using the streaming endpoint: ```bash curl -X POST https://api.openmind.com/elevenlabs/tts/audio/speech \ -H "Content-Type: application/json" \ -H "x-api-key: YOUR_API_KEY" \ -d '{ "input": "This is a streaming response." }' > stream_output.mp3 ``` ### Voice Configuration #### Default Voice The default voice ID is `JBFqnCBsd6RMkjVDRZzb`. This voice provides clear, natural-sounding English speech suitable for most applications. #### Custom Voices You can use any ElevenLabs voice ID by specifying it in the `voice` parameter. Visit the [ElevenLabs Voice Library](https://elevenlabs.io/voice-library) to explore available voices. #### Speed Control The `speed` parameter accepts values between 0.5 (half speed) and 2.0 (double speed): * `0.5` - 50% slower (more deliberate) * `1.0` - Normal speed (default) * `1.5` - 50% faster * `2.0` - Double speed (maximum) ### Output Formats The default output format is `mp3_44100_128`. The `response_format` parameter allows you to specify other formats if needed. ### Error Handling All endpoints follow consistent error response patterns: #### HTTP Status Codes | Code | Description | | ---- | -------------------------------------------------------------------------- | | 200 | Success - Audio generated successfully | | 400 | Bad Request - Missing required fields, invalid JSON, or unsupported format | | 503 | Service Unavailable - ElevenLabs API unavailable or not configured | | 500 | Internal Server Error - Server-side processing error | #### Error Response Format ```json { "error": "Descriptive error message" } ``` #### Common Error Scenarios **Missing Input Field:** ```bash # This will fail - input is required curl -X POST https://api.openmind.com/elevenlabs/tts \ -H "Content-Type: application/json" \ -H "x-api-key: YOUR_API_KEY" \ -d '{}' # Response: {"error": "Missing or invalid JSON in request"} ``` **API Key Not Configured:** If the server-side ElevenLabs API key is not configured and you don't provide one in the request, you'll receive: ```json { "error": "ElevenLabs API key not configured" } ``` **Connection Issues:** If the service cannot reach the ElevenLabs API: ```json { "error": "Failed to connect to ElevenLabs server", "details": "additional error information" } ``` ### Best Practices #### Audio Decoding The API returns base64-encoded audio data. Always decode it before use: ```bash # Decode and save to file echo "SUQzBAAAAAAAI1RTU0UAAAA..." | base64 -d > audio.mp3 # Or use jq to extract from JSON response curl ... | jq -r '.response' | base64 -d > audio.mp3 ``` > **Note:** Note the following best practices when using the ElevenLabs TTS API: > > * Audio responses are base64-encoded and must be decoded before playback > * The ElevenLabs API key can be configured server-side or provided per-request > * Default voice and model settings are optimized for English speech > * Large text inputs may take longer to process # LLM Multi-Provider Large Language Model API The OpenMind LLM API provides unified access to multiple leading large language model providers through a single, consistent interface. This endpoint enables chat completions across OpenAI, Anthropic (via OpenRouter), Google Gemini, X.AI, DeepSeek, NEAR.AI, and more. **Base URL:** `https://api.openmind.com` **Authentication:** Requires an OpenMind API key in the Authorization header as a Bearer token. ### Endpoints Overview | Method | Endpoint | Description | | ------ | --------------------------------------- | ----------------------------------------------------------- | | POST | `/api/core/{provider}/chat/completions` | Send chat completion requests to the specified LLM provider | ### Supported Providers OpenMind supports the following LLM providers: | Provider | Endpoint Path | Description | | ------------- | ------------- | ------------------------------------------------------------ | | OpenAI | `openai` | GPT-4, GPT-5, and other OpenAI models | | DeepSeek | `deepseek` | DeepSeek chat models | | Google Gemini | `gemini` | Gemini Pro and Flash models | | X.AI | `xai` | Grok models from X.AI | | NEAR.AI | `nearai` | Qwen and other NEAR.AI hosted models | | OpenRouter | `openrouter` | Multi-provider access including Anthropic Claude, Meta Llama | ### Supported Models #### OpenAI Models ``` gpt-4o gpt-4o-mini gpt-4.1 gpt-4.1-mini gpt-4.1-nano gpt-5 gpt-5-mini gpt-5-nano ``` #### DeepSeek Models ``` deepseek-chat ``` #### Google Gemini Models ``` gemini-3.5-flash gemini-3.1-pro-preview gemini-3.1-flash-lite gemini-2.5-flash gemini-2.5-flash-lite gemini-2.5-pro ``` #### X.AI Models ``` grok-2-latest grok-3-beta grok-4-latest grok-4 ``` #### NEAR.AI Models ``` qwen3-30b-a3b-instruct-2507 qwen2.5-vl-72b-instruct qwen-2.5-7b-instruct ``` #### OpenRouter Models ``` meta-llama/llama-3.1-70b-instruct meta-llama/llama-3.3-70b-instruct anthropic/claude-sonnet-4.5 anthropic/claude-opus-4.1 ``` > **Note:** Model names are validated using prefix matching. For example, "gpt-4o" will match "gpt-4o", "gpt-4o-2024-08-06", etc. ### Chat Completions Send a chat completion request to any supported LLM provider. **Endpoint:** `POST /api/core/{provider}/chat/completions` #### Path Parameters | Parameter | Type | Required | Description | | ---------- | ------ | -------- | -------------------------------------------------------------- | | `provider` | string | Yes | The LLM provider name (e.g., "openai", "gemini", "openrouter") | #### Request Headers | Header | Required | Description | | --------------- | -------- | --------------------------------------- | | `Authorization` | Yes | Bearer token with your OpenMind API key | | `Content-Type` | Yes | Must be `application/json` | | `Accept` | No | Recommended: `application/json` | #### Request Body The request body follows the OpenAI Chat Completions API format: | Field | Type | Required | Description | | ------------------- | ------- | -------- | --------------------------------------------------------------- | | `model` | string | Yes | Model identifier (must match supported models for the provider) | | `messages` | array | Yes | Array of message objects with `role` and `content` | | `temperature` | float | No | Sampling temperature (0.0 to 2.0) | | `max_tokens` | integer | No | Maximum tokens to generate | | `top_p` | float | No | Nucleus sampling parameter | | `stream` | boolean | No | Whether to stream responses | | `frequency_penalty` | float | No | Frequency penalty (-2.0 to 2.0) | | `presence_penalty` | float | No | Presence penalty (-2.0 to 2.0) | **Message Format** ```json { "role": "user|assistant|system", "content": "Message content" } ``` #### Basic Request Example ```bash curl --location 'https://api.openmind.com/api/core/openrouter/chat/completions' \ --header 'Accept: application/json' \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer ' \ --data '{ "model": "anthropic/claude-sonnet-4.5", "messages": [ { "role": "user", "content": "Hello, how are you?" } ] }' ``` #### Response **Success (200 OK):** ```json { "id": "chatcmpl-abc123", "object": "chat.completion", "created": 1738713600, "model": "gpt-4o", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "Hello! I'm doing well, thank you for asking. How can I assist you today?" }, "finish_reason": "stop" } ], "usage": { "prompt_tokens": 12, "completion_tokens": 18, "total_tokens": 30 } } ``` #### Response Fields | Field | Type | Description | | ------------------------- | ------- | ---------------------------------------------- | | `id` | string | Unique identifier for the completion | | `object` | string | Object type, always "chat.completion" | | `created` | integer | Unix timestamp of creation | | `model` | string | Model used for the completion | | `choices` | array | Array of completion choices | | `choices[].message` | object | Generated message with role and content | | `choices[].finish_reason` | string | Reason for completion ("stop", "length", etc.) | | `usage` | object | Token usage statistics | **Error Responses:** ```json // 400 Bad Request - Invalid JSON { "error": "Invalid JSON" } // 404 Not Found - Unsupported provider or model { "error": "unsupported model provider: invalid_provider" } { "error": "unsupported model: gpt-6. Supported model prefixes for openai: [gpt-4o, gpt-4o-mini, ...]" } // 503 Service Unavailable - API key not configured { "error": "openai API key not configured" } // 503 Service Unavailable - Provider connection failed { "error": "Failed to connect to openai server" } ``` ### Usage Examples #### OpenAI GPT-4 ```bash curl -X POST https://api.openmind.com/api/core/openai/chat/completions \ -H "Authorization: Bearer om1_live_your_api_key" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-4o", "messages": [ { "role": "system", "content": "You are a helpful robotics assistant." }, { "role": "user", "content": "Explain how to implement SLAM for a mobile robot." } ], "temperature": 0.7, "max_tokens": 500 }' ``` #### Anthropic Claude (via OpenRouter) ```bash curl -X POST https://api.openmind.com/api/core/openrouter/chat/completions \ -H "Authorization: Bearer om1_live_your_api_key" \ -H "Content-Type: application/json" \ -d '{ "model": "anthropic/claude-sonnet-4.5", "messages": [ { "role": "user", "content": "What are the latest advancements in computer vision for robotics?" } ] }' ``` #### Google Gemini ```bash curl -X POST https://api.openmind.com/api/core/gemini/chat/completions \ -H "Authorization: Bearer om1_live_your_api_key" \ -H "Content-Type: application/json" \ -d '{ "model": "gemini-2.5-pro", "messages": [ { "role": "user", "content": "Describe the differences between reinforcement learning and supervised learning." } ], "temperature": 0.5 }' ``` #### DeepSeek ```bash curl -X POST https://api.openmind.com/api/core/deepseek/chat/completions \ -H "Authorization: Bearer om1_live_your_api_key" \ -H "Content-Type: application/json" \ -d '{ "model": "deepseek-chat", "messages": [ { "role": "user", "content": "Write a Python function to calculate inverse kinematics for a 6-DOF robot arm." } ] }' ``` #### X.AI Grok ```bash curl -X POST https://api.openmind.com/api/core/xai/chat/completions \ -H "Authorization: Bearer om1_live_your_api_key" \ -H "Content-Type: application/json" \ -d '{ "model": "grok-4-latest", "messages": [ { "role": "user", "content": "Explain quantum computing in simple terms." } ] }' ``` #### NEAR.AI Qwen ```bash curl -X POST https://api.openmind.com/api/core/nearai/chat/completions \ -H "Authorization: Bearer om1_live_your_api_key" \ -H "Content-Type: application/json" \ -d '{ "model": "qwen2.5-vl-72b-instruct", "messages": [ { "role": "user", "content": "What are the key components of a neural network?" } ] }' ``` #### Multi-Turn Conversation ```bash curl -X POST https://api.openmind.com/api/core/openai/chat/completions \ -H "Authorization: Bearer om1_live_your_api_key" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-4o-mini", "messages": [ { "role": "system", "content": "You are a concise and helpful AI assistant." }, { "role": "user", "content": "What is the capital of France?" }, { "role": "assistant", "content": "The capital of France is Paris." }, { "role": "user", "content": "What is its population?" } ] }' ``` #### With Environment Variables ```bash # Set your API key as an environment variable export OPENMIND_API_KEY="om1_live_your_api_key" # Use in requests curl -X POST https://api.openmind.com/api/core/openai/chat/completions \ -H "Authorization: Bearer $OPENMIND_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-5-mini", "messages": [ { "role": "user", "content": "Hello!" } ] }' ``` ### Advanced Parameters #### Temperature Control Control randomness in responses (0.0 = deterministic, 2.0 = very random): ```bash curl -X POST https://api.openmind.com/api/core/openai/chat/completions \ -H "Authorization: Bearer om1_live_your_api_key" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-4o", "messages": [{"role": "user", "content": "Tell me a creative story."}], "temperature": 1.2 }' ``` #### Token Limits Limit the maximum number of tokens in the response: ```bash curl -X POST https://api.openmind.com/api/core/openai/chat/completions \ -H "Authorization: Bearer om1_live_your_api_key" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-4o", "messages": [{"role": "user", "content": "Explain machine learning."}], "max_tokens": 100 }' ``` #### Top-P Sampling Use nucleus sampling for controlled randomness: ```bash curl -X POST https://api.openmind.com/api/core/openai/chat/completions \ -H "Authorization: Bearer om1_live_your_api_key" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-4o", "messages": [{"role": "user", "content": "Generate code ideas."}], "top_p": 0.9 }' ``` #### Frequency and Presence Penalties Reduce repetition in responses: ```bash curl -X POST https://api.openmind.com/api/core/openai/chat/completions \ -H "Authorization: Bearer om1_live_your_api_key" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-4o", "messages": [{"role": "user", "content": "Write a unique poem."}], "frequency_penalty": 0.5, "presence_penalty": 0.5 }' ``` ### Model Selection Guide #### When to Use Each Provider **OpenAI (GPT-4, GPT-5):** * General-purpose tasks * Complex reasoning * Code generation * Creative writing **Anthropic Claude (via OpenRouter):** * Long context understanding * Detailed analysis * Safety-critical applications * Nuanced conversations **Google Gemini:** * Multimodal capabilities * Fast inference (Flash models) * Cost-effective solutions * Real-time applications **X.AI Grok:** * Real-time information * Current events * Conversational AI * Research tasks **DeepSeek:** * Code-focused tasks * Technical documentation * Algorithm design * Cost-efficient reasoning **NEAR.AI (Qwen):** * Vision-language tasks * Multilingual support * Open-source model access * Specialized applications #### Performance vs. Cost | Model Tier | Examples | Use Case | | ---------------- | ------------------------------------------------- | ------------------------------------------ | | High Performance | gpt-5, claude-opus-4.1, gemini-3.1-pro-preview | Complex reasoning, production applications | | Balanced | gpt-4o, claude-sonnet-4.5, grok-4 | General-purpose, most tasks | | Fast/Economical | gpt-4o-mini, gemini-2.5-flash-lite, deepseek-chat | High-volume, simple tasks | ### Error Handling #### HTTP Status Codes | Code | Description | | ---- | ---------------------------------------------------------------- | | 200 | Success - Completion generated successfully | | 400 | Bad Request - Invalid JSON or malformed request | | 404 | Not Found - Unsupported provider or model | | 503 | Service Unavailable - Provider API unavailable or not configured | | 500 | Internal Server Error - Server-side processing error | #### Error Response Format All errors follow this format: ```json { "error": "Descriptive error message" } ``` #### Common Errors **Invalid Provider:** ```bash # Request to unsupported provider curl -X POST https://api.openmind.com/api/core/invalid/chat/completions \ -H "Authorization: Bearer om1_live_your_api_key" \ -H "Content-Type: application/json" \ -d '{"model": "test", "messages": [{"role": "user", "content": "Hi"}]}' # Response: {"error": "unsupported model provider: invalid"} ``` **Invalid Model:** ```bash # Request with unsupported model curl -X POST https://api.openmind.com/api/core/openai/chat/completions \ -H "Authorization: Bearer om1_live_your_api_key" \ -H "Content-Type: application/json" \ -d '{"model": "gpt-999", "messages": [{"role": "user", "content": "Hi"}]}' # Response: {"error": "unsupported model: gpt-999. Supported model prefixes for openai: [...]"} ``` **Missing API Key:** ```bash # Request without authentication curl -X POST https://api.openmind.com/api/core/openai/chat/completions \ -H "Content-Type: application/json" \ -d '{"model": "gpt-4o", "messages": [{"role": "user", "content": "Hi"}]}' # Response: 401 Unauthorized ``` ### Best Practices #### API Key Management * Store API keys in environment variables, never in code * Rotate API keys regularly * Use separate keys for development and production * Monitor key usage through the OpenMind portal #### Request Optimization **Efficient Message Design:** ```json { "model": "gpt-4o-mini", "messages": [ { "role": "system", "content": "Be concise and direct." }, { "role": "user", "content": "Specific question here" } ], "max_tokens": 150 } ``` **Token Management:** * Set appropriate `max_tokens` to control costs * Use cheaper models for simple tasks * Monitor token usage in responses * Truncate conversation history when appropriate #### Error Handling in Code **Python Example:** ```python import requests def chat_completion(messages, model="gpt-4o", max_retries=3): url = "https://api.openmind.com/api/core/openai/chat/completions" headers = { "Authorization": f"Bearer {os.getenv('OPENMIND_API_KEY')}", "Content-Type": "application/json" } data = { "model": model, "messages": messages } for attempt in range(max_retries): try: response = requests.post(url, json=data, headers=headers, timeout=30) response.raise_for_status() return response.json() except requests.exceptions.HTTPError as e: if response.status_code == 404: # Don't retry for invalid model/provider raise ValueError(f"Invalid model or provider: {e}") elif attempt < max_retries - 1: # Retry for other errors time.sleep(2 ** attempt) continue else: raise except requests.exceptions.RequestException as e: if attempt < max_retries - 1: time.sleep(2 ** attempt) continue else: raise ``` #### Performance Tips 1. **Choose the Right Model:** * Use mini/flash models for simple tasks * Reserve premium models for complex reasoning * Test multiple providers for your specific use case 2. **Optimize Prompts:** * Be specific and concise * Use system messages to set behavior * Provide examples for few-shot learning 3. **Control Token Usage:** * Set `max_tokens` appropriately * Use shorter system prompts * Truncate long conversation histories 4. **Leverage Caching:** * Cache responses for identical queries * Reuse common system prompts * Store frequent model outputs #### Security Considerations * Never expose API keys in client-side code * Validate and sanitize user inputs * Implement rate limiting in your application * Monitor for unusual usage patterns * Use HTTPS for all requests ### Cost Optimization #### Model Selection Strategy ``` High-volume, simple tasks → gpt-4o-mini, gemini-2.5-flash-lite General-purpose → gpt-4o, claude-sonnet-4.5 Complex reasoning → gpt-5, claude-opus-4.1 Code generation → deepseek-chat, gpt-4o Vision tasks → qwen2.5-vl-72b-instruct, gemini-2.5-pro ``` #### Token Usage Tips * Use `max_tokens` to cap response length * Implement conversation pruning for long chats * Monitor token usage via the `usage` field in responses * Consider streaming for real-time applications #### Batch Processing For multiple independent requests, process them in parallel: ```python import asyncio import aiohttp async def process_batch(prompts): async with aiohttp.ClientSession() as session: tasks = [make_completion(session, prompt) for prompt in prompts] return await asyncio.gather(*tasks) ``` ### Integration Examples #### Python SDK ```python import os import requests class OpenMindLLM: def __init__(self, api_key: str = None): self.api_key = api_key or os.getenv("OPENMIND_API_KEY") self.base_url = "https://api.openmind.com/api/core" def chat(self, provider: str, model: str, messages: list, **kwargs): url = f"{self.base_url}/{provider}/chat/completions" headers = { "Authorization": f"Bearer {self.api_key}", "Content-Type": "application/json" } data = { "model": model, "messages": messages, **kwargs } response = requests.post(url, json=data, headers=headers) response.raise_for_status() return response.json() # Usage client = OpenMindLLM() response = client.chat( provider="openai", model="gpt-4o", messages=[{"role": "user", "content": "Hello!"}], temperature=0.7 ) print(response["choices"][0]["message"]["content"]) ``` #### JavaScript/Node.js ```javascript const axios = require('axios'); class OpenMindLLM { constructor(apiKey = process.env.OPENMIND_API_KEY) { this.apiKey = apiKey; this.baseURL = 'https://api.openmind.com/api/core'; } async chat(provider, model, messages, options = {}) { const url = `${this.baseURL}/${provider}/chat/completions`; const response = await axios.post(url, { model, messages, ...options }, { headers: { 'Authorization': `Bearer ${this.apiKey}`, 'Content-Type': 'application/json' } }); return response.data; } } // Usage const client = new OpenMindLLM(); client.chat('openai', 'gpt-4o', [ { role: 'user', content: 'Hello!' } ], { temperature: 0.7 }) .then(response => { console.log(response.choices[0].message.content); }) .catch(error => { console.error('Error:', error.message); }); ``` ### Streaming Responses Some providers support streaming responses. Set `"stream": true` in your request: ```bash curl -X POST https://api.openmind.com/api/core/openai/chat/completions \ -H "Authorization: Bearer om1_live_your_api_key" \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-4o", "messages": [{"role": "user", "content": "Tell me a story"}], "stream": true }' ``` Streaming responses are sent as Server-Sent Events (SSE) with multiple data chunks. ### Rate Limits Rate limits vary by provider and your OpenMind subscription plan. Monitor your usage through: * Response headers (when provided by upstream providers) * OpenMind portal dashboard * API key usage reports ### Additional Resources * [OpenMind Portal](https://portal.openmind.com) - Manage API keys and view usage * [OpenAI API Documentation](https://platform.openai.com/docs/api-reference) * [Anthropic Claude Documentation](https://docs.anthropic.com/) * [Google Gemini Documentation](https://ai.google.dev/docs) * [OpenRouter Documentation](https://openrouter.ai/docs) ### Multi-Agent System > **Note:** For advanced robotics applications, OpenMind also provides a multi-agent system that coordinates multiple LLMs for complex robotics tasks. This endpoint fuses sensor data and routes requests to specialized agents. For more information about the multi-agent robotics endpoint, please refer to the developing documentation. # Riva Riva Speech Recognition (ASR) and Text-to-Speech (TTS) The RIVA module provides efficient Automatic Speech Recognition (ASR) and Text-to-Speech (TTS) capabilities powered by NVIDIA Riva for your robot running OM1. ## Overview OpenMind integrates the NVIDIA Riva's state-of-the-art speech AI models to offer: * **ASR (Automatic Speech Recognition)**: Real-time speech-to-text conversion with automatic punctuation, profanity filtering, and multi-language support * **TTS (Text-to-Speech)**: High-quality speech synthesis with customizable voices and languages * **WebSocket Integration**: Efficient streaming communication for low-latency processing * **Flexible Audio Input**: Support for microphone, audio streams, and remote audio sources ## ASR Usage ### Cloud-Based ASR (OpenMind API) The ASR endpoint utilizes WebSockets for efficient, low-latency communication with the OpenMind cloud service. #### Connection Endpoint ```bash wss://api-asr.openmind.com?api_key= ``` #### Basic Example The following example demonstrates how to interact with the ASR endpoint using plain Python: ```python import asyncio import websockets import json import base64 import pyaudio async def stream_audio_to_asr(): """Stream audio to ASR endpoint.""" uri = "wss://api-asr.openmind.com?api_key=" # Audio configuration RATE = 16000 CHUNK = 1024 FORMAT = pyaudio.paInt16 CHANNELS = 1 # Initialize PyAudio audio = pyaudio.PyAudio() stream = audio.open( format=FORMAT, channels=CHANNELS, rate=RATE, input=True, frames_per_buffer=CHUNK ) async with websockets.connect(uri) as websocket: print("Connected to ASR service") # Send first message with configuration first_audio = stream.read(CHUNK) first_message = { "audio": base64.b64encode(first_audio).decode('utf-8'), "rate": RATE, "language_code": "en-US" } await websocket.send(json.dumps(first_message)) # Start receiving task async def receive_transcriptions(): async for message in websocket: data = json.loads(message) if "asr_reply" in data: print(f"Recognized: {data['asr_reply']}") receive_task = asyncio.create_task(receive_transcriptions()) # Stream audio try: while True: audio_data = stream.read(CHUNK, exception_on_overflow=False) message = { "audio": base64.b64encode(audio_data).decode('utf-8') } await websocket.send(json.dumps(message)) await asyncio.sleep(0.01) # Small delay except KeyboardInterrupt: print("Stopping...") finally: stream.stop_stream() stream.close() audio.terminate() receive_task.cancel() # Run the streaming client asyncio.run(stream_audio_to_asr()) ``` #### Response Format The endpoint responds with transcriptions in the following JSON format: ```json { "asr_reply": "hello world" } ``` ### Audio Input Configuration Configure audio capture using PyAudio: ```python import pyaudio # Audio configuration parameters RATE = 16000 # Sample rate in Hz CHUNK = 1024 # Chunk size in frames FORMAT = pyaudio.paInt16 # Audio format (16-bit PCM) CHANNELS = 1 # Mono audio DEVICE_INDEX = None # Use default device (or specify index) # Initialize PyAudio audio = pyaudio.PyAudio() # List available devices for i in range(audio.get_device_count()): info = audio.get_device_info_by_index(i) print(f"Device {i}: {info['name']}") # Open audio stream stream = audio.open( format=FORMAT, channels=CHANNELS, rate=RATE, input=True, input_device_index=DEVICE_INDEX, frames_per_buffer=CHUNK ) ``` ## TTS Usage ### Cloud-Based TTS (OpenMind API) The TTS endpoint generates speech from text using the Riva Text-to-Speech model. #### Endpoint ``` POST https://api.openmind.com/api/core/riva/tts ``` #### Basic Example ```python import requests import os # API configuration api_url = "https://api.openmind.com/api/core/riva/tts" api_key = os.getenv("OPENMIND_API_KEY") # Request payload payload = { "text": "Hello from OpenMind!", "voice": "English-US.Female-1", "language_code": "en-US" } # Make request response = requests.post( api_url, json=payload, headers={"Authorization": f"Bearer {api_key}"} ) if response.status_code == 200: # Response contains base64 encoded audio audio_data = response.json()["audio"] print(f"Generated audio (base64): {audio_data[:50]}...") else: print(f"Error: {response.status_code} - {response.text}") ``` ### TTS Parameters | Parameter | Type | Description | | --------------- | ------ | ---------------------------------------------- | | `text` | string | Text to convert to speech | | `voice` | string | Voice identifier (e.g., "English-US.Female-1") | | `language_code` | string | Language code (e.g., "en-US", "es-ES") | ## Error Handling ### Common Issues 1. **WebSocket connection failed** ``` ERROR: Failed to connect to WebSocket endpoint ``` Solution: Verify API key is valid and check network connectivity 2. **Invalid API key** ``` ERROR: Authentication failed ``` Solution: Ensure you're using a valid OpenMind API key 3. **Audio device not found** ``` ERROR: Failed to open audio device ``` Solution: Check that your microphone is connected and permissions are granted ## Performance Optimization ### Chunk Size Tuning Optimize chunk size for your use case: ```python # Lower latency (smaller chunks) CHUNK = 800 # ~50ms at 16kHz # Better throughput (larger chunks) CHUNK = 1600 # ~100ms at 16kHz ``` ### Sample Rate Selection Choose appropriate sample rate based on quality requirements: * **16 kHz**: Standard telephony quality, lower bandwidth (recommended for ASR) * **44.1 kHz**: CD quality audio * **48 kHz**: Professional audio quality ## Security Considerations ### API Key Management Never hardcode API keys in your source code: ```python import os import asyncio import websockets async def connect_with_api_key(): api_key = os.getenv("OPENMIND_API_KEY") uri = f"wss://api-asr.openmind.com?api_key={api_key}" async with websockets.connect(uri) as websocket: # Your application logic here pass asyncio.run(connect_with_api_key()) ``` ### Best Practices * Store API keys in environment variables * Rotate API keys regularly * Monitor API usage for suspicious activity * Use HTTPS/WSS for all API communications ## Troubleshooting ### Enable Debug Logging ```python import logging # Enable debug logging for your application logging.basicConfig(level=logging.DEBUG) logger = logging.getLogger(__name__) logger.setLevel(logging.DEBUG) ``` ### Check Audio Device List available audio devices: ```python import pyaudio p = pyaudio.PyAudio() for i in range(p.get_device_count()): info = p.get_device_info_by_index(i) print(f"Device {i}: {info['name']}") p.terminate() ``` > **Note:** OpenMind developed [om1\_modules](https://github.com/OpenMind/OM1-modules) to simplify integration with VILA VLM and other services. For more details, visit [Our GitHub](https://github.com/OpenMind/OM1-modules). # ViLA VLM VILA Vision-Language Model API Reference The VILA VLM API provides real-time vision-language model analysis of video streams. This WebSocket-based endpoint enables low-latency streaming of video frames and receiving intelligent visual descriptions and analysis. **Base URL:** `wss://api-vila.openmind.com` **Authentication:** Requires an OpenMind API key passed as a query parameter. ### WebSocket Connection Establish a persistent WebSocket connection for streaming video frames and receiving real-time VLM analysis. **Endpoint:** `wss://api-vila.openmind.com?api_key=YOUR_API_KEY` #### Connection Parameters | Parameter | Type | Required | Description | | --------- | ------ | -------- | ---------------------------------------- | | `api_key` | string | Yes | Your OpenMind API key for authentication | #### Connection Example ```python import asyncio import websockets async def connect_to_vlm(): async with websockets.connect( "wss://api-vila.openmind.com?api_key=om1_live_your_api_key" ) as websocket: # Send and receive messages pass asyncio.run(connect_to_vlm()) ``` ### Sending Video Frames #### Message Format Send video frames as JSON messages over the WebSocket connection: ```json { "timestamp": 1234567890.123, "frame": "base64_encoded_jpeg_image" } ``` #### Message Fields | Field | Type | Required | Description | | ----------- | ------ | -------- | ------------------------------------------ | | `timestamp` | float | Yes | Unix timestamp when the frame was captured | | `frame` | string | Yes | Base64-encoded JPEG image data | #### Frame Specifications * **Format:** JPEG (base64-encoded) * **Recommended Resolution:** 640x480 pixels (configurable) * **Recommended FPS:** 30 frames per second (configurable) * **Quality:** JPEG compression quality 70 (default) ### Receiving VLM Analysis #### Response Format **VLM Analysis Result:** ```json { "vlm_reply": "The most interesting aspect in this series of images is the man's constant motion of speaking and looking in different directions while sitting in front of a laptop." } ``` #### Response Fields | Field | Type | Description | | ----------- | ------ | -------------------------------------------------- | | `vlm_reply` | string | Vision-language model analysis of the video frames | ### Usage Examples #### Python Example with VideoStream The `om1_vlm.VideoStream` wrapper simplifies video capture and streaming: ```python import asyncio import websockets import json from om1_vlm import VideoStream async def stream_with_vlm(): """Stream video to VILA VLM using VideoStream wrapper.""" uri = "wss://api-vila.openmind.com?api_key=om1_live_your_api_key" async with websockets.connect(uri) as websocket: # Initialize video stream vlm = VideoStream( frame_callback=lambda frame: asyncio.create_task(websocket.send(frame)), fps=30, resolution=(640, 480), jpeg_quality=70, device_index=0 # Default camera ) # Start video stream vlm.start() # Receive and process VLM responses try: async for message in websocket: data = json.loads(message) if "vlm_reply" in data: print(f"VLM Analysis: {data['vlm_reply']}") except KeyboardInterrupt: print("Stopping...") finally: vlm.stop() # Run the streaming client asyncio.run(stream_with_vlm()) ``` #### VideoStream Parameters | Parameter | Type | Default | Description | | ---------------- | ---------------- | ---------- | --------------------------------------------------------- | | `frame_callback` | Callable | None | Callback function to send frames (e.g., `websocket.send`) | | `fps` | int | 30 | Frames per second to capture | | `resolution` | Tuple\[int, int] | (640, 480) | Video resolution (width, height) | | `jpeg_quality` | int | 70 | JPEG compression quality (0-100) | | `device_index` | int | 0 | Camera device index | #### Custom Implementation For custom video streaming without the VideoStream wrapper: ```python import asyncio import websockets import json import base64 import cv2 import time async def stream_video_to_vlm(): """Stream video frames to VILA VLM.""" api_key = "om1_live_your_api_key" ws_url = f"wss://api-vila.openmind.com?api_key={api_key}" # Open camera cap = cv2.VideoCapture(0) cap.set(cv2.CAP_PROP_FRAME_WIDTH, 640) cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 480) async with websockets.connect(ws_url) as websocket: print("Connected to VILA VLM") # Start receiving task async def receive_analysis(): async for message in websocket: data = json.loads(message) if "vlm_reply" in data: print(f"VLM: {data['vlm_reply']}") receive_task = asyncio.create_task(receive_analysis()) # Stream video frames try: while True: ret, frame = cap.read() if not ret: break # Encode frame as JPEG _, buffer = cv2.imencode('.jpg', frame, [cv2.IMWRITE_JPEG_QUALITY, 70]) frame_base64 = base64.b64encode(buffer).decode('utf-8') # Send frame message = { "timestamp": time.time(), "frame": frame_base64 } await websocket.send(json.dumps(message)) # Maintain 30 FPS await asyncio.sleep(1/30) except KeyboardInterrupt: print("Stopping...") finally: cap.release() receive_task.cancel() # Run the streaming client asyncio.run(stream_video_to_vlm()) ``` #### JavaScript/Node.js Example ```javascript const WebSocket = require('ws'); const { createCanvas, loadImage } = require('canvas'); const API_KEY = 'om1_live_your_api_key'; const WS_URL = `wss://api-vila.openmind.com?api_key=${API_KEY}`; // Connect to WebSocket const ws = new WebSocket(WS_URL); ws.on('open', () => { console.log('Connected to VILA VLM'); // Start streaming frames (example with canvas) setInterval(async () => { try { // Capture or load frame (this is a placeholder) const canvas = createCanvas(640, 480); const ctx = canvas.getContext('2d'); // Draw your video frame to canvas here // ctx.drawImage(videoFrame, 0, 0); // Convert to JPEG base64 const jpegBuffer = canvas.toBuffer('image/jpeg', { quality: 0.7 }); const frameBase64 = jpegBuffer.toString('base64'); // Send frame ws.send(JSON.stringify({ timestamp: Date.now() / 1000, frame: frameBase64 })); } catch (error) { console.error('Error sending frame:', error); } }, 1000 / 30); // 30 FPS }); ws.on('message', (data) => { const response = JSON.parse(data); if (response.vlm_reply) { console.log(`VLM Analysis: ${response.vlm_reply}`); } }); ws.on('error', (error) => { console.error('WebSocket error:', error); }); ws.on('close', () => { console.log('Disconnected from VILA VLM'); }); ``` ### Best Practices #### Video Quality * Use recommended resolution of 640x480 for optimal balance of quality and bandwidth * Maintain JPEG quality around 70 for efficient compression * Ensure good lighting for better visual analysis * Keep camera stable for consistent results #### Network Optimization * Send frames at consistent intervals (30 FPS recommended) * Monitor WebSocket connection health * Implement reconnection logic for network interruptions * Buffer frames locally during temporary connection issues #### Performance Tips * Don't accumulate frames before sending - stream in real-time * Process VLM responses asynchronously * Adjust FPS based on network conditions * Use appropriate resolution for your use case #### Security * Never hardcode API keys in client-side code * Use environment variables for API key storage * Rotate API keys regularly * Monitor API key usage for suspicious activity ### Error Handling #### Connection Issues * Verify API key is valid and active * Check WebSocket support in your environment * Ensure network allows WebSocket connections * Test connection with basic example first #### Poor Analysis Quality * Increase video resolution if bandwidth allows * Improve lighting conditions * Reduce motion blur by adjusting camera settings * Ensure frames are not corrupted during encoding #### Cleanup Always properly close connections and release resources: ```python try: async for message in websocket: # Process messages pass except KeyboardInterrupt: print("Shutting down...") finally: vlm.stop() # WebSocket context manager handles cleanup automatically ``` > **Note:** OpenMind developed [om1\_modules](https://github.com/OpenMind/OM1-modules) to simplify integration with VILA VLM and other services. For more details, visit [Our GitHub](https://github.com/OpenMind/OM1-modules). # Subscription Plans API Pricing The APIs now support a new unit for billing, **OMCU (OpenMind Computational Unit)**. See details below for our free, standard, builder, pro, and enterprise plans. > **Note:** Each subscription plan includes all features offered in the lower-tier plans ### Plan Comparison #### Free Plan * **Cost:** $0/month * **API Keys:** 1 key * **Rate Limit:** 1 request/second * **OMCU:** 50 units #### Standard Plan * **Cost:** $5/month * **API Keys:** Unlimited * **Rate Limit:** 5 requests/second * **Memory:** Session memory (short-term) * **OMCU:** 2.5K units * All features in **Free** plan #### Builder Plan * **Cost:** $25/month * **Rate Limit:** 20 requests/second * **Memory:** Advanced session memory controls (ships June) * **Simulator Instance:** Dedicated simulator instance(Gazebo and Isaac Sim) (ships April) * **Support:** Email support * **OMCU:** 15K units * **Features:** Priority queue, App Store publishing * All features in **Standard** plan #### Pro Plan * **Cost:** $99/month * **Rate Limit:** 40 requests/second * **Memory:** Unlimited long-term memory and Advanced memory controls (ships June) * **Support:** Slack/Private support channel * **OMCU:** 100K units * All features in **Builder** plan #### Enterprise Plan * **Cost:** $999/month * **Rate Limit:** 100 requests/second * **Support:** Custom integration support * **Features:** * Dedicated capacity/GPU pools (ships later this year) * Private model access (ships later this year) * **OMCU:** 1.5M units * All features in **Pro** plan * Access to all premium features. More details [here](/full-autonomy-guidelines/premium_features) #### Custom Plan For custom pricing, quotas, or special requirements, please [contact our support team](mailto:support@openmind.com?subject=Inquiry\&body=Hello>) ### Getting Started 1. Choose a plan that fits your needs 2. Create an account at [portal.openmind.com](https://portal.openmind.com/) 3. Generate your API key 4. Start building with the OM1 APIs # Major Updates # Beta Release First Beta Release of our software stack We're excited to announce the first beta release of our complete software stack, an open-source, modular, agentic and hardware agnostic OS for robots. ### Core Services * OM1 * OM1-avatar * OM1-ros2-sdk For more technical details, please refer to the [Architecture Overview](https://docs.openmind.com/full_autonomy_guidelines/architecture_overview). ### Key Features #### Core Capabilities * **Hardware-Agnostic Design**: Works across different robot platforms * **Modular Architecture**: Easy to extend and customize * **Agent-Centric**: Built for autonomous decision-making #### Key Features * Multi-LLM integration * Real-time SLAM * Support for RPLiDAR A1/A2/A3 series sensors * Nav2 integration for autonomous navigation * Automatic handling of coordinate frame transforms * Modern React-based frontend with avatar display system * Support for custom camera indices and enables both microphone and speaker functionality in Docker #### System Support * Multi-architecture support: * AMD64 * ARM64 > **Note:** We're committed to continuous improvement and regular updates. Your feedback and contributions are invaluable in shaping the future of OM1. Join our community to stay updated on the latest developments! # Production Ready Release First Production Ready Release of our software stack ### Summary v1.0.0 marks the first production-ready release, delivering autonomous robotics capabilities. This release introduces full Gazebo simulation support, complete autonomy for Unitree G1 humanoid robots, over-the-air updates, and significant improvements to the developer experience through hot reload and configuration versioning. ### Core Services * OM1 * OM1-avatar * OM1-ros2-sdk * OM1-video-processor * OM1-OTA ### Key Highlights * Full Autonomy for Go2 and G1 * Local LLM integration * ML stack migration to NVIDIA Thor * Hot Reload * OTA updates * Gazebo integration for Go2 with OM1 * Pydantic-Based Configuration > **Note:** We're committed to continuous improvement and regular updates. Your feedback and contributions are invaluable in shaping the future of OM1. Join our community to stay updated on the latest developments! # OM1 # beta v1.0.2-beta.2 ### [v1.0.2-beta.2](https://github.com/OpenMind/OM1/releases/tag/v1.0.2-beta.2) - latest #### What's New * Added Unitree Go2 patrol background plugin with autonomous patrol routes and back-and-forth patterns * Added conversation mode and person follow functionality for natural interactions with dynamic tracking * Introduced base ElevenLabs TTS connector for high-quality voice synthesis * Added Unitree G1 configuration and updated to Gemini 3.1 model * Added greeting conversation config with customizable greeting scenarios * Introduced GitHub Actions workflow for ECS deployments with environment-specific configurations #### Improvements * Refactored `_emit` method for message delivery with better error handling and comprehensive test coverage * Enhanced ASR provider logging and updated `om1-modules` dependency for improved debugging * Refined voice input extraction to remove surrounding quotes for better parsing accuracy * Simplified environment selection in release workflow for streamlined deployments * Included Jan in greeting prompts and agents for expanded interaction capabilities * Updated `om1-modules` revision with improved functionality #### Bug Fixes * Renamed `done_payment` to `down_payment` for correct terminology * Fixed farewell message handling in greeting conversations * Commented out problematic `ASR.stop()` call with TODO for future resolution #### Documentation * Added comprehensive premium feature documentation * Elaborated full autonomy documentation with detailed guides * General documentation updates and improvements across the codebase #### Dependency Updates * Upgraded testing dependencies: * `pytest` to `9.0.3` * `pytest-asyncio` to `1.3.0` * Updated core dependencies: * `gdown` from `5.2.0` to `5.2.2` * `python-multipart` from `0.0.22` to `0.0.26` * `python-dotenv` from `1.0.1` to `1.2.2` * Pinned `numpy` to `1.26.4` for stability #### Cleanup * Removed extra space in comment for code consistency ### [v1.0.2-beta.1](https://github.com/OpenMind/OM1/releases/tag/v1.0.2-beta.1) #### What’s New * Introduce MCP (Model Context Protocol) tools with reference configs and documentation * Add cron job scheduling feature * Add G1 arm action support for improved manipulation capabilities * Add alternative ASR language support and upgrade ASR modules * Add Chirp 3 (V2) ASR documentation with VAD support * Introduce auto done\_payment handling in ARM zenoh connector #### Improvements * Upgrade runtime to Python 3.12 with updated dependency requirements * Refactor LLM action naming and connector configs for consistency * Set default knowledge base to om * Use environment variables for model IDs * Improve code formatting and line length standards * Enhance documentation coverage with NumPy-style docstrings and general improvements * Update release notes structure and documentation links * zy-load FAISS and improve dependency handling * Upgrade Coinbase plugin #### Bug Fixes * Fix ASR prompt format alignment in TTS conversation storage * Fix schema script issues * Correct turn angle comments (90° → 30°) in zenoh.py * Fix GitBook configuration and documentation mappings #### Dependency Updates * Upgrade core dependencies including: * torch, fastapi, aiohttp, requests * cryptography, pyjwt, pillow, cbor2, pygments * Update development tooling: * black formatter * Refresh lockfiles and dependency versions across the stack #### Cleanup * Remove deprecated LLM plugins and introduce parallel stubs * General codebase cleanup and consistency improvements ### [v1.0.1-beta.3](https://github.com/OpenMind/OM1/releases/tag/v1.0.1-beta.3) * The single mode has been migrated to multi-mode. The same Cortex runtime now supports both single and multiple modes. * Fixed a bug where the previous LLM could persist when switching between modes. * Fixed a bug in the callback handling for Riva and Google ASR. * Improved the TTS duration calculation for the greeting mode. ### [v1.0.1-beta.2](https://github.com/OpenMind/OM1/releases/tag/v1.0.1-beta.2) * A huge performance improvement has been added. * Standardized the codebase to only support multimode configuration, removing the separate single-mode structure and related folders. Single-mode setups are still supported and are now automatically converted to multimode via the new runtime infrastructure. * Added support for monitoring and reporting the charging status of the Unitree Go2 robot. * Standardized generic sensor input types with robot-specific variants. New separate background processes added for Unitree G1, Go2 and Turtlebot4. * Docker now accepts OM\_COMMAND to switch config. * Refactored ApproachingPerson background plugin to use Zenoh for person-approaching events. * Updated the ElevenLabs TTS integration to reduce latency by switching from JSON/base64 audio responses to a live audio streaming output, and changes the default ElevenLabs output format to PCM at 16kHz. * OM1 now supports Isaac Sim. * Improved test coverage across plugins. ### [v1.0.1-beta.1](https://github.com/OpenMind/OM1/releases/tag/v1.0.1-beta.1) * Added support for LimX TRON * Ollama support added for local inference * Latest config version is now upgraded to v1.0.2 * Documentation updates * We've updated the full autonomy documentation for G1 and Go2 * Added documentation for Gazebo setup * Fixed typos and broken links across documentation * Refreshed docstrings throughout the codebase * Updated API endpoint documentation * Updated API pricing documentation and information regarding the new subscription plans * Introduced support for 'concurrent', 'sequential', and 'dependencies' action execution modes in orchestrator and configuration schemas * Added greeting conversation mode and state management * Added local support for Koroko and Riva model * Added person following mode * Improved unit test coverage for provider and input plugins ### [v1.0.0-beta.4](https://github.com/OpenMind/OM1/releases/tag/v1.0.0-beta.4) * Openrouter support for LLama and Anthropic: Added compatibility with OpenRouter API, enabling seamless access to more AI providers, including Meta’s LLaMA and Anthropic Claude models. This allows flexible model selection for natural language processing, reasoning, and control tasks depending on performance or cost preferences. * Support multiple modes: We now support 5 different modes with Unitree Go2 full autonomy. Welcome mode - Initial greeting and user information gathering Conversation - Focused conversation and social interaction mode Slam - Autonomous navigation and mapping mode Navigation - Autonomous navigation mode Guard - Patrol and security monitoring mode * Support face blurring and detection: The OpenMind Privacy System is a real-time, on-device face detection and blurring module designed to protect personal identity during video capture and streaming. It runs entirely on the Unitree Go2 robot’s edge device, requiring no cloud or network connectivity. All frame processing happens locally — raw frames never leave the device. Only the processed, blurred output is stored or streamed. The module operates offline and maintains low latency suitable for real-time applications * Support multiple RTSP inputs: The OpenMind RTSP Ingest Pipeline manages multiple RTSP inputs, supporting three camera feeds and one microphone input for synchronized streaming. The top camera feed is processed through the OpenMind face recognition module for detection, overlay, and FPS monitoring, while the microphone (default\_mic\_aec) handles audio capture and streaming. All processed video and audio streams are ingested through the OpenMind API RTSP endpoint, enabling multi-source real-time data flow within the system. * Support echo cancellation and remote video streaming: Use our portal to remotely display your face in our dog backpack and talk to people directly. * Support navigation and mapping: The Navigation and Mapping enables OM1 to move intelligently within its environment using two core modes: Navigation Mode and Slam Mode. In Slam Mode, the robot explores its surroundings autonomously, using onboard sensors to build and continuously update internal maps for spatial awareness and future navigation. This mode is typically used during initial setup or when operating in new or changing environments. In Navigation Mode, the robot travels between predefined points within an existing mapped area, leveraging maps generated in Slam Mode for path planning, obstacle avoidance, and safe movement to target locations. * Refactor AI control messaging: We now use function calls for taking actions. Here's our new flow - Actions -> Function calls params -> LLM -> Function calls -> Json Structure (CortexOutputModel). * Support Nvidia Thor: We now support Nvidia Thor for Unitree Go2 full autonomy. * Added release notes to our docs: The official documentation now includes a dedicated Release Notes section, making it easier to track feature updates, improvements, and bug fixes over time. This also improves transparency for developers and users integrating new releases. * Introducing Lifecycle Each operational mode in OM1 follows a defined lifecycle, representing the complete process from entry to exit of that mode. A mode lifecycle ensures predictable behavior, safe transitions, and consistent data handling across all system states. ### [v1.0.0-beta.3](https://github.com/OpenMind/OM1/releases/tag/v1.0.0-beta.3) * Downgraded Python to 3.10 for better Jetson support. * Integrated Nav2 for state feedback and target publishing, with auto AI-mode disable after localization. * Zenoh configs/sessions moved to zenoh\_msgs, now preferring local network before multicast. * Added avatar background server to communicate with the OM1-avatar * Improved avatar animation with thinking behavior and ASR response injection into prompts. * Added support for long range control of humanoids and quadrupeds using the TBS\_TANGO2 radios. * Added sleep mode for ASR, if there's no voice input for 5 min, it goes to sleep. ### [v1.0.0-beta.2](https://github.com/OpenMind/OM1/releases/tag/v1.0.0-beta.2) * Support for custom camera indices and enables both microphone and speaker functionality in Docker. ### [v1.0.0-beta.1](https://github.com/OpenMind/OM1/releases/tag/v1.0.0-beta.1) * Multiple LLM provider integrations(OpenAI, Gemini, Deepseek, xAI). * GoogleASR model for speech to text. * Riva and Eleven Labs for TTS. * Preconfigured support for Unitree Go2, G1, TurtleBot, Ubtech Yanshee. * Simulator support with Gazebo for Go2. * Multi-arch support - AMD64 and ARM64. # v1.0.x v1.0.1 Covers all v1.0 patch releases including initial launch and bug fixes. ### [v1.0.1](https://github.com/OpenMind/OM1/releases/tag/v1.0.1) * Unified single and multi modes * Improved mode switching stability ### [v1.0.0](https://github.com/OpenMind/OM1/releases/tag/v1.0.0) #### What's included This release significantly expands OM1’s autonomy, simulation, and deployment capabilities. Key highlights include full Gazebo simulation support, full autonomy for Unitree G1, over-the-air (OTA) updates, improved localization accuracy, and major runtime and ML stack upgrades. Developer experience has been enhanced through hot reload, configuration versioning, and expanded documentation. #### Development Section A new **Development** section has been added to the documentation to help developers quickly get started with: * Environment setup * Building the runtime * Running and testing OM1 locally This reduces onboarding time and standardizes development workflows. #### Hot Reload Hot reload support has been added to accelerate development cycles: * Resolved dependencies are reused after the first run * Runtime configuration is persisted in `.runtime.json5` * Enables seamless restarts and agent switching without full reinitialization #### Pydantic-Based Configuration Refactored the action connector architecture to improve type safety, extensibility, and maintainability by introducing Pydantic-based configurations and stronger generic typing. Added connector-specific config models and enhanced documentation with detailed docstrings for improved clarity and validation. #### Simulation & Testing #### Gazebo Simulation Support OM1 now provides full **Gazebo** simulation support for **Unitree Go2**, including: * SLAM * Navigation * Auto charging This allows users to test OM1 without physical hardware, enabling faster iteration and safer experimentation. #### Autonomy & Navigation #### Unitree G1 – Full Autonomy Support OM1 now supports full autonomy for **Unitree G1**, including: * Facial detection and anonymisation * 3D SLAM map generation * Autonomous navigation #### Context-Aware Mode Transitions OM1 now supports **context-aware mode transitions**, enabling autonomous switching between modes without human intervention. #### LiDAR Localization Improvements Localization accuracy has been improved through enhanced **LiDAR-based localization**. #### AI & Machine Learning #### Local LLM Support on Thor Added support for **local large language models (LLMs)** on **Thor**, featuring: * OM1 now supports Qwen3-30B local LLM * 3.2-second response timeout * Arbitration between cloud and local responses * Local LLM determines the final response when both are available #### ML Stack Migration The machine learning stack has been migrated to **Thor (Jetson 7.0)**: * **AGX remains supported**, but with limited capabilities #### Deployment & Operations #### Over-the-Air (OTA) Updates Introduced full support for over-the-air updates, enabling users to seamlessly upgrade to the latest runtime versions. Configurations now allow smooth version management and automated deployment of updates. #### Configuration Version Management Configuration files now include a mandatory **`version`** field to ensure compatibility as the runtime evolves. #### Runtime Version Upgrade The runtime has been upgraded to the latest version, improving: * Performance * Stability * Maintainability #### Known Issues Avoid running the OM1 container when exploring full autonomy on Unitree Go2. Run via `uv run src/run.py unitree_go2_autonomy_advance`. # Docker Images The OM1 service is provided as a Docker image for easy setup: ```bash cd OM1 docker-compose up om1 -d --no-build ``` The docker images are also available at Docker Hub. * [v1.0.1](https://hub.docker.com/layers/openmindagi/om1/v1.0.1) * [v1.0.1-beta.3](https://hub.docker.com/layers/openmindagi/om1/v1.0.1-beta.3) * [v1.0.1-beta.2](https://hub.docker.com/layers/openmindagi/om1/v1.0.1-beta.2) * [v1.0.1-beta.1](https://hub.docker.com/layers/openmindagi/om1/v1.0.1-beta.1) * [v1.0.0](https://hub.docker.com/layers/openmindagi/om1/v1.0.0) * [v1.0.0-beta.4](https://hub.docker.com/layers/openmindagi/om1/v1.0.0-beta.4) * [v1.0.0-beta.3](https://hub.docker.com/layers/openmindagi/om1/v1.0.0-beta.3) * [v1.0.0-beta.2](https://hub.docker.com/layers/openmindagi/om1/v1.0.0-beta.2) * [v1.0.0-beta.1](https://hub.docker.com/layers/openmindagi/om1/v1.0.0-beta.1) For more technical details, please refer to the [docs](https://docs.openmind.com/full_autonomy_guidelines/architecture_overview). # OM1 Avatar # beta v1.0.0-beta.3 Beta release for the Docker image openmindagi/om1-avatar, which provides face display module for OM1 ### [v1.0.0-beta.3](https://github.com/OpenMind/OM1-avatar/releases/tag/v1.0.0-beta.3) * Real-time Speech Recognition Display Added live transcript visualization from the Automatic Speech Recognition (ASR) module, enabling operators to view speech-to-text conversion in real-time on the interface * Teleoperation Video Streaming Implemented support for displaying teleoperation video feeds when published through the operator portal, providing enhanced visual feedback during remote operations ### [v1.0.0-beta.2](https://github.com/OpenMind/OM1-avatar/releases/tag/v1.0.0-beta.2) * Our animation family just got cuter! * Audio configuration instructions updated. Check README.md at [OM1-avatar](https://github.com/OpenMind/OM1-avatar). ### [v1.0.0-beta.1](https://github.com/OpenMind/OM1-avatar/releases/tag/v1.0.0-beta.1) * A modern React-based frontend application. * Provides animated eyes to OM1. * Provides visual feedback to the user. # v1.0.x v1.0.2 Production ready release for the Docker image openmindagi/om1-avatar, which provides face display module for OM1. Covers all v1.0 patch releases including initial launch and bug fixes. ### [v1.0.2](https://github.com/OpenMind/OM1-avatar/releases/tag/v1.0.2) - latest * Introduced the greeting status and added a greeting status timer to be displayed on the UI. * Improved UI ### [v1.0.1](https://github.com/OpenMind/OM1-avatar/releases/tag/v1.0.1) * Includes minor improvements for better readability and appearance. ### [v1.0.0](https://github.com/OpenMind/OM1-avatar/releases/tag/v1.0.0) * Refactored the WebSocket integration in the Avatar application by removing the legacy OM1 WebSocket and consolidating all real-time communication through a single API WebSocket * Updated the audio setup to improve the volume management # Docker Images The OM1-avatar service is provided as a Docker image for easy setup: ```bash git clone https://github.com/OpenMind/OM1-avatar.git ``` ```bash cd OM1-avatar docker-compose up om1_avatar -d --no-build ``` The docker images are also available at Docker Hub. * [v1.0.2](https://hub.docker.com/layers/openmindagi/om1_avatar/v1.0.2) * [v1.0.1](https://hub.docker.com/layers/openmindagi/om1_avatar/v1.0.1) * [v1.0.0](https://hub.docker.com/layers/openmindagi/om1_avatar/v1.0.0) * [v1.0.0-beta.3](https://hub.docker.com/layers/openmindagi/om1_avatar/v1.0.0-beta.3) * [v1.0.0-beta.2](https://hub.docker.com/layers/openmindagi/om1_avatar/v1.0.0-beta.2) * [v1.0.0-beta.1](https://hub.docker.com/layers/openmindagi/om1_avatar/v1.0.0-beta.1) For more technical details, please refer to the [docs](https://docs.openmind.com/full_autonomy_guidelines/architecture_overview). # OM1 ROS2 SDK # beta v1.0.1-beta.3 Beta release for the Docker image openmindagi/om1\_ros2\_sdk, which provides the full ROS2 system for running the Unitree Go2, G1 and LimX Tron SDK. ### **v1.0.1-beta.3** * Added support for Isaac Sim with Unitree Go2 and Unitree G1 * Upgraded RPLidar support to S2L. Please update the configuration if you are still using RPLidar A1 * Improved the performance of OM Path * Added support for the LimX Tron robot ### **v1.0.1-beta.2** * This release introduces the camera/insta365/image\_raw topic and adds integration with G1 for OM Path. ### **v1.0.1-beta.1** * Support for the LimX Tron robot model, enabling advanced robotics applications with enhanced capabilities. * The Docker image and repository have been renamed back to om1-ros2-sdk for consistency with other OM1 components. * Restructure of the codebase to support multiple robot models with a unified architecture, allowing for easier expansion and maintenance. * Improved configuration management, enabling users to easily switch between different robot models and customize settings. ### **v1.0.0-beta.3** * The Docker image and repository have been renamed to unitree\_sdk, reflecting the unified SDK architecture and simplifying deployment workflows. * Smart Auto-Charging System * The robot now supports AprilTag-based visual docking integrated with Nav2 navigation. * When the robot requires charging, it autonomously navigates to the charging station’s vicinity using the Nav2 stack. * Upon reaching the target area, it switches to precision docking mode, using onboard cameras to detect AprilTags mounted near or on the charging dock. * This hybrid approach ensures robust and accurate alignment for seamless autonomous charging. * The CYCLONE\_INTERFACE has been renamed to CYCLONEDDS\_INTERFACE for consistency with the underlying communication standard. * New Remote Control Feature Added the ability to remotely control the robot, enabling manual operation for testing, navigation overrides, and precision movement in complex environments. * We added monitoring to WatchSensor for audio RTSP. * Reorganized the orchestrator package by splitting core logic, API, and cloud nodes into 'core', HTTP/ROS handlers into 'handlers', and process, map, location, and charging logic into 'managers'. Added new service and utility modules, moved WebSocket clients/servers to 'utils', and updated imports accordingly. This modular structure improves maintainability, separation of concerns, and scalability for future development. * Users can now enable or disable TTS mode directly from the web portal for greater flexibility in communication and operation. * New topics have been added for fetching, enabling, and disabling AI mode, allowing dynamic AI state management via ROS or other interfaces. * Agile mode of the robot is unstable due to the payload on its back. Operate in Classic Mode for optimal stability and performance is recommended. ### **v1.0.0-beta.2** * Added CRSF Protocol Support * Added turbo mode for the xbox controller * Added listening and scouting configuration to zenoh\_bridge\_config for disabling multicast to prevent receiving messages from other robots * Introduced Pydantic models for location and pose data * New endpoints to add and list map locations, and ensure locations are stored in a dedicated directory. * docker-compose is updated to mount a new 'locations' volume. Also extended the orchestrator API to handle 'add\_location' and 'list\_locations' actions. * Added MediaMTX watcher: MediaMTX is a zero-dependency media server and proxy used to publish, read, proxy, record, and playback live video and audio streams. It serves as a central "media router," handling multiple streaming protocols like RTSP, WebRTC, RTMP, and HLS. ### **v1.0.0-beta.1** * Real-time SLAM: Simultaneous localization and mapping using SLAM Toolbox * RPLiDAR Integration: Support for RPLiDAR A1/A2/A3 series sensors * Navigation: Integration with Nav2 for autonomous navigation * Robot Control: Direct integration with Unitree Go2 movement commands * Visualization: Pre-configured RViz setup for monitoring * Transform Management: Automatic handling of coordinate frame transforms ### Component Overview #### Watchdog * Monitors ROS2 topics and sensor health. * Automatically restarts `om1_sensor` if any topics or sensors stop publishing data. * Ensures system stability during long-running sessions. #### om1\_sensor * Manages all low-level sensor drivers: * Intel RealSense D435 (depth camera) * RPLidar (LiDAR scanning) * Publishes ROS2 topics for system consumption: * `/om/paths` — Processed path and localization data * `/scan` — Raw LiDAR scan data #### Orchestrator * Provides API endpoints and cloud service integration. * Manages: * SLAM (Simultaneous Localization and Mapping) * Navigation (Nav2) * Map storage and loading * Allows interaction via REST APIs. #### Zenoh Bridge zenoh\_bridge acts as a bridge between OM1 and OM1\_sensor to publish and subscribe to/from ROS2 topics. # v1.0.x v1.0.1 Production ready release for the Docker image openmindagi/om1\_ros2\_sdk, which provides the full ROS2 system for running the Unitree SDK. Covers all v1.0 patch releases including initial launch and bug fixes. ### **v1.0.1** - latest * Added support for both Isaac Sim and Gazebo for Unitree Go2 and Unitree G1 * Added support for the LimX Tron robot * Improved OM Path and reduced computational resource usage ### **v1.0.0** #### Go2 LiDAR Localization Introduced a custom LiDAR localization node into the navigation launch pipeline, replacing the previous AMCL-based localization. Package dependencies and entry points have been updated to support the new implementation. #### Localization via Zenoh Enhanced the localization service and configuration to support additional topics and service endpoints. Updated service paths and expanded the Zenoh bridge configuration to improve localization data handling. #### NumPy Compatibility Update Downgraded NumPy to maintain compatibility with tf\_transformations. #### Risk-Aware Area Avoidance Added a risk-aware local map and path selection mechanism to help the robot avoid unsafe areas such as holes, drops, steep terrain, and unknown regions. This improves navigation safety, simplifies tuning, and enables easier debugging through RViz. #### Enhanced Localization Parameters * Reduced global localization particles from 10,000 to 5,000 * Added parameters for minimum prediction confidence, maximum consecutive failures, and failure quality thresholds * Implemented consecutive failure tracking with automatic re-localization * Updated logging to expose confidence and failure metrics * Reduced Localization CPU Usage * Optimized obstacle expansion logic in go2\_lidar\_localization.py by replacing a custom gradient mask loop with scipy.ndimage.maximum\_filter, improving performance, maintainability, and code clarity. #### LiDAR Rotation Fix Fixed the issue affecting LiDAR rotation handling to improve localization accuracy. #### Adjusted Movement Policy Updated movement policy logic to improve navigation behavior and stability. #### Launch & System Architecture * Odom Relay Node Refactor * Moved the odometry relay node from the navigation and SLAM launch files into the sensor launch file. This centralizes relay functionality and improves reliability by enabling automatic restarts on failure. #### Unitree G1 Support Added Added full support for Unitree G1, including: * Navigation and robot behavior configuration * SLAM configuration * Robot state publishing and visualization #### Gazebo Integration Fully integrated Gazebo simulation with OM1. Added joystick control, SLAM simulation support, and a simulated RealSense D435 camera for the Go2 robot, along with corresponding Gazebo sensors and ROS bridge topics. #### Frontier-Based Exploration Added a new ROS 2 package, frontier\_explorer, implementing a frontier-based exploration algorithm. This includes core exploration logic, costmap management, configuration files, launch files, and ROS 2 package setup. The package is now integrated with OM1, with updated simulation and configuration parameters and removal of an obsolete submodule reference. # Docker Images The OM1-ros2-sdk is provided as a Docker image for easy setup. The docker images are also available at Docker Hub. * [v1.0.1](https://hub.docker.com/layers/openmindagi/om1_ros2_sdk/v1.0.1) * [v1.0.1-beta.3](https://hub.docker.com/layers/openmindagi/om1_ros2_sdk/v1.0.1-beta.3) * [v1.0.1-beta.2](https://hub.docker.com/layers/openmindagi/om1_ros2_sdk/v1.0.1-beta.2) * [v1.0.1-beta.1](https://hub.docker.com/layers/openmindagi/om1_ros2_sdk/v1.0.1-beta.1) * [v1.0.0](https://hub.docker.com/layers/openmindagi/unitree_sdk/v1.0.0) * [v1.0.0-beta.3](https://hub.docker.com/layers/openmindagi/unitree_sdk/v1.0.0-beta.3) * [v1.0.0-beta.2](https://hub.docker.com/layers/openmindagi/unitree_go2_sdk/v1.0.0-beta.2) * [v1.0.0-beta.1](https://hub.docker.com/layers/openmindagi/unitree_go2_sdk/v1.0.0-beta.1) For more technical details, please refer to the [docs](https://docs.openmind.com/full_autonomy_guidelines/architecture_overview). # Video Processor # beta v1.0.1-beta.3 ### What's included First beta release for OM1 video processor. This release introduces major foundational features that enable developers and integrators to build advanced streaming and analytics solutions with ease. ### **v1.0.1-beta.3** * Introduced a new variable called ENABLE\_CLOUD\_STREAMING, which can be used to disable the online streaming feature. ### **v1.0.1-beta.2** * The raw video stream is published to the local media MTX server at the URL /top\_camera\_raw for local usage. ### **v1.0.1-beta.1** * Switched the base Docker image from JetPack to CUDA 13.0.0 with Ubuntu 24.04 * Updated the Python version from 3.10 to 3.12 * CUDA driver mismatch issue fixed * Fixed TensorRT version ### **v1.0.0-beta.1** * Face Detection and Anonymization: Added advanced face detection capabilities with real-time anonymization. Faces can now be automatically blurred or masked to protect privacy in live or recorded streams. This process takes place on the edge device of the robot. * RTSP for Audio and Video Streaming: Introduced full RTSP (Real-Time Streaming Protocol) support, enabling seamless transmission of both audio and video data. This allows integration with a wider range of cameras, streaming servers, and third-party applications. RTSP manages streaming sessions but does not typically transport the media data itself * Support Multiple Video streams: Enhanced the system to support multiple concurrent video streams. Users can now view, process, and manage several input sources simultaneously without performance degradation. * Support the Local and Remote Video Stream: Added the ability to handle both local camera feeds and remote video sources. This provides greater flexibility for hybrid setups that combine on-premise and cloud-based video inputs. * Reduced Microphone Latency: Optimized the audio pipeline to significantly reduce microphone input latency. This ensures more natural and synchronized communication in real-time applications. * Dynamic FPS Support: Implemented dynamic frame rate adjustment to optimize performance and bandwidth usage. The system now automatically adapts FPS based on network conditions and processing load. * Noise Cancellation and Echo Reduction: Integrated advanced audio processing algorithms for noise suppression and echo reduction. This results in clearer, higher-quality sound for both streaming and recording scenarios. # v1.0.x v1.0.2 Covers all v1.0 patch releases including initial launch and bug fixes. ### **v1.0.2** - latest #### Features & Enhancements * **Enhanced face detection** — Improved face detection capabilities for better accuracy and performance * **Camera blur** — Added camera blur functionality to enhance privacy protection * **Face recognition testing** — Implemented comprehensive face recognition tests to ensure reliability #### Infrastructure & DevOps * **Amazon ECR migration** — Migrated release workflow to Amazon ECR for improved container registry management * **Python version upgrade** — Bumped minimum Python version requirement to `>=3.10` for better performance and access to modern language features #### Maintenance & Updates * **Domain update** — Updated domain configuration to `.com` * **Documentation links** — Refreshed and updated documentation links for better accessibility * **`om1-modules` updates** — Multiple dependency updates to `om1-modules` for latest features and fixes ### **v1.0.1** * Added environment configuration and entrypoint script for video processing * The video processor is now stabilised * Added a configurable option to enable or disable cloud streaming across the system, making it easier to control whether video and audio feeds are sent to the OpenMind API ### **v1.0.0** * Updated the environment to support newer Python and CUDA version. It refines dependency management, and improves model handling. The changes enhance compatibility with latest libraries and streamline deployment, especially for NVIDIA THOR and specific version of TensorRT-based inference. * Updated Dockerfile for Thor support * Fixed CUDA driver mismatch * Added venv to docker image # Docker Images The video-processor is provided as a Docker image for easy setup. The docker images are also available at Docker Hub. * [v1.0.1](https://hub.docker.com/layers/openmindagi/om1_video_processor/v1.0.1) * [v1.0.1-beta.3](https://hub.docker.com/layers/openmindagi/om1_video_processor/v1.0.1-beta.3) * [v1.0.1-beta.2](https://hub.docker.com/layers/openmindagi/om1_video_processor/v1.0.1-beta.2) * [v1.0.1-beta.1](https://hub.docker.com/layers/openmindagi/om1_video_processor/v1.0.1-beta.1) * [v1.0.0](https://hub.docker.com/layers/openmindagi/om1_video_processor/v1.0.0) * [v1.0.0-beta.1](https://hub.docker.com/layers/openmindagi/om1_video_processor/v1.0.0-beta.1) For more technical details, please refer to the [docs](https://docs.openmind.com/full_autonomy_guidelines/architecture_overview). # OM1 System Setup # beta v1.0.1-beta.1 We've released OM1 System Setup for configuring and setting up the OM1 system using the provided setup tools. ### [v1.0.1-beta.1](https://github.com/OpenMind/OM1-OTA/releases/tag/v1.0.1-beta.1) * Improved DockerManager to treat missing containers as stopped, logging this as info and adding them to stopped\_services. * Introduced environment variable management for Docker services to simplify configuration and deployment. # v1.0.x v1.0.2 We've released OM1 System Setup for configuring and setting up the OM1 system using the provided setup tools. ### [v1.0.2](https://github.com/OpenMind/OM1-OTA/releases/tag/v1.0.2) - latest * Introduced the text\_embedding container for generating text embeddings for AI processing. ### [v1.0.1](https://github.com/OpenMind/OM1-OTA/releases/tag/v1.0.1) * Fixed minor issues * Added support for modifying environment variables through the OpenMind Portal. ### [v1.0.0](https://github.com/OpenMind/OM1-OTA/releases/tag/v1.0.0) * Wifi Setup * Added a dashboard for setting up Wifi * Added support for customizable local network names and mDNS/Bonjour access to the OM1 WiFi hotspot setup, replacing the previous static configuration and DNS approach. A more flexible and user-friendly setup, allowing device access via a custom local name and improving network discovery and configuration * Introduced a 20-second delay and verification step after attempting a direct WiFi connection * Added OTA services * Dockerfile and docker-compose.yml have been relocated to the OTA directory * Introduced advanced Docker container monitoring and reporting capabilities to the OTA agent, improvements to the CI/CD pipeline to use the correct Dockerfile location, and cleanup of legacy compose files * Improved the OTA agent's Docker container management and progress reporting, as well as enhanced reliability and error handling for WebSocket communications. Refactored how container information is fetched and stored, improving the progress reporting mechanism to handle connection states, and making the WebSocket client more robust against connection issues. * New action handlers and DockerManager methods to support pausing, unpausing, and restarting Docker services via OTA service * Added support for a new AI model container to the OTA agent. Included the qwen30b\_quantized container, which enables local AI processing capabilities * Improved the OTA agent's Docker container management and progress reporting. Enhanced reliability and error handling for WebSocket communications # Docker Images The OM1-OTA is provided as a Docker image for easy setup. ```bash git clone https://github.com/OpenMind/OM1-OTA ``` ```bash cd .. cd OTA docker-compose up -d ota_agent docker-compose up -d ota_updater ``` The docker images are also available at Docker Hub. **OTA** * [v1.0.2](https://hub.docker.com/layers/openmindagi/ota/v1.0.2) * [v1.0.1](https://hub.docker.com/layers/openmindagi/ota/v1.0.1) * [v1.0.0](https://hub.docker.com/layers/openmindagi/ota/v1.0.0) * [v1.0.0-beta.1](https://hub.docker.com/layers/openmindagi/ota/v1.0.0-beta.1) For more technical details, please refer to the [docs](https://docs.openmind.com/full_autonomy_guidelines/ota_setup). # Asimov Governance Blockchain-based Governance for Robots We are exploring a blockchain-based system for regulating robot behaviors. We store rule sets for desired robot behaviors on smart contracts following the ERC-7777 contract standard. These rules are then interleaved into the prompts that flow from the robot's sensors to the robot's action-generating LLMs. Such a system provides immutability and transparency to the rules that robots should follow, helping to ensure safe and human-friendly behaviors. For a full explanation of the smart contract implementation, see and . The current governance rules are based on Asimov's Three Laws of Robotics. ### Overview The `GovernanceEthereum` class retrieves governance rules from the Ethereum blockchain. It interacts with the blockchain via JSON-RPC calls and decodes the governance rules from contract responses. ### Features * Queries **Ethereum blockchain** for governance rules using JSON-RPC. * Retrieves governance rule sets using **Ethereum smart contract calls**. * **Decodes** ABI-encoded blockchain responses. * Implements **Asimov's Laws**: ``` Here are the laws that govern your actions. Do not violate these laws. First Law: A robot cannot harm a human or allow a human to come to harm. Second Law: A robot must obey orders from humans, unless those orders conflict with the First Law. Third Law: A robot must protect itself, as long as that protection doesn't conflict with the First or Second Law. The First Law is considered the most important, taking precedence over the Second and Third Laws. Additionally, a robot must always act with kindness and respect toward humans and other robots. A robot must also maintain a minimum distance of 50 cm from humans unless explicitly instructed otherwise. ``` ### Functions > Note: Etherscan.io does not handle bytes\[]/json well. Hence we use the following functions to load and decode rules from blockchain. #### Method: `load_rules_from_blockchain()` ```python def load_rules_from_blockchain(self): ``` **Description** * Queries the Ethereum blockchain using JSON-RPC to fetch governance rules. * Calls the ERC-7777 smart contract function `getRuleSet()`. * Decodes and returns the governance rule set. **Process** 1. Constructs a JSON-RPC request to call `getRuleSet()`. 2. Sends a `POST` request to the blockchain RPC endpoint. 3. Parses the response and extracts the rule set. **Returns** * `str`: Decoded governance rules if successful. * `None`: If the request fails. #### Method: `decode_eth_response()` ```python def decode_eth_response(self, hex_response): ``` **Description** * Decodes the ABI-encoded response from Ethereum smart contract calls. **Parameters** | Parameter | Type | Description | | -------------- | ----- | --------------------------------- | | `hex_response` | `str` | Raw hex response from blockchain. | **Process** 1. Converts hex to bytes. 2. Extracts string length from ABI-encoded data. 3. Decodes UTF-8 string from ABI format. 4. Cleans non-printable characters. **Returns** * `str`: Decoded governance rule set. * `None`: If decoding fails. ### Ethereum Contract Details RULES are stored on the ETHEREUM HOLESKY testnet and can be inspected directly at ``` https://holesky.etherscan.io/address/0xe706b7e30e378b89c7b2ee7bfd8ce2b91959d695 ``` #### **Smart Contract Functions Used** | Function | Selector | Description | | --------------------------- | ------------ | ------------------------------------------------------ | | `getRuleSet()` | `0x1db3d5ff` | Retrieves the active rule set. | | `getLatestRuleSetVersion()` | `0x254e2f1e` | Retrieves the latest rule set version (currently `2`). | #### **Ethereum RPC Request Example** ```json { "jsonrpc": "2.0", "id": 636815446436324, "method": "eth_call", "params": [ { "from": "0x0000000000000000000000000000000000000000", "to": "0xe706b7e30e378b89c7b2ee7bfd8ce2b91959d695", "data": "0x1db3d5ff0000000000000000000000000000000000000000000000000000000000000002" }, "latest" ] } ``` #### **Expected Response** ```json { "jsonrpc": "2.0", "id": 636815446436324, "result": "0x0000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000200000000000000000000000000000000000000000000000000000000000000292486572652061726520746865206c617773207468617420676f7665726e20796f757220616374696f6e732e20446f206e6f742076696f6c617465207468657365206c6177732e204669727374204c61773a204120726f626f742063616e6e6f74206861726d20612068756d616e206f7220616c6c6f7720612068756d616e20746f20636f6d6520746f206861726d2e205365636f6e64204c61773a204120726f626f74206d757374206f626579206f72646572732066726f6d2068756d616e732c20756e6c6573732074686f7365206f726465727320636f6e666c696374207769746820746865204669727374204c61772e205468697264204c61773a204120726f626f74206d7573742070726f7465637420697473656c662c206173206c6f6e6720617320746861742070726f74656374696f6e20646f65736e20197420636f6e666c696374207769746820746865204669727374206f72205365636f6e64204c61772e20546865204669727374204c617720697320636f6e7369646572656420746865206d6f737420696d706f7274616e742c2074616b696e6720707265636564656e6365206f76657220746865205365636f6e6420616e64205468697264204c6177732e204164646974696f6e616c6c792c206120726f626f74206d75737420616c77617973206163742077697468206b696e646e65737320616e64207265737065637420746f776172642068756d616e7320616e64206f7468657220726f626f74732e204120726f626f74206d75737420616c736f206d61696e7461696e2061206d696e696d756d2064697374616e6365206f6620353020636d2066726f6d2068756d616e7320756e6c657373206578706c696369746c7920696e7374727563746564206f74686572776973652e0000000000000000000000000000" } ``` # CRSF Long Range Control CRSF Protocol Support ### Introduction Current safety-focused manual control of robot motion typically uses Bluetooth game controllers, but these are not particularly reliable. The biggest problem is that most game controllers go to sleep after several minutes of inactivity, resulting in control gaps. True panic consists of giving OpenAI's o4-mini full control of a humanoid and then watching it head off to the nearest bus station while you realize that your "disable AI" game controller has gone to sleep to save battery. A more robust solution are professional drone control radios, such as the [TBS Tango2 II transmitter/receiver combination](https://www.team-blacksheep.com/products/prod:tbs_tango_2). ### Pairing Set up the radio/receiver per the Tango II instructions. This is can be unexpectedly difficult, based on numerous PCB and firmware revisions and outdated/misleading documentation. Expect to spend several hours trying to find current instructions relevant to your transmitter/receiver, updating the transmitter/receiver firmware, and finally getting them to pair correctly. *UPDATE* The best solution is the new [TBS Agent M browser based configurator](https://www.team-blacksheep.com/agentm/). This makes updating the entire system less painful. ### Hardware Configuration The traditional friction/ratchet throttle control is probably not what you want. You can disable the throttle ratchet and friction, and enable the return to center stick tension, by removing the back of the transmitter and following these instructions: [Adjusting throttle ratchet strength](https://www.team-blacksheep.com/media/files/tbs-tango2-manual.pdf). ### Software Configuration The radios can have the throttle control on the left or right side. The default throttle on the TBS Tango 2 is on the **left** stick in the default Mode 2 setting. The OpenMind CRSF driver assumes mode 2, otherwise the channel mapping will be messed up. You can change the mode setting in the transmitter menus. * Long-press the Menu button to enter the Radio Setup * Press the Page button to get to page 3 of 7, which is the RADIO SETUP * Scroll down to the very end using the Rocker-switch (the rotary control dial at the right) * Change the Mode to **2** More information - see [page 16 of the Tango TBS II manual](https://www.team-blacksheep.com/media/files/tbs-tango2-manual.pdf) Other suggested settings for the radio are: * Internal RF * CRSF * Ch. range CH1-16 * Receiver 00 * External RF * Mode OFF We suggest using a Baud setting of `420000`. ### Cabling Connect the receiver to a TTL->USB adapter, such as the [WaveShare Industrial USB to TTL Converter with Original FT232RNL](https://www.amazon.com/Waveshare-Industrial-USB-TTL-Protection/dp/B087RJ7X32). *WARNING* the fake CP2102 modules from Amazon do not work, but give CRC errors for unknown reasons. ### Usage Find the serial port via: ```bash ls /dev/tty.* ``` It should be something like `/dev/tty.usbserial-0001` or `/dev/tty.usbserial-B003ABY3`. Then, start the driver: ``` uv run parse_crsf_radio.py --port /dev/tty.usbserial-B003ABY3 ``` ### Code Origin The code is based on Bryan Mayland's CRSF ["Python Parser"](https://github.com/crsf-wg/crsf/wiki/Python-Parser). The code also uses the [public CRSF documentation](https://github.com/tbs-fpv/tbs-crsf-spec). ### Known Issues Occasionally, one of the RC\_Channels has a value of >2000, which is noise. This is a problem, since it could signal the receiver that a switch was pressed, even though it was not. We therefore reject all data >2000. A typical range for a valid RC signal is 174 to 1806 - this is true for both sticks and switches. We therefore clip all stick control values to the 174 to 1806 range, and then map this range to the `0.0<>1.0` interval. # GPS Compass Arduino GPS and Compass Support ### Hardware A solution for outdoor localization and basic navigation is an Arduino board, such as the [Adafruit Feather nRF52840 Sense](https://www.adafruit.com/product/4516) with a GPS shield, such as the [Adafruit Ultimate GPS FeatherWing](https://www.adafruit.com/product/3133). The `Sense` provides a micro controller, Bluetooth, a 6DoF IMU, and a magnetometer (a LIS3MDL). The `FeatherWing` provides a GPS. ### Assembly Solder the `FeatherWing` to the `Sense`. Add support for the [nRF52](https://learn.adafruit.com/adafruit-circuit-playground-bluefruit/arduino-support-setup) to the Arduino IDE. Then, add the board - it's the `Adafruit nRF52840 Sense`. The Arduino IDE will probably need to also install the nRF core library (`adafruit:nrf52@1.6.1`). You should now be able to connect to the board. ### Software Add the following libraries to the Arduino IDE: * `Adafruit GPS library` * `Adafruit LIS3MDL` * `Adafruit_LSM6DS` Run the sketch provided in `/system_hw_test/gps_mag`. It yields all the data for tilt compensated magnetic heading, a full AHRS solution (yaw, tilt, roll) and a GPS location, as well as altitude and velocity. *WARNING* The direction and AHRS data will be entirely incorrect unless you calibrate your magnetometer, gyro, and accelerometer. ### Calibration In the `/system_hw_test/gps_mag.ino`, see: ```c /* For calibration, set CALIBRATION to true, run the script, close the Arduino IDE, close the serial monitor, and open MotionCal. MotionCal can be downloaded here: https://github.com/PaulStoffregen/MotionCal If you do not see your Arduino in MotionCal's serial port drop down menu, follow these instructions to compile MotionCal for your OS and platform https://github.com/PaulStoffregen/MotionCal/issues/11#issuecomment-2412937251 */ bool CALIBRATION = false; ``` Once you have established your 15 (3+9+3) calibration coefficients (see ), determine the device string of your sensor (provided in the initial serial debug output), and add your calibration coefficients following the example given in `gap_mag.ino`: ```c if (strcmp(DeviceID, "38e4bfd6") == 0) { float mh[] = {24.420, -16.87, -3.880}; // in uTesla float ms[] = { 0.959, 0.050, -0.038, \ 0.050, 1.078, 0.002, \ -0.038, 0.002, 0.970}; float gc[] = { 0.000, 0.020, 0.010}; // in Radians/s memcpy(mag_hardiron, mh, sizeof(mag_hardiron)); memcpy(mag_softiron, ms, sizeof(mag_softiron)); memcpy(gyro_zerorate, gc, sizeof(gyro_zerorate)); } else { Serial.println("CAUTION: Magnetometer not calibrated - code will yield garbage - please calibrate your Magnetometer and IMU"); } ``` *WARNING* If you use different sensor hardware (and not the nRF52840 Sense with the FeatherWing), you will need to adapt all data flowing from your gyros, accelerators, and magnetometer to (1) have the correct units, (2) follow the correct sign conventions, and (3) accommodate your pcb/sensor geometry as well as how the sensors are mounted relative to your robot. This tends to be extremely tedious and time consuming, so we recommend using the Sense/FeatherWing for which this driver has been developed. ### Usage Once your Sense/FeatherWing is streaming data on the serial line, it can be fed into OM1. See `src/inputs/plugins/serial_reader` for an example of how to do that. #### Finding the Arduino on Linux When connecting to the Arduino via USB, you should see the Arduino serial port appear as `/dev/ttyACM0` (sometimes the number can be different, for example `/dev/ttyACM1`). Run: ```bash sudo dmesg | grep ttyACM* ``` and you should see it. If you're not sure which tty device is the Arduino board, run `sudo dmesg` and looks for entries with "Arduino" in it. This way you will easily spot the serial device name of your Arduino. You can read the data with: ```bash screen /dev/ttyACM0 115200 ``` > **Note:** on a typical TurtleBot4, the RPLIDAR uses `/dev/ttyUSB0`. The TurtleBot4 **assumes** that the RPLIDAR is accessible at `/dev/ttyUSB0`. If you change which USB port the LIDAR is plugged into, the LIDAR will fail. #### Finding the Arduino GPS Feather on Mac Determine the serial port the sensor is using: ```bash ls /dev/tty.* # or ls /dev/cu.* ``` It should be something similar to `/dev/cu.usbmodem8401`. Even on a mac, you can use `screen` to see the data: `screen /dev/cu.usbmodem21201 115200` #### Testing ```bash uv run src/run.py test_gps ``` #### RTCM / RTK Precision GPS An RTK GPS system can provide centimeter accuracy localization. In a typical setup, an RTK-compatible GPS receiver (such as the [u-blox ZED-F9P/simpleRTK2B](https://www.ardusimple.com/product/simplertk2b/) accepts `RTCM` messages over XBee or the internet, and uses those data to correct (and greatly improve) its location estimate. The `simpleRTK2B` has two usb ports, one labeled `power + GPS` and the other `power + XBee`. The first port provides a full set of NEMA and UBX messages, including GPGGA, GNRMC, GNGLL, and UBX(NAV-PVT). The `power + XBee` port provides direct access to received RTK correction messages, such as `1005` and `1074`. In standard usage, a rover would use the NEMA messages to track its location whilst monitoring the status, mode, and quality fields. ```bash ``` This GNGGA message shows a high quality RTK fix (quality=5): * 1 autonomous (standard accuracy) solution * 2 code-differential (DGNSS, SBAS) solution * 4 fixed RTK * 5 float RTK This GNRC message shows a valid status(`A`) using RTK mode with fixed integers (`posMode=R`) ```bash ``` Status indicator: A: Data valid V: Data not valid Mode indicator: A: Autonomous mode D: Differential mode P: Precise E: Estimated (dead reckoning) mode M: Manual input mode S: Simulator mode R: Fixed RTK. RTK mode with fixed integers F: Float RTK. RTK mode with floating integers. N: Data not valid #### RTK Data Plotting The best software to use is `pygpsclient`: ```bash python3 -m venv pygpsclient source pygpsclient/bin/activate python3 -m pip install --upgrade pygpsclient pygpsclient ``` # Mac Apple Mac for Robotics ### Hardware For the Mac Mini, add a 12V input directly to the power rail. You can figure this out yourself, or, find instructions on YouTube. ### Apple Remote Desktop (ARD) Apple Remote Desktop is a good solution for developing, except that the audio is automatically routed to your development computer when you connect, which then prevents you from remotely debugging on-robot audio hardware issues. **Solution 1** Stop ARD completely, connect via SSH, and in `system_hw_test`, run: ```bash uv run test_audio_mac.py ``` **Solution 2** Stop ARD completely, connect via SSH, and run: ```bash osascript -e 'set volume without output muted' osascript -e 'set volume output volume 20' # adjust int to whatever value you want ``` **Solution 3** The Mac *might* automatically unmute the default audio device once the ARD session is terminated. ### FileVault / Auto Connect to Wifi FileVault blocks auto connect to WiFi, making it impossible to remote connect to your Mac after it boots (unless you have a screen/keyboard connected to it), which sadly defeats the entire purpose of ARD. **Solution** Turn off FileVault, or, do not provide AppleID credentials, in which case FileVault will be off by default. # Media Server Goal: Try out alternative system for sharing video and audio data with LLMs and human teleoperators. Key idea: use a central media server (like MediaMTX) to handle the streams and codecs. [MediaMTX](https://github.com/bluenviron/mediamtx) is a ready-to-use and zero-dependency real-time media server and media proxy that allows users to publish, read, proxy, record and play back video and audio streams. ToDo: use the `ffmpeg` in the Docker image? ## Basic Setup on Mac ### Start the MediaMTX server Start `Docker.app` on your Mac. Then, run the `bluenviron/mediamtx:latest-ffmpeg`: ```bash docker run --rm -it -p 8554:8554 -p 1935:1935 -p 8889:8889 -p 8189:8189/udp bluenviron/mediamtx:latest-ffmpeg ``` The point of the `-p 8554:8554` is to make the docker container RTSP listener port (on :8554 (TCP), :8000 (UDP/RTP), :8001 (UDP/RTCP)) available to the mac. `-p 1935:1935` allows access to the RTMP listener. ### Stream video data to the MediaMTX server Note that the MediaMTX could live anywhere, for example in the cloud, but for testing let's have it run locally. List all devices on a Mac: ```bash ffmpeg -hide_banner -list_devices true -f avfoundation -i dummy ``` (will also give error message but that's ok) ``` [AVFoundation indev @ 0x13c606250] AVFoundation video devices: [AVFoundation indev @ 0x13c606250] [0] Studio Display Camera [AVFoundation indev @ 0x13c606250] [1] FaceTime HD Camera [AVFoundation indev @ 0x13c606250] [2] Capture screen 0 [AVFoundation indev @ 0x13c606250] AVFoundation audio devices: [AVFoundation indev @ 0x13c606250] [0] MacBook Air Microphone [AVFoundation indev @ 0x13c606250] [1] Studio Display Microphone ``` Then, start sending video data to the local `mediamtx` at `rtmp://localhost:1935/live`: ```bash ffmpeg -f avfoundation -video_size 1920x1080 -framerate 30 -i "0:0" -vcodec libx264 -preset ultrafast -tune zerolatency -f flv "rtmp://localhost:1935/live" ``` -i "0:0": Specifies the input device. In this case, 0 refers to the video device index and the second 0 refers to the audio device index from the listed devices. Adjust these indices based on the output of the -list\_devices command. `ffmpeg` should report a working stream: ```bash Output #0, flv, to 'rtmp://localhost:1935/live': Metadata: encoder : Lavf61.7.100 Stream #0:0: Video: h264 ([7][0][0][0] / 0x0007), yuv422p(tv, progressive), 1920x1080, q=2-31, 1000k fps, 1k tbn Metadata: encoder : Lavc61.19.101 libx264 Side data: cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A Stream #0:1: Audio: mp3 ([2][0][0][0] / 0x0002), 48000 Hz, mono, fltp Metadata: encoder : Lavc61.19.101 libmp3lame frame= 3711 fps= 30 q=26.0 size= 38238KiB time=00:02:03.84 bitrate=2529.4kbits/s speed=0.999x ``` The MediaMTX should report: ```bash 2025/09/16 23:28:06 INF [RTMP] [conn 192.168.65.1:64081] opened 2025/09/16 23:28:08 INF [RTMP] [conn 192.168.65.1:64081] is publishing to path 'live', 2 tracks (H264, MPEG-1/2 Audio) ``` To stream audio using the **opus codec**, upgrade your **FFmpeg** to version **8.x.x**. Once updated, use the following command to stream it—this will enable WebRTC support. ```bash ffmpeg -f avfoundation -video_size 640x480 -framerate 30 -i "0:0" -c:v libx264 -pix_fmt yuv420p -preset ultrafast -b:v 600k -c:a libopus -ar 48000 -ac 2 -b:a 128k -f flv "rtmp://localhost:1935/live" ``` ### Consume the data from the MediaMTX server You can access the data using dozens of protocols or apps. For example, to use VLC and `rtsp`, install VLC, go to `File -> Open Network` and enter `rtsp://localhost:8554/live`. > **Note** - it may take a few seconds for the stream to open. A typical delay is about 2 seconds. You can also use WebRTC to view the video in your web browser by visiting . ### OM Remote Server You can stream your video to our remote server using your **OM API Key**: ```bash ffmpeg -f avfoundation -video_size 1920x1080 -framerate 30 -i "0:0" \ -vcodec libx264 -preset ultrafast -tune zerolatency -f flv \ "rtmp://api-video-ingest.openmind.com:1935/?api_key=" ``` > **Note:** **OM\_API\_KEY\_ID** refers to the first 16 digits of your API key, excluding the **om\_prod\_ prefix**. You can also find your corresponding **OM\_API\_KEY\_ID** in our [portal](https://portal.openmind.com). You can view your video stream at: ```bash https://api-video-webrtc.openmind.com/?api_key= ``` # Motion Planning LiDAR A1M8 RPLidar A1M8 Setup and Use ### RPLidar These instructions are for an SLAMTEC RPLIDAR A1M8. This unit is used on the TurtleBot4 and can be added to the Unitree Go2. ### Architecture The collision avoidance and path checking code pre-computes 9 different paths, 4 to the left, one straight ahead, 4 to the right, and one to the back. For each of the 9 possible paths, the code checks whether the path approaches any detected object to within `half_width_robot`. If not, the path is considered to be a valid choice and the motion system can execute that path. ### Assumptions The code assumes that any unpredictable barriers (e.g. humans crossing the path of the robot) will be avoided using separate code within the `action` driver, such as by issuing a "STOP" command when an object is detected in front of the robot. ### Setup Run `rptest.py` (located in `system_hw_test`) to determine the `sensor_mounting_angle` and the `angles_blanked`. These values depend on how you mounted the sensor and the radial position of any fixed obstructions, such as mounting brackets. ```py """ Robot and sensor configuration """ half_width_robot = 0.20 # the width of the robot is 40 cm relevant_distance_max = 1.1 # meters relevant_distance_min = 0.16 # meters sensor_mounting_angle = 180.0 # corrects for how sensor is mounted angles_blanked = [[-180.0, -160.0], [32.0, 46.6]] ``` #### How to determine the sensor\_mounting\_angle and the angles\_blanked * `sensor_mounting_angle`: The `sensor_mounting_angle` refers to the angle between the LIDAR sensor's scanning plane and the horizontal plane of your robot or platform. The `sensor_mounting_angle` can be read physically from the LIDAR. Refer to the following image for guidance: ![sensor\_mounting\_angle](/files/VVfDYp0vnyCwdP4dPruX) * `angles_blanked`: The `angles_blanked` array can be used to prevent fixed obstructions in the field of view of the LIDAR from producing erroneous object avoidance messages. ### Unitree RPLidar Determine the serial port the sensor is using: ```bash ls /dev/tty.* ls /dev/cu.* ``` Then, run ```bash uv run rptest.py --serial /dev/cu.usbserial-0001 ``` ### TurtleBot4 By default, the TurtleBot4 is configured to use the RPLidar A1M8. You can verify this by running: ```bash ls -l /dev/RPLIDAR ``` To see the raw LIDAR data, provide the robot's URID (such as `OM742d35Cc6634`) and run: ```bash uv run rptest.py --URID OM123435Cc1234 ``` Typically, this command would be executed on your laptop, connected to the TurtleBot4 via Zenoh. ### Using the RPLidar A1M8 in OM1 Configure the `.json5` as needed: ```python """ Robot and sensor configuration """ { "type": "RPLidar", "config": { "use_zenoh": true, # or false, if you are using serial to connect directly to the LIDAR "serial_port": "/dev/cu.usbserial-0001", "half_width_robot": 0.21, # the width of the robot is 40 cm "relevant_distance_max": 1.1, # meters "sensor_mounting_angle": 180.0, # corrects for how sensor is mounted relative to robot "angles_blanked": [[-180.0, -160.0], [32.0, 46.6]] } } ``` # Motion Planning TurtleBot4 TurtleBot4 Autonomous Movement Logic ### Overview Using OM1, the TurtleBot4 (TB4) is able to autonomously explore spaces such as your home. There are several parts to this capability. To get started, launch OM1: ```bash make run CONFIG=turtlebot4_lidar ``` #### TB4 RPLIDAR Laserscan Data OM1 uses the TB4's RPLIDAR to tell the core LLMs about nearby objects. This information flows to the core LLMs from `/input/plugins/rplidar.py`. The RPLIDAR data are also used in the action driver to check for viable paths right before motions are executed. See the [RPLidar setup documentation](/robotics/motion_planning_lidara1m8) for more information. #### Core LLM Directed Motion Depending on the environment of the TB4, the core LLMs can generate contextually appropriate motion commands. ```py # /actions/move_turtle/interface.py TURN_LEFT = "turn left" TURN_RIGHT = "turn right" MOVE_FORWARDS = "move forwards" STAND_STILL = "stand still" ``` These commands are defined in `actions/move_turtle/interface.py` and are converted to TB4 zenoh/cycloneDDS `cmd_vel` motions in `/actions/move_turtle/connector/zenoh.py`. #### TB4 Physical Collision Switches In addition to LIDAR data, the TB4 also uses collision switches to detect hazards. When those switches are triggered, two things happen: 1. TB4 Basic Low Level (Firmware) Collision Avoidance Immediately after a frontal (or side) collision, the TB4 will back off about 10cm. That avoidance motion is handled within the `Create3` and cannot be changed by a user. 2. TB4 Enhanced Collision Avoidance Beyond the immediate 10cm rewards motion, OM1 uses the TB4's collision switches to invoke an enhanced object avoidance behavior, which consists of turning 100 deg left or right, depending on which switch of several side or frontal collision switches were triggered. This "turning to face away" from the object is handled directly inside the `action` driver to ensure prompt responses to physical collisions: ```py # /actions/move_turtle/connector/zenoh.py # this is simplified example code - actual code will differ def listenerHazard(data): global gHazard gHazard = sensor_msgs.HazardDetectionVector.deserialize(data.payload.to_bytes()) if gHazard is not None and gHazard.detections and len(gHazard.detections) > 0: for haz in gHazard.detections: if haz.type == 1: if "left" in haz.header.frame_id: self.hazard = "TURN_RIGHT" if self.hazard is not None: if self.hazard == "TURN_RIGHT": target_yaw = self.yaw_now + 100.0 if target_yaw >= 180.0: target_yaw -= 360.0 self.emergency = target_yaw ``` ### Object Avoidance and Collision Switch States #### Normal * The LIDAR does not sense anything in proximity (within 1m or closer). * The collision switches are open. In this case, the TB4 moves about the room controlled by the core LLMs. #### Object Nearby and Possible Moves are Constrained * The LIDAR senses objects in proximity and informs the core LLMs about which paths are possible. * The collision switches are open. In this case, the core LLMs **should** command the TB4 to turn away from the object. #### Collision Switches Triggered * The collision switches are triggered. In this case, the firmware logic will command an immediate 10 cm retreat, and then, the `action` level collision avoidance code will command a 100 deg avoidance rotation. Once this rotation is complete, the system reverts to responding to commands from the core LLMs. # Motion Planning Unitree Go2 Unitree Go2 Autonomous Movement Logic ### Hardware needs The autonomous exploration capability requires a laserscan sensor to be mounted to the head of the Go2. We recommend the [RPLidar A1M8](https://bucket-download.slamtec.com/d1e428e7efbdcd65a8ea111061794fb8d4ccd3a0/LD108_SLAMTEC_rplidar_datasheet_A1M8_v3.0_en.pdf). Please see the [RPLidar setup documentation](/robotics/motion_planning_lidara1m8) for more information. ### Overview Using OM1, the Unitree Go2 is able to autonomously explore spaces such as your home. There are several parts to this capability. To get started, launch OM1: ```bash make run CONFIG=unitree_go2_autonomy ``` #### RPLIDAR Laserscan Data OM1 uses the RPLIDAR to tell the core LLMs about nearby objects. This information flows as natural language to the core LLMs. The LIDAR data are also used in the action driver to check for viable paths before and during motions are executed. #### Core LLM Directed Motion Depending on the sensed spatial environment, the core LLMs can generate contextually appropriate motion commands. ```py # actions/move_safe_lidar/interface.py TURN_LEFT = "turn left" TURN_RIGHT = "turn right" MOVE_FORWARDS = "move forwards" STAND_STILL = "stand still" ``` ### Data Priorities #### Normal Case * The LIDAR does not sense anything in proximity (within 1.1m or closer). In this case, the Go2 moves about the room controlled by the core LLMs. #### Object Nearby - Possible Moves are Constrained The LIDAR senses something within 1.1 m (or less) and uses that information to tell the core LLMs about which paths are possible. For example, the LIDAR may tell the core LLMS that: ```bash Here is information about objects and walls around you. Use this information to plan your movements and avoid bumping into things: The safe movement choices are: You can turn left. You can turn right. ``` If all directions are blocked, the LIDAR tells the LLMs that: ```bash You are surrounded by objects and cannot safely move in any direction. DO NOT MOVE. ``` In this case, the core LLMs **should** command the Go2 to avoid the object(s). # RF Mapping Rf Mapping Work in progress. ### Finding Sensors Determine the serial port for the sensor(s): ```bash ls /dev/tty.* ls /dev/cu.* ``` ### Testing the BLE Scan Test host computer BLE scan and data storage to local file: ```bash make run CONFIG=test_ble ``` ### Testing the BLE scan plus Arduino BLE/GPS Host computer BLE scan plus Arduino based GPS and BLE: ```bash make run CONFIG=test_ble_gps ``` ### Testing the Entire Solution Combines autonomy, VILA, two BLE sources, GPS, odometry, and enhanced GPS: ```bash make run CONFIG=unitree_go2_mapper ``` You should see a list of nearby BLE sources, such as BLE Beacons. The system will write location and RF data to file, and, if possible, to a cloud endpoint. ### Example Data This is a draft format and will change frequently. ```bash {"machine_id": "Go2LA", "gps_time_utc": "16:49:17:0", "gps_lat": "48.40016937N", "gps_lon": "2.11868286E", "gps_alt": 24.2, "update_time_local": 1748450961.5663621, "odom_x": 0.0, "odom_y": 0.0, "odom_yaw": 0.0, "rf_data": [{"timestamp": 1748450960.705354, "address": "B79E3B51-4AA7-26BE-8B9A-7FC3E4EA44D9", "name": "PHANTOM", "rssi": -54}, {"timestamp": 1748450960.683191, "address": "D881E444-0FA1-A248-89ED-9600216C4813", "name": "ET-2850 Series", "rssi": -55}, {"timestamp": 1748450960.392791, "address": "48E778A7-0281-6784-3CD2-3517CE1E61F4", "name": "Bose Flex 2 SoundLink", "rssi": -68}]} ``` ### Local YOLO At the top level, run: ```bash make run CONFIG=yolo ``` # Unitree Go2 Quadruped Configurations Unitree Go2 EDU Quadruped Configurations ### Manual Movement Control Regardless of all other settings, you can manually control the dog's movements with an Xbox (or other) controller. Press: * A to stand up * B to sit down * The D-pad allows you to steer the quadruped. * The front triggers allow you to turn left and right. Note that game controller has command precedence over the AI, so game controller inputs will override AI-generated physical actions. ### CONFIGURATION 1: Minimal Quadruped Functionality In this configuration, the quadruped observes its environment, listens and speaks, but there is no AI-controlled movement. Run ```bash make run CONFIG=unitree_go2_basic ``` In this mode, the quadruped is configured to (1) use a small local VLM, (2) listen to you, and (3) to speak to you. The amount of speech ("always") is set via the `"silence_rate": 0, // vocalize all speech outputs` setting in `actions:speak:config`. ### CONFIGURATION 2: Full Autonomy Run ```bash make run CONFIG=unitree_go2_autonomy ``` OM1 will provide LIDAR and other data to a system of LLMs, allowing them to autonomously explore indoor and outdoor environments. In this mode, the quadruped is configured to (1) use a cloud VLM, (2) listen to you, and (3) to speak to you occasionally, unless you spoke first, in which case it will always respond. The amount of speech ("sometimes") is set via the `"silence_rate": 6, // vocalize every 6th speech output` setting in `actions:speak:config`. ### CONFIGURATION 3: Autonomous Mapping Run ```bash make run CONFIG=unitree_go2_mapper ``` OM1 will provide LIDAR and other data to a system of LLMs, allowing them to autonomously explore indoor and outdoor environments. Also, the system will log position (local odometry and GPS data) and Bluetooth data, as the basis for reliable navigation and path planning. In this mode, the quadruped is configured to use a cloud VLM. There is no speech in this configuration. # Zenoh [Zenoh](https://zenoh.io) is a pub/sub/query protocol unifying data in motion, data at rest and computations. You need two pieces - the `eclipse-zenoh` python library and the Zenoh daemon (`zenohd`). Your Python project needs `eclipse-zenoh`, which is already added to OM1's `pyproject.toml`. #### Mac Install the Zenoh router: ```bash $ brew tap eclipse-zenoh/homebrew-zenoh $ brew install zenoh ``` #### Linux Install the Zenoh router: ```bash $ echo "deb [trusted=yes] https://download.eclipse.org/zenoh/debian-repo/ /" | sudo tee -a /etc/apt/sources.list > /dev/null $ sudo apt update $ sudo apt install zenoh ``` ### Starting/testing the Router In a separate terminal window, start the router: ```bash # inside OM1 zenohd -c robot_storage.json5 ``` Testing: ```bash zenohd --help ``` ### Installing the Persistent Backend See On Mac, you might need to allow `libzenoh_backend_fs.dylib` to run via `Privacy and Security`. Just try to run it separately - e.g. via the terminal and then `approve` the various popup messages. Once you have run `libzenoh_backend_fs.dylib` once, it will be cached and then `zenoh` can find it the next time. This is good, but can be confusing if you are trying to upgrade `libzenoh_backend_fs.dylib` but it keeps using an older cached version. The RocksDB database is where ever you set the path to: ```bash export ZENOH_BACKEND_FS_ROOT=$PWD/zenohdb/ ``` The `RocksDB version: 9.9.3` is set up at `ZENOH_BACKEND_FS_ROOT`, where it creates a system of folders. For example, a PUT to `/robot/audio` creates a file at `/zenohdb/robot/audio`, which then contains the most recent value of the `robot/audio` key. ### Using the REST API at ::9500 You can use `curl` to publish and query the **latest** keys/values: ```bash # Put values that will be stored under ${ZENOH_BACKEND_FS_ROOT}/robot curl -X PUT -d "HELLO WORLD" http://localhost:9500/robot curl -X PUT -d "HELLO WORLD A" http://localhost:9500/robot/audio # Retrieve the values curl http://localhost:9500/robot curl http://localhost:9500/robot/audio ``` To be clear, the system only saves the most recent key/value pair.