Home AI: from conversation to action in my environment.

An assistant with context for my work.

I wanted an assistant that would not need my environment explained from scratch in every conversation. I also needed a way to connect conversation with practical tools: reading configuration, diagnostics and preparing files.

Open WebUI provides the conversation interface; models interpret requests and select tools. My own part of the project connects them to documentation, knowledge retrieval and execution services. It also covers startup, event logging and error handling.

From a goal to a checked result.

I begin with a goal. The selected model profile has an assigned set of tools. Depending on the task, the agent reads documentation, retrieves relevant knowledge or uses PC Operator. The tool result returns to the conversation and informs the next step.

Available capabilities depend on the profile and the permissions of each service. Calling a tool does not establish that the task was completed correctly. I check the outcome: file contents, a service response or application state.

Context matched to the question.

HomeAI Knowledge combines direct documentation access with RAG: retrieving relevant knowledge before preparing a response. A short source map helps the model choose the right domain without adding the entire history to every conversation.

The collections are stored in Qdrant and queried through a local search service. Results are passed to the selected model. When I use an external backend, retrieved context may be sent to it as well.

Tools for working with Windows.

PC Operator exposes file operations, PowerShell commands, background jobs and Windows UI inspection and interaction. Separate services run with user and administrator privileges, each with its own authentication and logs.

Operations marked as requiring approval go through user confirmation. This supports oversight, but a model’s assessment of an arbitrary command is not a security guarantee. The account and execution service permissions also determine what can actually happen.

When an operation’s outcome is unknown, I check the state first. Automatically retrying could perform the same change twice.

Services started in a deliberate order.

The environment starts after Windows sign-in. A script launches the required servers, waits for Docker Engine and starts the relevant Compose projects. PC Operator and the model router also have separate logon tasks. Containers use restart policies and selected services have health checks.

Starting a model server does not immediately load every model’s weights into VRAM. This separates service readiness from the resources needed by a particular task.

Open WebUI is the interface used from my home devices. Execution services and their APIs have separate access rules. The public portfolio on the NAS is a separate site and gives visitors no control over the agent.

An HTTPS layer supports features that require a secure connection. In the current installation, its private certificate needs to be trusted by the client device. Windows UI interaction requires an available desktop session.

Modules that extend the environment.

Autopilot — longer tasks

A separate executor stores the goal, progress, tool results and approval state. It has its own task memory and can work independently of an open chat. It does not automatically inherit all Open WebUI configuration or memory.

GPU controller — model requests

It queues requests passing through its gateway. In the current cooperative mode, direct applications can still use the GPU. It coordinates its own traffic, without exclusive control of the entire card.

ComfyUI — image generation

It runs image workflows on the same computer. This integration supports tasks that combine text and images, with separate checks of the generated result.

Example: a report from documentation and service state.

This scenario outlines a planned demonstration using sample data. It combines knowledge retrieval, a tool call and verification of the final file.

1. Source

   Read the document describing the sample environment.

2. State check

   Select PC Operator actions and gather service responses.

3. Report

   Save a document separating documentation from observations.

4. Verification

   Compare file contents with the actual tool results.

Code, configuration and reproduction.

Most of the work happens between components.

I learn the most at the boundaries between services: when a tool returns an incomplete result, a process is still running in the background, or documentation lags behind configuration. These details determine how useful the whole system is.

I develop Home AI through concrete scenarios and checking outcomes. Models may change; I want the integrations, knowledge access and result handling to remain understandable and testable.