Building a Personal AI Agent: Architecture and Lessons from LifeOS

Life administration never stops. Bank statements arrive as PDFs buried in email. Investment portfolios drift without monitoring. Contacts get neglected past their natural cadence. Tasks accumulate in mental queues that overflow and crash.

Most productivity systems add cognitive overhead: more apps to check, more processes to remember, more ways to fail. I wanted the opposite — a safety net that catches things before they slip through cracks, requiring zero maintenance and minimal thought.

The solution turned out to be a personal AI agent with deep contextual knowledge about my life, running on infrastructure I control. After a weekend of intense building, it's changed how I think about the relationship between humans and automated systems.

This is the technical story of building LifeOS.

Philosophy: Safety Net, Not Productivity Tool

The foundational insight: this is not a productivity system. Productivity tools demand engagement, optimization, and constant care. Safety nets catch you when you're falling.

The design philosophy:

• Reduce cognitive load — never require remembering two things when one would do
• Cross-update everything — information enters once, propagates everywhere automatically
• Gentle defaults — suggest what to do next, don't mandate productivity
• Defense in depth — multiple security layers, no single point of failure
• Human-readable data — YAML and JSONL, not SQL schemas and migrations

When life gets overwhelming (and it does), the system keeps working. That's what makes it valuable.

Why Not Just Use OpenClaw Directly?

LifeOS runs on OpenClaw, an open-source agent gateway. A fair question: why not just deploy OpenClaw and talk to it?

You can. Out of the box, OpenClaw gives you a lot:

• Gateway: WebSocket server routing messages between channels and any LLM
• Channel integrations: Telegram, WhatsApp, Discord — you're chatting with an AI in minutes
• Tool execution: File operations, shell commands, web browsing, browser automation
• Session management: Context windows, memory search, conversation history
• Sub-agents: Spawn background tasks, parallel work
• Scheduling: Cron jobs, heartbeats, periodic check-ins
• Model-agnostic: Claude, GPT, Ollama, Groq — swap freely

With vanilla OpenClaw, you get a capable AI assistant that can read files, run commands, search the web, and hold conversations across messaging platforms. It's genuinely useful as-is.

But it doesn't know anything about your life.

It doesn't know your bank accounts, your investment strategy, your family relationships, your financial goals, or that you're a contractor who needs to pay quarterly advance tax. It can't parse your bank statements because it doesn't have parsers for your specific banks. It can't track your portfolio because it has no connection to your broker. It can't remind you to call your parents because it doesn't know who your parents are or when you last spoke.

What OpenClaw Solves vs What It Doesn't

The pattern is clear: OpenClaw provides mechanisms, LifeOS provides meaning. OpenClaw knows how to read a file; LifeOS knows which files matter. OpenClaw can run a cron job; LifeOS knows that 9 AM is time for a morning briefing. OpenClaw can execute shell commands; LifeOS has the bank statement parser that turns PDFs into structured data.

The Gap: Domain Knowledge and Personal Context

The real gap vanilla OpenClaw doesn't fill:

1. Structured personal data. OpenClaw has no opinion on how to store your financial accounts, contacts, or preferences. LifeOS defines a YAML-based data layer with clear schemas, permission levels (read-only policies vs updateable defaults), and cross-referencing between files.

2. Domain-specific integrations. Connecting to Zerodha's trading API requires a daily TOTP login flow, session caching, and instrument mapping. Parsing Axis Bank statement PDFs requires knowing the password scheme and regex patterns. These are custom integrations that live in LifeOS skills.

3. Behavioral instructions. AGENTS.md tells the agent: read these files at startup, update contacts when I mention people, append transactions when I mention spending, don't nag about missed commitments, surface one thing not twenty. This is the "personality programming" that makes the agent useful vs generic.

4. Security policies beyond tool sandboxing. OpenClaw blocks dangerous tool calls. LifeOS adds: never send messages as me, never spend above ₹2K without asking, never delete data, always prefer trash over rm. Application-level guardrails on top of framework-level safety.

5. Cross-updating logic. The idea that mentioning a phone call should automatically update a contact's last_contact date isn't an OpenClaw feature — it's an instruction in AGENTS.md that the agent follows because it understands the data schema.

The Skills Bridge

Skills are where framework and application meet. Each skill is a directory with SKILL.md (instructions) and optional scripts:

skills/finance/
├── SKILL.md          # Instructions any agent can read
├── scripts/          # 7 market data scripts
│   ├── kite_data.py  # Broker API integration
│   ├── macro_data.py # India + global macro indicators
│   ├── screener.py   # Custom stock screener
│   └── ...
└── README.md

skills/journal/
├── SKILL.md          # Session summary format
└── templates/

OpenClaw discovers skills via the system prompt and reads SKILL.md before executing related tasks. But the skill itself — the parser code, the data schemas, the domain logic — is all LifeOS.

Someone could take skills/journal/ and drop it into their own OpenClaw instance with minimal changes. The finance skill would need more adaptation (different banks, different broker), but the pattern transfers.

Think of it like this: OpenClaw is Linux. LifeOS is your home directory, your dotfiles, your scripts, your cron jobs. The OS provides the substrate; you provide the life.

System Architecture

Everything runs on a single Hetzner CAX21 server (ARM, 4 vCPU, 8GB RAM) costing ~$15/month total. The core is an AI agent accessible through Telegram and WhatsApp, with persistent memory backed by flat files.

Two-Directory Architecture

This separation is fundamental to the entire system:

~/lifeos/              # Personal data. Local git, NEVER pushed to GitHub.
├── data/              # YAML configuration and state
├── memory/            # Daily logs and long-term memory
├── knowledge/         # Decisions, journal entries
└── skills/            # Task-specific directories

~/lifeos-infra/        # Code. Git repo pushed to GitHub.
├── scripts/           # Data processing and automation
├── services/          # Web app, email processor
└── blog/              # Public content

Why this matters: Personal data must never accidentally end up in a git commit. Code should be version-controlled and deployable. The agent works across both directories but understands the boundary.

When I mention calling family, the agent updates ~/lifeos/data/people/contacts.yaml. When I commit code changes, they go to ~/lifeos-infra/ and can be pushed to GitHub safely. Separation of concerns enforced at the filesystem level.

Skills Architecture

Skills are the extension mechanism. Each skill is a directory with SKILL.md (instructions for the agent) and optional scripts:

skills/
├── capture/
│   ├── SKILL.md           # How to capture quick notes
│   └── scripts/
├── finance/
│   ├── SKILL.md           # Financial tracking instructions
│   └── scripts/           # 7 market data scripts
├── journal/
│   ├── SKILL.md           # Session summary format
│   └── scripts/
└── people/
    ├── SKILL.md           # Relationship tracking
    └── scripts/

The agent discovers skills via the system prompt, reads SKILL.md before executing any skill-related task, and can run the associated scripts. Adding new capabilities means creating a new skill directory.

Examples from actual skills:
- Finance skill: Market data fetching, portfolio analysis, trade signal backtesting
- Journal skill: Session summaries written to dated markdown files
- Capture skill: Quick inbox for thoughts and tasks
- People skill: Contact cadence tracking and relationship maintenance

YAML Data Layer

Instead of databases, the system uses human-readable YAML files that the agent can read and write natively:

# data/policies.yaml - READ-ONLY for agent
budget:
  daily_spend_cap: 2000  # ₹2K max without approval
  monthly_investment_target: 350000
approval_required:
  - sending_messages
  - calendar_commitments  
  - sharing_personal_data
  - purchases_above_threshold

# data/defaults.yaml - AGENT CAN UPDATE
daily_rhythm:
  wake_time: "07:00"
  work_start: "09:30"
  market_close_routine: "15:30"
location_preferences:
  home_city: "indore"
  travel_notifications: true

# data/people/contacts.yaml
family:
  parents:
    contact_cadence: "weekly"
    last_contact: "2026-02-08"
    preferred_method: "call"
friends:
  college_friend:
    contact_cadence: "monthly"
    last_contact: "2026-01-15"
    context: "startup_cofounder"

Why YAML over databases:
- Human-readable and diff-friendly
- No ORM, migrations, or schema management
- Agent reads/writes natively without SQL
- Easy to backup, version, and inspect
- Works across programming languages

The separation between policies.yaml (read-only constraints) and defaults.yaml (updateable preferences) creates a permission model the agent respects but can't circumvent.

Cross-Updating: The Killer Feature

Information enters the system once and propagates everywhere automatically. This is what transforms it from a collection of scripts into a living system.

How it actually works: There's no cross-updating "engine" or event bus. The rules live in AGENTS.md as plain English instructions that the LLM reads at session start:

# From AGENTS.md
| He mentions...           | Update                                          |
|--------------------------|--------------------------------------------------|
| Called/texted/met someone | data/people/contacts.yaml → last_contact        |
| Spent money              | data/finances/transactions.jsonl → append        |
| Made a life decision     | knowledge/decisions/ → log it                    |
| New person in his life   | data/people/contacts.yaml → add with defaults    |

The agent follows these because it understands the data schema and has file write access. No code, no webhooks, no custom middleware — just an LLM that reads instructions and has the tools to act on them. This is surprisingly robust because the LLM handles ambiguity naturally: "grabbed coffee with Rahul" triggers a contact update even though it doesn't literally say "I called someone."

This cross-updating is what makes it a safety net rather than another app to maintain. No manual data entry, no duplication, no inconsistency.

Financial Data Pipeline

The most developed subsystem tracks assets across multiple countries and account types — Indian banks, US brokerages, mutual funds, gold bonds, startup equity, government schemes.

The Agent's Own Inbox

The agent needs to read bank statements and transaction alerts from email. The obvious approach — OAuth into my personal Gmail — would give an AI agent access to everything: conversations, medical records, legal documents. That's a non-starter.

Instead, the agent has its own dedicated mailbox on a self-hosted email server (Mailcow) running on the same infrastructure. It receives only forwarded financial emails and nothing else.

The setup: Gmail forwarding rules on my personal and family email accounts filter specific senders — bank alerts, broker statements, credit card notifications — and forward only those to the agent's mailbox. Everything else stays in Gmail untouched. The agent never sees personal conversations, never gets OAuth to anyone's inbox, never has "read all mail" permission anywhere.

This is minimum-access by design. I control what gets forwarded via Gmail filter rules. The agent gets exactly the emails it needs to do its job, into a mailbox on infrastructure I own. Adding a new bank just means adding a new forwarding rule — the agent doesn't need any additional permissions.

Real-Time Statement Processing

Parser Architecture:
- Regex-based for speed and reliability (milliseconds, no API costs)
- Handles real-world messiness (different PDF formats, password schemes)
- Bank-specific parsers encode institutional quirks
- LLM layer sits on top for analysis, not extraction

Market Data Scripts:
Seven specialized scripts handle different data sources:
- Kite Connect API for live quotes and historical data
- BSE API for quarterly results and corporate actions
- NSE fallback (datacenter IPs blocked) via yfinance
- Macro data aggregation (indices, currency, commodities)
- RSS news aggregation from financial sources
- Custom stock screener with filter syntax
- Daily digest compilation

Trading Integration

Connected to trading APIs but with strict safety rails:

Exchange-Level Restrictions:
- Kite Connect configured with MF-only permissions at broker level
- Even if agent is compromised, API literally cannot place equity orders
- Permissions enforced by exchange, not just application code

Approval Flow:
1. Agent analyzes market data and generates signals
2. Signal sent to Telegram with trade recommendation
3. Human reviews reasoning and decides YES/NO
4. Human executes trade manually (no automated trading)

No automated execution, ever. The agent provides analysis and signals; humans make decisions and execute trades.

What the Agent Built

All of this — the parsing, the portfolio analysis, the fund research, the deployment plan — culminates in artifacts the agent generates and publishes. Here's what happened when I asked the agent to analyze my portfolio:

It started by identifying spare cash sitting idle across multiple accounts — savings accounts, maturing FDs, uninvested balances. Then it researched mutual funds across categories, compared dozens of options on returns, expense ratios, fund manager track records, and portfolio overlap. It picked the best funds for each category and built detailed sub-pages explaining why each fund was selected — not just "this fund is good" but the specific metrics and reasoning.

Then it got practical: given my monthly cash flow, it calculated exactly how much to put into which instrument and when. Lump sum deployments for the idle cash, SIP amounts calibrated to my income, staggered entry plans to avoid timing risk, and a timeline for deploying FD proceeds as they mature. The whole thing — analysis, fund selection, deployment plan, 10-year projections — in one conversation, zero manual work.

The agent didn't just crunch numbers — it designed the page, wrote the HTML, chose the color scheme, and built detailed analysis sub-pages for each recommended fund: Parag Parikh Flexi Cap, Edelweiss Mid Cap, Bandhan Small Cap, and MO Midcap 150 Index. Then it deployed the whole thing behind auth so I could share it with family.

Sharing Agent Work: The Web Platform

One problem I hit early: the agent generates useful artifacts — financial reports, ledgers, comparison docs — but they're stuck on the server. I can't easily share them with specific people without making everything public.

The solution: a lightweight web platform on the same server, behind auth.

How It Works

The stack is simple: Caddy handles TLS and reverse-proxies to a FastAPI app that serves everything — blog posts, static pages, and agent-generated reports.

Caddy (443) → FastAPI (8090)
                ├── /blog/*           → Markdown rendered to HTML
                ├── /static/*         → Agent-generated reports
                ├── /login            → Auth gate
                └── /health           → System dashboard

Per-User Access Control

Every page is gated by a YAML ACL file that the agent can update:

prateek:
    admin: true
    access:
      - "/health"
      - "/api/views"

shaurya:
    access:
      - "/shaurya-balances/*"

public:
    access:
      - "/blog/*"
      - "/composio-vs-n8n-design-doc.html"

The public key controls what unauthenticated visitors can see. Everything else requires login. When I tell the agent "share the Shaurya ledger with Shaurya," it creates an account, sets the ACL, and gives me a link to send — all without exposing anything else.

The Agent as Publisher

The agent doesn't just crunch data — it publishes results as web pages. Some examples from the first weekend:

• A debt ledger for a friend, with every transaction, repayment, and running balance — served at a private URL only they can access
• A financial comparison doc (Composio vs n8n) that I wanted to share publicly with colleagues
• A system health dashboard with live CPU, memory, and service status charts — admin-only

The flow is always the same: agent generates the artifact, drops it in ~/lifeos/static/, and the web platform serves it with the right access controls. No S3 buckets, no deployment pipelines, no "upload to Google Drive and share the link." Just files on disk, served through auth.

View Tracking

Every page tracks views (stored in SQLite), so I can see which reports people actually look at. The blog posts show view counts inline; the admin API gives me breakdowns by time period.

This turned out to be one of the most useful features. The agent isn't just a CLI tool — it's a system that can produce and share work products with anyone I choose, while keeping everything else private.

Memory Architecture

The agent wakes up with no memory each session. Continuity comes from a carefully structured file system:

Session Bootstrap Process:
1. SOUL.md — who I am, what I value, communication style
2. USER.md — timezone, preferences, practical details
3. data/today.md — daily briefing (weather, calendar, contacts, finances)
4. memory/YYYY-MM-DD.md — yesterday's and today's logs
5. MEMORY.md — curated long-term memories (main sessions only)

Memory Layers:
- Policies (data/policies.yaml) — hard limits the agent reads but cannot modify
- Context (data/defaults.yaml) — preferences the agent can update
- State (data/*.yaml) — accounts, contacts, current reality
- History (memory/*.md) — what happened when
- Knowledge (knowledge/) — decisions, learnings, insights

Total context is typically <50K tokens. Not RAG in the traditional sense — the relevant files are small enough to fit in the prompt entirely.

The key insight: the agent updates these files proactively during conversations. Memory persistence through file writes, not through LLM context windows.

Security: Layered Defense

Security isn't just privilege separation — it's defense in depth across multiple layers:

Layer 1: User Separation

• lifeos user: Agent's identity. Broad filesystem and shell access.
• lifeos-secrets user: Owns credentials. Agent cannot read this user's files.
• Scripts needing credentials are root-owned, invoked via sudoers.

Layer 2: Controlled Secret Access

# /etc/sudoers.d/lifeos-secrets  
lifeos ALL=(lifeos-secrets) NOPASSWD: /path/to/with-secrets.sh *

# Usage
sudo -u lifeos-secrets /path/to/with-secrets.sh python3 script.py

The with-secrets.sh wrapper sources credentials and passes environment variables to child processes. Agent cannot directly read ~/.lifeos-secrets even via shell commands.

Layer 3: PreToolUse Hooks

OpenClaw hooks that intercept agent tool calls before execution:
- Block attempts to cat ~/.lifeos-secrets
- Block exec commands targeting secret files
- Block file operations on sensitive paths
- Cannot be bypassed even if agent attempts workarounds

Layer 4: API-Level Restrictions

• Exchange APIs configured with minimal permissions at source
• Kite Connect: MF transactions only, no equity/derivatives access
• API keys scoped to specific operations, not broad access
• Rate limiting and request logging at API layer

Layer 5: Policy Enforcement

# data/policies.yaml - READ-ONLY for agent
approval_required:
  - sending_messages_as_user
  - purchases_above_2000_inr  
  - calendar_commitments
  - sharing_personal_data
  - deleting_data

destructive_commands:
  always_use_trash: true
  never_auto_delete: true

Agent reads these policies but cannot modify them. Only humans can change approval thresholds.

Layer 6: Message Safety

The agent's message tool — which can send messages on Telegram and WhatsApp — is denied at the gateway config level via tools.deny: ["message"]. This is enforced by OpenClaw's tool policy engine before the LLM ever sees the tool. The agent cannot call it, cannot work around it, and cannot convince the system to let it through. deny always wins in the policy hierarchy.

The agent can still reply to me in our direct conversation (that's the channel transport, not the tool). It just cannot proactively message anyone — me or anyone else — through the messaging APIs.

When we want proactive notifications back (e.g., "urgent email arrived"), we'll build a before_tool_call plugin hook that selectively allows sends to the system owner while blocking everything else. Surgical re-enablement, not blanket access.

Philosophy: No single layer provides complete security. Each layer limits blast radius if other layers fail.

What Worked

Flat files over databases. For a single-user system, YAML and JSONL are easier to debug, backup, and reason about. No ORM complexity, no migration scripts, no schema evolution challenges. The agent reads and writes them natively.

Regex parsers for structured data. Bank statements have consistent formats. Regex parsing runs in milliseconds, costs nothing, and doesn't hallucinate. LLM analysis sits on top for reconciliation and insights.

IMAP IDLE over polling. Real-time email processing means salary alerts are logged within seconds. No cron overhead, no missed emails during polling gaps.

Privilege separation from day one. The two-user security model adds development friction but prevents entire classes of security problems. Worth the complexity for a system with bank account access.

Memory as files, not context. Persisting state through file writes rather than LLM memory windows creates reliable continuity that survives session restarts and model switches.

What Didn't Work

NSE API from datacenter IPs. NSE blocks non-residential servers, breaking FII/DII flow data and detailed market metrics. Residential proxy would solve this but adds infrastructure complexity.

LLM-based PDF parsing. Early versions used GPT for statement parsing. Too slow, expensive, and unreliable for financial data. Regex is boring but bulletproof.

Complex approval workflows. Initial designs had multi-step approval processes with different thresholds. Simplified to binary YES/NO decisions with clear policy boundaries.

Database-driven configuration. Started with SQLite for all configuration. Migrated to YAML when I realized the agent should read config files directly, not query through an API.

Production Lessons

Email processing reliability. IMAP IDLE connections drop occasionally. The processor runs as a systemd service with automatic restart, plus monitoring that alerts if no emails are processed for >6 hours.

Market data timing. Some APIs have strict rate limits or time windows. The Kite Connect login flow must complete within a 30-second TOTP window, requiring careful timing logic.

Backup strategy. Personal data lives outside git, so backup is critical. Daily encrypted uploads to cloud storage, plus local copies on multiple devices. Scripts are in git and recoverable.

Log management. Agent conversations generate substantial log data. Automated rotation and compression keeps disk usage manageable while preserving debugging capability.

Update deployment. Code changes go through git, but data updates happen live. Developed conventions for which changes require service restarts vs. hot reloads.

Current Status

The system currently handles:
- Financial tracking across 15+ accounts in two countries
- Investment monitoring with daily portfolio updates and market analysis
- Contact management with automatic cadence tracking
- Email processing with real-time statement parsing
- Trade signals requiring human approval before execution
- Life administration capture and organization

Infrastructure runs ~$15/month. LLM costs depend on your choice — a cloud API subscription, pay-per-token, or $0 with self-hosted models. The entire system runs on a 4-core ARM box, which is also enough to run a quantized Llama 3 locally if you want everything on-premises with no data leaving the server.

Key metrics:
- Zero missed bank statements since deployment
- 95% of spending auto-categorized correctly
- Contact reminders reduced relationship maintenance overhead by ~80%
- Financial dashboard updates within minutes of transactions

What's Next

The system handles the basics well — finances, memory, email processing, web publishing. But there's a lot of ground left to cover:

Automated trading with guardrails. The market data layer and backtesting are done. The next step is a signal-to-execution pipeline: agent generates trade signals, sends them to Telegram with reasoning, I approve with a tap, and the broker API executes. Position size limits at the broker level, daily loss caps, per-trade approval thresholds, kill switches. The agent advises; the human decides.

Voice-first interaction. Speech-to-text is working (Groq Whisper), but the loop isn't closed — I still read text responses. Adding text-to-speech for replies would make the agent usable while walking, cooking, driving. The goal is a system where I can manage my life without looking at a screen.

Proactive nudges. Right now the agent is mostly reactive — I ask, it responds. The cron system can check things periodically, but it doesn't yet surface insights unprompted. "Your Axis account has ₹20L sitting idle — want me to move some into the SIP?" or "You haven't left the house in 3 days — want to grab coffee somewhere?" Gentle, not nagging. The line between helpful and annoying is thin.

Health integration. Sleep, exercise, diet — the areas I'm struggling with most. Wearable data (Apple Watch, Oura) piped into the system could let the agent correlate sleep quality with productivity, suggest rest days, or just track patterns I'm too overwhelmed to notice myself.

Shared skills ecosystem. The skills architecture (markdown instructions + scripts) is designed to be portable. A skill for tracking finances, managing contacts, or processing email could work for anyone running OpenClaw — not just me. Publishing reusable skills is the path from "personal project" to "something others can build on."

Travel planning. "I need to go home to Indore next weekend" → the agent checks flight prices, suggests the best options based on my schedule, books once I approve, and adds it to my calendar. Same for hotels, cab bookings, trip itineraries. All the research and comparison that eats an hour of my evening, automated.

Shopping and life admin. Product research, price comparison, reorder reminders — the kind of stuff that lives in 15 open browser tabs. "Find me a good air purifier under ₹15K" and the agent researches, compares, and gives me one recommendation instead of a list of 50.

Relationship maintenance. The contact tracking already works, but the next step is gentle follow-ups: "You haven't caught up with Rohan in 3 weeks — want me to draft a message?" or "It's Amit's birthday next week." Not a CRM — just a friend who remembers things I forget.

Hard controls everywhere. Every soft policy (don't delete files, don't spend over ₹2K, don't share personal data) should eventually become a hard technical control — before_tool_call hooks that block actions before they execute, not instructions the LLM might ignore. Zero trust in LLM compliance is the end state.

What This Pattern Means

Personal AI agents with deep life context represent a different approach to automation. Not productivity optimization, but cognitive load reduction. Not task acceleration, but safety net deployment.

The technical requirements are different too:
- Memory persistence matters more than inference speed
- Data integration across personal silos
- Security by design when handling sensitive life data
- Human-readable state for transparency and control

This pattern won't work for everyone. It requires comfort with self-hosting, willingness to invest in setup, and trust in AI systems with broad access to personal data.

But for those who build it, the shift is profound. From managing systems to being supported by them. From remembering to maintain things to having them maintained automatically. From cognitive overhead to cognitive relief.

The future of personal AI isn't about becoming more productive. It's about being more human — focusing on what matters while technology handles what doesn't.

This post is served by the system it describes — a FastAPI app deployed by the AI agent, running on the same ARM server that hosts everything else.