butterfly/projects
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
50e5df07c0
projects/progress.md

5.9 KiB
Raw Blame History

Butterfly Desktop Environment — Progress

Overview

A remote desktop environment with a Rust (Actix) backend and Angular 21 frontend. The system mimics a traditional Windows-like desktop in the browser, receiving display/audio from VM agents with minimal lag.

Architecture

┌─────────────┐     WebSocket      ┌──────────────────┐     WebSocket      ┌─────────────┐
│  Angular 21 │◄──────────────────►│  Rust Actix Server│◄──────────────────►│ VM Agent exe │
│  (Browser)  │   display/audio    │  (REST + WS)      │   display/audio    │  (Rust)      │
│             │   HUD commands     │                    │   HUD commands     │             │
└─────────────┘                    └──────────────────┘                    └─────────────┘

Checklist

Phase 1: Rust Backend ✅ (builds & runs)

  • server/Cargo.toml — Dependencies: actix-web 4, actix-ws 0.4, actix-cors, dashmap, parking_lot, serde, uuid, chrono
  • server/src/main.rs — Actix HTTP server with CORS, compression, static file serving, SPA fallback
  • server/src/config.rs — Env-based config (BUTTERFLY_HOST, BUTTERFLY_PORT, etc.)
  • server/src/models.rs — Session, AgentConnection, WsMessage enum (serde-tagged), ApiResponse, HealthInfo (with connected_viewers)
  • server/src/state.rs — AppState with DashMap sessions/agents/viewers/agent_channels, FrameBuffer ring buffer, broadcast/forward methods
  • server/src/api/ — REST endpoints: GET/POST/DELETE /api/sessions, GET /api/health, POST /api/sessions/{id}/hud (wired to agent channel)
  • server/src/ws/ — WebSocket handler: agent/viewer connect, per-instance mpsc channels, real bidirectional relay, viewer catch-up, heartbeat timeout
  • server/src/stream/ — StreamStats tracker (frame count, byte relay, uptime)
  • server/static/index.html — Placeholder loading page
  • cargo build succeeds

Phase 1.5: Backend Relay Fix ✅

  • Replaced stub broadcast_to_viewers() with real mpsc channel-based broadcast to all connected viewers
  • Replaced stub forward_to_agent() with real mpsc channel send to agent WS task
  • Added viewer registry (DashMap<String, Vec>) per session
  • Added agent channel registry (DashMap<String, Mutex>) per session
  • New viewers receive the latest buffered frame immediately on connect
  • HUD command REST endpoint now forwards through agent channel
  • Refactored WS handler to use tokio::select! for multiplexed read/write with resettable idle timeout
  • Fixed stats() to count only active sessions (not all sessions)
  • Added connected_viewers to HealthInfo
  • Removed unused dependencies (rand, tokio-stream)

Phase 2: Angular 21 Frontend ✅ (builds & serves)

  • Project scaffold with Angular CLI 21
  • Windows-like desktop shell (taskbar, start menu, window manager)
  • Remote display component (per-instance WebSocket, canvas frame rendering, FPS counter)
  • HUD overlay (mouse click/move/wheel, keyboard down/up forwarding)
  • Window Manager service (open, close, focus, minimize, maximize, drag, resize)
  • WebSocket service (typed message streams, heartbeat)
  • API service (health, sessions CRUD, HUD command forwarding)
  • Built-in apps: File Explorer, Terminal, Text Editor, Settings, Web Browser
  • Session picker dialog (create/connect to remote sessions)
  • Production build: 322KB total (84KB gzipped), output to dist/browser/
  • Dark theme with animated gradient desktop background

Phase 2.5: Frontend Bug Fixes ✅

  • Taskbar clock now updates every second (was static computed signal)
  • Terminal auto-scrolls on output (added AfterViewChecked hook)
  • Remote display uses per-instance WebSocket (was shared singleton — broke multi-session)
  • Remote display canvas resizes with container via ResizeObserver
  • Browser iframe uses DomSanitizer for safe URL binding
  • Browser refresh uses key-based reload instead of URL hack
  • Removed unused imports (RouterModule, ViewChild, etc.)
  • HealthInfo interface updated with connected_viewers

Phase 3: VM Agent Executable 🔲 (next)

  • Rust desktop agent that captures display and audio
  • Streams display frames (JPEG/PNG) and audio (Opus/PCM) via WebSocket
  • Receives HUD commands (mouse, keyboard, resize)
  • Low-latency design for LAN usage

Phase 4: Integration & Polish 🔲

  • End-to-end testing (agent → server → browser)
  • Session management UI
  • Multi-session support (frontend done, backend ready)
  • Authentication (JWT)
  • Performance optimization
  • Audio playback (frontend plumbing exists, needs Web Audio integration)
  • Start menu search filtering
  • Window snap/edge-docking

Recent Commits

  • dd70696 api: add connected_viewers to HealthInfo interface
  • fb2323f fix: remove signal type annotation for safeUrl (TS2749)
  • 6f07a1b fix: move sanitizer call to constructor (was used before init)
  • cecca9f fix: browser uses DomSanitizer for iframe, key-based reload
  • ef1b8df fix: remote display now uses per-instance WebSocket
  • b7d6c95 fix: terminal now auto-scrolls on output
  • f3da41d fix: taskbar clock now updates every second
  • d096d20 cleanup: remove unused currentTime computed, RouterModule
  • 5ea4cdf cargo: remove unused rand and tokio-stream dependencies
  • 4bc5e09 api: wire up real agent channel for HUD commands
  • 2344060 ws/handler: implement real bidirectional relay
  • 29eda76 state: add viewer/agent channel registries
  • dcfaceb desktop: production build works (328KB, 85KB gzip)
  • bae17ae server: fix pong bytes — backend compiles