95 lines
5.9 KiB
Markdown
95 lines
5.9 KiB
Markdown
# Butterfly Desktop Environment — Progress
|
|
|
|
## Overview
|
|
A remote desktop environment with a Rust (Actix) backend and Angular 21 frontend. The system mimics a traditional Windows-like desktop in the browser, receiving display/audio from VM agents with minimal lag.
|
|
|
|
## Architecture
|
|
```
|
|
┌─────────────┐ WebSocket ┌──────────────────┐ WebSocket ┌─────────────┐
|
|
│ Angular 21 │◄──────────────────►│ Rust Actix Server│◄──────────────────►│ VM Agent exe │
|
|
│ (Browser) │ display/audio │ (REST + WS) │ display/audio │ (Rust) │
|
|
│ │ HUD commands │ │ HUD commands │ │
|
|
└─────────────┘ └──────────────────┘ └─────────────┘
|
|
```
|
|
|
|
## Checklist
|
|
|
|
### Phase 1: Rust Backend ✅ (builds & runs)
|
|
- [x] `server/Cargo.toml` — Dependencies: actix-web 4, actix-ws 0.4, actix-cors, dashmap, parking_lot, serde, uuid, chrono
|
|
- [x] `server/src/main.rs` — Actix HTTP server with CORS, compression, static file serving, SPA fallback
|
|
- [x] `server/src/config.rs` — Env-based config (BUTTERFLY_HOST, BUTTERFLY_PORT, etc.)
|
|
- [x] `server/src/models.rs` — Session, AgentConnection, WsMessage enum (serde-tagged), ApiResponse, HealthInfo (with connected_viewers)
|
|
- [x] `server/src/state.rs` — AppState with DashMap sessions/agents/viewers/agent_channels, FrameBuffer ring buffer, broadcast/forward methods
|
|
- [x] `server/src/api/` — REST endpoints: GET/POST/DELETE /api/sessions, GET /api/health, POST /api/sessions/{id}/hud (wired to agent channel)
|
|
- [x] `server/src/ws/` — WebSocket handler: agent/viewer connect, per-instance mpsc channels, real bidirectional relay, viewer catch-up, heartbeat timeout
|
|
- [x] `server/src/stream/` — StreamStats tracker (frame count, byte relay, uptime)
|
|
- [x] `server/static/index.html` — Placeholder loading page
|
|
- [x] `cargo build` succeeds
|
|
|
|
### Phase 1.5: Backend Relay Fix ✅
|
|
- [x] Replaced stub `broadcast_to_viewers()` with real mpsc channel-based broadcast to all connected viewers
|
|
- [x] Replaced stub `forward_to_agent()` with real mpsc channel send to agent WS task
|
|
- [x] Added viewer registry (DashMap<String, Vec<ViewerEntry>>) per session
|
|
- [x] Added agent channel registry (DashMap<String, Mutex<Sender>>) per session
|
|
- [x] New viewers receive the latest buffered frame immediately on connect
|
|
- [x] HUD command REST endpoint now forwards through agent channel
|
|
- [x] Refactored WS handler to use `tokio::select!` for multiplexed read/write with resettable idle timeout
|
|
- [x] Fixed `stats()` to count only active sessions (not all sessions)
|
|
- [x] Added `connected_viewers` to HealthInfo
|
|
- [x] Removed unused dependencies (rand, tokio-stream)
|
|
|
|
### Phase 2: Angular 21 Frontend ✅ (builds & serves)
|
|
- [x] Project scaffold with Angular CLI 21
|
|
- [x] Windows-like desktop shell (taskbar, start menu, window manager)
|
|
- [x] Remote display component (per-instance WebSocket, canvas frame rendering, FPS counter)
|
|
- [x] HUD overlay (mouse click/move/wheel, keyboard down/up forwarding)
|
|
- [x] Window Manager service (open, close, focus, minimize, maximize, drag, resize)
|
|
- [x] WebSocket service (typed message streams, heartbeat)
|
|
- [x] API service (health, sessions CRUD, HUD command forwarding)
|
|
- [x] Built-in apps: File Explorer, Terminal, Text Editor, Settings, Web Browser
|
|
- [x] Session picker dialog (create/connect to remote sessions)
|
|
- [x] Production build: 322KB total (84KB gzipped), output to `dist/browser/`
|
|
- [x] Dark theme with animated gradient desktop background
|
|
|
|
### Phase 2.5: Frontend Bug Fixes ✅
|
|
- [x] Taskbar clock now updates every second (was static computed signal)
|
|
- [x] Terminal auto-scrolls on output (added AfterViewChecked hook)
|
|
- [x] Remote display uses per-instance WebSocket (was shared singleton — broke multi-session)
|
|
- [x] Remote display canvas resizes with container via ResizeObserver
|
|
- [x] Browser iframe uses DomSanitizer for safe URL binding
|
|
- [x] Browser refresh uses key-based reload instead of URL hack
|
|
- [x] Removed unused imports (RouterModule, ViewChild, etc.)
|
|
- [x] HealthInfo interface updated with connected_viewers
|
|
|
|
### Phase 3: VM Agent Executable 🔲 (next)
|
|
- [ ] Rust desktop agent that captures display and audio
|
|
- [ ] Streams display frames (JPEG/PNG) and audio (Opus/PCM) via WebSocket
|
|
- [ ] Receives HUD commands (mouse, keyboard, resize)
|
|
- [ ] Low-latency design for LAN usage
|
|
|
|
### Phase 4: Integration & Polish 🔲
|
|
- [ ] End-to-end testing (agent → server → browser)
|
|
- [ ] Session management UI
|
|
- [ ] Multi-session support (frontend done, backend ready)
|
|
- [ ] Authentication (JWT)
|
|
- [ ] Performance optimization
|
|
- [ ] Audio playback (frontend plumbing exists, needs Web Audio integration)
|
|
- [ ] Start menu search filtering
|
|
- [ ] Window snap/edge-docking
|
|
|
|
## Recent Commits
|
|
- `dd70696` api: add connected_viewers to HealthInfo interface
|
|
- `fb2323f` fix: remove signal type annotation for safeUrl (TS2749)
|
|
- `6f07a1b` fix: move sanitizer call to constructor (was used before init)
|
|
- `cecca9f` fix: browser uses DomSanitizer for iframe, key-based reload
|
|
- `ef1b8df` fix: remote display now uses per-instance WebSocket
|
|
- `b7d6c95` fix: terminal now auto-scrolls on output
|
|
- `f3da41d` fix: taskbar clock now updates every second
|
|
- `d096d20` cleanup: remove unused currentTime computed, RouterModule
|
|
- `5ea4cdf` cargo: remove unused rand and tokio-stream dependencies
|
|
- `4bc5e09` api: wire up real agent channel for HUD commands
|
|
- `2344060` ws/handler: implement real bidirectional relay
|
|
- `29eda76` state: add viewer/agent channel registries
|
|
- `dcfaceb` desktop: production build works (328KB, 85KB gzip)
|
|
- `bae17ae` server: fix pong bytes — backend compiles
|