Polestar / Skill Lifecycle
Project Description
An enterprise Agent Skill factory for internal business scenarios. The platform enables business users to generate ERP / business Skills through natural language, then evaluate, publish, monitor, and optimize them through a governed lifecycle. It focuses on three key problems: uncontrolled Skill generation quality, possible fake tool-call results from Agents, and lack of observability and self-healing after release.
Responsibilities
- Contributed to the lifecycle pipeline covering Intent, Generate, Eval-Cases, Eval-Run, Publish, Monitor, Report, and Optimize.
- Designed the Stage 3 caseInputs + real spawn evaluation mechanism, where the system executes baicao api call and records stdout, exitCode, and duration.
- Built evaluation categories such as normal, edge, error, security, off_topic, and multi_turn, with Critical gates, Pass@K, Baseline Regression, and rubric scoring.
- Introduced LLM-as-Judge for structured assessment of open-ended outputs while keeping deterministic assertions for stable checks.
- Implemented Fix-Suggester menu-style self-healing and Telemetry instrumentation covering success rate, latency, tokens, toolchain, cohort, and content fingerprints.