English

SKILLS: Structured Knowledge Injection for LLM-Driven Telecommunications Operations

Software Engineering 2026-03-17 v1 Artificial Intelligence Cryptography and Security

Abstract

As telecommunications operators accelerate adoption of AI-enabled automation, a practical question remains unresolved: can general-purpose large language model (LLM) agents reliably execute telecom operations workflows through real API interfaces, or do they require structured domain guidance? We introduce SKILLS (Structured Knowledge Injection for LLM-driven Service Lifecycle operations), a benchmark framework comprising 37 telecom operations scenarios spanning 8 TM Forum Open API domains (TMF620, TMF621, TMF622, TMF628, TMF629, TMF637, TMF639, TMF724). Each scenario is grounded in live mock API servers with seeded production-representative data, MCP tool interfaces, and deterministic evaluation rubrics combining response content checks, tool-call verification, and database state assertions. We evaluate open-weight models under two conditions: baseline (generic agent with tool access but no domain guidance) and with-skill (agent augmented with a portable SKILL.md document encoding workflow logic, API patterns, and business rules). Results across 5 open-weight model conditions and 185 scenario-runs show consistent skill lift across all models. MiniMax M2.5 leads (81.1% with-skill, +13.5pp), followed by Nemotron 120B (78.4%, +18.9pp), GLM-5 Turbo (78.4%, +5.4pp), and Seed 2.0 Lite (75.7%, +18.9pp).

Keywords

Cite

@article{arxiv.2603.15372,
  title  = {SKILLS: Structured Knowledge Injection for LLM-Driven Telecommunications Operations},
  author = {Ivo Brett},
  journal= {arXiv preprint arXiv:2603.15372},
  year   = {2026}
}
R2 v1 2026-07-01T11:22:25.792Z