Related papers: Evaluation-as-a-Service: Overview and Outlook
The networking industry, compared to the compute industry, has been slow in evolving from a closed ecosystem with limited abstractions to a more open ecosystem with well-defined sophisticated high level abstractions. This has resulted in an…
AI evaluations have become critical tools for assessing large language model capabilities and safety. This paper presents practical insights from eight months of maintaining $inspect\_evals$, an open-source repository of 70+…
Software-as-a-Service (SaaS) is a viable option for some companies bearing their business processes. There is a considerable adoption rate, with companies already using more than two services for over two years. However, while some…
Structuring related work is a daunting task encompassing literature review, classification, comparison (primarily in the form of concepts), and gap analysis. Building taxonomies is a compelling way to structure concepts in the literature…
Artificial Intelligence (AI) / Machine Learning (ML)-based systems are widely sought-after commercial solutions that can automate and augment core business services. Intelligent systems can improve the quality of services offered and…
Following the rapid increase in Artificial Intelligence (AI) capabilities in recent years, the AI community has voiced concerns regarding possible safety risks. To support decision-making on the safe use and development of AI systems, there…
Artificial Intelligence Impact Assessments ("AIIAs"), a family of tools that provide structured processes to imagine the possible impacts of a proposed AI system, have become an increasingly popular proposal to govern AI systems. Recent…
The meteoric rise of AI, with its rapidly expanding market capitalization, presents both transformative opportunities and critical challenges. Chief among these is the urgent need for a new, unified paradigm for trustworthy evaluation, as…
SOA is a prominent paradigm for accomplishing reuse of services. Service reusability is one dominant factor which has a greater influence on achieving quality in SOA systems. There exists sufficient research in this area and researchers…
Involving users in early phases of software development has become a common strategy as it enables developers to consider user needs from the beginning. Once a system is in production, new opportunities to observe, evaluate and learn from…
Artificial Intelligence (AI) holds promise as a technology that can be used to improve government and economic policy-making. This paper proposes a new research agenda towards this end by introducing Social Environment Design, a general…
This special volume of Statistical Sciences presents some innovative, if not provocative, ideas in the area of reliability, or perhaps more appropriately named, integrated system assessment. In this age of exponential growth in science,…
The Software as a Service (SaaS) model is a distribution and licensing model that leverages pricing structures and subscriptions to profit. The utilization of such structures allows Information Systems (IS) to meet a diverse range of client…
Cloud computing is considered as the one of the most dominant paradigm in field of information technology which offers on demand cost effective services such as Software as a service (SAAS), Infrastructure as a service (IAAS) and Platform…
The increasing adoption of UAVs equipped with advanced sensors and GPU-accelerated edge computing has enabled real-time AI-driven applications in domains such as precision agriculture, wildfire monitoring, and environmental conservation.…
To facilitate the widespread acceptance of AI systems guiding decision-making in real-world applications, it is key that solutions comprise trustworthy, integrated human-AI systems. Not only in safety-critical applications such as…
The maintained artifact in an AI-enabled system is not code plus settings, but a versioned governed program space: domains, structural constraints, eligibility, evaluation assets, and a statistical release gate. AI-enabled systems operate…
Electroencephalography (EEG) is an extensively-used and well-studied technique in the field of medical diagnostics and treatment for brain disorders, including epilepsy, migraines, and tumors. The analysis and interpretation of EEGs require…
Cloud computing, undoubtedly, has become the buzzword in the IT industry today. Looking at the potential impact it has on numerous business applications as well as in our everyday life, it can certainly be said that this disruptive…
To unleash the full potential of AI for Science, we must untether the agents from a purely digital environment. The agent's ability to control and explore in real-world labs is essential because the physical lab remains foundational to…