Related papers: Evaluation-as-a-Service: Overview and Outlook
Artificial intelligence and machine learning are reshaping how we approach scientific discovery, not by replacing established methods but by extending what researchers can probe, predict, and design. In this roadmap we provide a…
Artificial intelligence (AI) technologies (re-)shape modern life, driving innovation in a wide range of sectors. However, some AI systems have yielded unexpected or undesirable outcomes or have been used in questionable manners. As a…
The rapid growth and diversity in service offerings and the ensuing complexity of information technology ecosystems present numerous management challenges (both operational and strategic). Instrumentation and measurement technology is, by…
The impact of using artificial intelligence (AI) to guide patient care or operational processes is an interplay of the AI model's output, the decision-making protocol based on that output, and the capacity of the stakeholders involved to…
Generative AI (GenAI) models have become vital across industries, yet current evaluation methods have not adapted to their widespread use. Traditional evaluations often rely on benchmarks and fixed datasets, frequently failing to reflect…
This article offers several contributions to the interdisciplinary project of responsible research and innovation in data science and AI. First, it provides a critical analysis of current efforts to establish practical mechanisms for…
Software engineering research is evolving and papers are increasingly based on empirical data from a multitude of sources, using statistical tests to determine if and to what degree empirical evidence supports their hypotheses. To…
Collaborative AI experimentation in industry and academia requires environments that support rapid trials while maintaining controlled access, organisational isolation, and traceable workflows. Although interest in AI sandboxes is…
Cloud Service Providers deliver their products in form of 'as-a-Service', which are typically categorized by the level of abstraction. This approach hides the implementation details and shows only functionality to the user. However, the…
The next generation of PaaS technology accomplishes the true promise of object-oriented and 4GLs development with less effort. Now PaaS is becoming one of the core technical services for application development organizations. PaaS offers a…
Despite existing work in machine learning inference serving, ease-of-use and cost efficiency remain challenges at large scales. Developers must manually search through thousands of model-variants -- versions of already-trained models that…
How do we evaluate experiences in immersive environments? Despite decades of research in immersive technologies such as virtual reality, the field remains fragmented. Studies rely on overlapping constructs, heterogeneous instruments, and…
Cloud-based services with resources to be provisioned for consumers are increasingly the norm, especially with respect to Big data, spatiotemporal data mining and application services that impose a user's agreed Quality of Service (QoS)…
Software as a service (SaaS) has recently enjoyed much attention as it makes the use of software more convenient and cost-effective. At the same time, the arising of users' expectation for high quality service such as real-time information…
Business Process Management (BPM) in the cloud is focused on how to provide Business Process as a Service (BPaaS) and also implementing the Elastic Business Process Management (eBPM). These approaches are finding models, techniques and…
Evaluating teaching effectiveness at scale remains a persistent challenge for large universities, particularly within engineering programs that enroll tens of thousands of students. Traditional methods, such as manual review of student…
As the amount of scientific data continues to grow at ever faster rates, the research community is increasingly in need of flexible computational infrastructure that can support the entirety of the data science lifecycle, including…
Cloud computing has penetrated the Information Technology industry deep enough to influence major companies to adopt it into their mainstream business. A strong thrust on the use of virtualization technology to realize…
With the development of cloud computing, service computing, IoT(Internet of Things) and mobile Internet, the diversity and sociality of services are increasingly apparent. To meet the customized user demands, Service Ecosystem is emerging…
Computer networks today typically do not provide any mechanisms to the users to learn, in a reliable manner, which paths have (and have not) been taken by their packets. Rather, it seems inevitable that as soon as a packet leaves the…