Related papers: Google Scholar Metrics 2013: nothing new under the…
We argue that the current practice of evaluating AI/ML time-series forecasting models, predominantly on benchmarks characterized by strong, persistent periodicities and seasonalities, obscures real progress by overlooking the performance of…
Deep neural networks have shown remarkable performance when trained on independent and identically distributed data from a fixed set of classes. However, in real-world scenarios, it can be desirable to train models on a continuous stream of…
We review our recent work on applying the Google PageRank algorithm to find scientific "gems" among all Physical Review publications, and its extension to CiteRank, to find currently popular research directions. These metrics provide a…
Academic codes associated with research papers are valuable resources for scholars. In specialized fields outside computer science, code availability is often limited, making effective code retrieval essential. Google Scholar is a crucial…
Smartphone manufacturers continue to release new models annually, yet the pace of meaningful innovation has slowed, with most changes limited to incremental updates in design, performance, or software. This study examines whether such…
The performance of Recommender Systems (RS) varies significantly across users, yet the underlying reasons for this variance remain poorly understood. This paper introduces a unified framework to analyze and explain this performance gap by…
Automated evaluation metrics as a stand-in for manual evaluation are an essential part of the development of text-generation tasks such as text summarization. However, while the field has progressed, our standard metrics have not -- for…
ResearchGate has emerged as a popular professional network for scientists and researchers in a very short span of time. Similar to Google Scholar, the ResearchGate indexing uses an automatic crawling algorithm that extracts bibliographic…
Systematic Literature Reviews (SLRs) are a widely employed research method in software engineering. However, there are several problems with SLRs, including the enormous time and effort to conduct them and the lack of obvious impacts of SLR…
Through academic publications, the authors of these publications form a social network. Instead of sharing casual thoughts and photos (as in Facebook), authors pick co-authors and reference papers written by other authors. Thanks to various…
Data quality is a key element for building and optimizing good learning models. Despite many attempts to characterize data quality, there is still a need for rigorous formalization and an efficient measure of the quality from available…
Meeting summarization has become a critical task considering the increase in online interactions. While new techniques are introduced regularly, their evaluation uses metrics not designed to capture meeting-specific errors, undermining…
Novel scientific knowledge is constantly produced by the scientific community. Understanding the level of novelty characterized by scientific literature is key for modeling scientific dynamics and analyzing the growth mechanisms of…
Large Language Models (LLMs) evaluation is a patchy and inconsistent landscape, and it is becoming clear that the quality of automatic evaluation metrics is not keeping up with the pace of development of generative models. We aim to improve…
This study aims to information security in academic information systems to provide recommendations for improvements in information security management by the expected maturity level based on ISO/IEC 27002:2013. By using a qualitative…
To continuously improve quality and reflect changes in data, machine learning applications have to regularly retrain and update their core models. We show that a differential analysis of language model snapshots before and after an update…
Surveying prior literature to establish a foundation for new knowledge is essential for scholarly progress. However, survey articles are resource-intensive and challenging to create, and can quickly become outdated as new research is…
Measuring science is based on comparing articles to similar others. However, keyword-based groups of thematically similar articles are dominantly small. These small sizes keep the statistical errors of comparisons high. With the growing…
Bibliographic metrics are commonly utilized for evaluation purposes within academia, often in conjunction with other metrics. These metrics vary widely across fields and change with the seniority of the scholar; consequently, the only way…
How does the published scientific literature used by scientific community? Many previous studies make analysis on the static usage data. In this research, we propose the concept of dynamic usage data. Based on the platform of…