Related papers: A Brief History of Web Crawlers
To protect users' privacy, legislators have regulated the usage of tracking technologies, mandating the acquisition of users' consent before collecting data. Consequently, websites started showing more and more consent management modules --…
The web graph is a commonly-used network representation of the hyperlink structure of a website. A network of similar structure to the web graph, which we call the session graph has properties that reflect the browsing habits of the agents…
With the advance of Web Services technologies and the emergence of Web Services into the information space, tremendous opportunities for empowering users and organizations appear in various application domains including electronic commerce,…
Event collections are frequently built by crawling the live web on the basis of seed URIs nominated by human experts. Focused web crawling is a technique where the crawler is guided by reference content pertaining to the event. Given the…
Browsers and their users can be tracked even in the absence of a persistent IP address or cookie. Unique and hence identifying pieces of information, making up what is known as a fingerprint, can be collected from browsers by a visited…
Achieving accessibility compliance is extremely important for many government agencies and businesses who wish to improve services for their consumers. With the growing reliance on dynamic web applications many organizations are finding it…
The web browser is one of the major channels to access the Internet on mobile devices. Based on the smartphone usage logs from millions of real-world Android users, it is interesting to find that about 38% users have more than one browser…
Users' detailed browsing activity - such as what sites they are spending time on and for how long, and what tabs they have open and which one is focused at any given time - is useful for a number of research and practical applications.…
Web usage mining is a type of web mining, which exploits data mining techniques to discover valuable information from navigation behavior of World Wide Web users. As in classical data mining, data preparation and pattern discovery are the…
Web Engineering is the application of systematic, disciplined and quantifiable approaches to development, operation, and maintenance of Web-based applications. It is both a pro-active approach and a growing collection of theoretical and…
The success of generative AI relies heavily on training on data scraped through extensive crawling of the Internet, a practice that has raised significant copyright, privacy, and ethical concerns. While few measures are designed to resist a…
Modern web scraping struggles with dynamic, interactive websites that require more than static HTML parsing. Current methods are often brittle and require manual customization for each site. To address this, we introduce Webscraper, a…
The World Wide Web is a vast and continuously changing source of information where searching is a frequent, and sometimes critical, user task. Searching is not always the user's primary goal but an ancillary task that is performed to find…
As the digital world evolves, so do the threats to our security do too. Keyloggers were once a large threat to the cyber world. Though undergoing many transformations alongside the technological advancements of today, it is important to…
Over the years, with the advancement of technology, Web technology has many improvements. In the early days, the web was one-way communication, and only the customer was able to see the content of the site and could not enter information.…
Web crawling projects or design is significant in the current information age. Using the web spider or crawler can automatically search and collect a huge amount of internet information. As one of the most popular web crawler frameworks,…
In the realm of cybersecurity, phishing stands as a prevalent cyber attack, where attackers employ various tactics to deceive users into gathering their sensitive information, potentially leading to identity theft or financial gain.…
In this paper we argue that policies are an increasing concern for organizations that are operating a web site. Examples of policies that are relevant in the domain of the web address issues such as privacy of personal data, accessibility…
The paper analyses current versions of top three used Internet browsers and compare their security levels to a research done in 2006. The security is measured by analyzing how user data is stored. Data recorded during different browsing…
The vastness of the web imposes a prohibitive cost on building large-scale search engines with limited resources. Crawl frontiers thus need to be optimized to improve the coverage and freshness of crawled content. In this paper, we propose…