This document is the final project report for our advanced operating system class. During this project, we mainly focused on applying multiprocessing and multi-threading technology to our whole project and utilized the map-reduce algorithm in our data cleaning and data analysis process. In general, our project can be divided into two components: data scraping and data processing, where the previous part was almost web wrangling with employing potential multiprocessing or multi-threading technology to speed up the whole process. And after we collect and scrape a large amount value of data as mentioned above, we can use them as input to implement data cleaning and data analysis, during this period, we take advantage of the map-reduce algorithm to increase efficiency.
Cite
@article{arxiv.2312.15158,
title = {Map-Reduce for Multiprocessing Large Data and Multi-threading for Data Scraping},
author = {Zefeng Qiu and Prashanth Umapathy and Qingquan Zhang and Guanqun Song and Ting Zhu},
journal= {arXiv preprint arXiv:2312.15158},
year = {2023}
}