Ferramenta de web scraping para produtos e-commerce

Abstract

The use of the internet is increasingly frequent in people's daily lives. In particular, e-commerce has gradually established itself as a highly popular business transaction segment. This huge digital market is object of study for several technological implementations. Scraping allows the extraction of data from e-commerce sites, converting them into structured information for the most diverse types of analysis. Scraping applications with specific functionality tend to become proprietary software and difficult to use for teaching. This work developed a prototype for scraping e-commerce products, with open source, customizable, scalable and that demonstrates all areas of study involved. The work described the structure and components of an e-commerce site, the components of a web scraping from two different perspectives and the manipulation of communication between the prototype and the sites. CSS Selectors techniques for data scraping and JSON API parsing were described and implemented, as well as an open-source base for a user to use any other technique, if desired. The features developed are registration and customizable configuration of each scraper, scraper testing, scraper execution, consultation of product data obtained in reports and in price comparison. The study tested the hypothesis that computational parallelism improves data collection performance. After implementation, the scrapers execution tests showed that the more processes are used, the lower the execution time, although the difference between each performance decreases as more processes are added. In general, this study provided a description of the functioning of a web scraper from conception to data analysis