Scraping large amount of data from different sources – Part 2
In Part 1, we explained different technologies which facilitates scraping and crawling many resources from the web. Although gathering the data from different sources is challenging, storing those data for future access also needs proper design and appropriate technology. In this part, our focus will be on different databases technologies avaliable in the industry. Database As […]
Scraping large amount of data from different sources – Part 1
The source of data for any data centeric application can come from scraping different websites. While there are many third party companies offering web scraping systems, due to sensitivity of information and potential competitors, some companies might be willing to build an in-house package to have the intellectual property of its scraping engine. Although an […]