go_spider: An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to a Individualized crawler easily or can only use the default crawl components.……
crawler4j: crawler4j is an open source web crawler for Java which provides a simple interface for crawling the Web. Using it, you can setup a multi-threaded web crawler in few minutes.……