arachnode.net

An open source .NET web crawler written in C# using SQL 2005/2008. Key Features .NET architecture arachnode.net is the most comprehensive open source C#/.NET site crawler available. Use arachnode.net from any .NET language. Configurable Rules and ActionsImplement custom pre- and post-request crawl rules and actions without source recompilation. The existing crawl rules and actions architecture easily enables crawling enhancements such as federation, partitioning and distributed caching. Lucene.NET IntegrationLucene.NET integration allows for full-text search through a familiar web interface. Easily integrate your search results into Solr or other Lucene index utilization solutions, whether they be in .NET, Java or any other language that supports Lucene. SQL Server 2005/2008 and full-text indexingSQL Server 2005/2008 full-text indexing is configured at all appropriate content storage locations for files, images and web pages. .DOC/.PDF/.PPT/.XLS IndexingCrawl, index and search Microsoft Word, PowerPoint and Excel and Adobe PDF documents. The indexing architecture is easily understood and customization is simple. HTML to XML and XHTMLDownloaded WebPages can be converted to XML stored in SQL Server 2005/2008 through the HtmlAgilityPack. Use xpath to Extract common elements from downloaded content using the pre-configured XML indexes. Multi-threading and ThrottlingArachnode.net can be configured to run any number of threads and to use as much or as Little processor time and memory. Respectful Crawling arachnode.net provides pre- and post-request rules governing address and content filtering, robots.txt behavior, request Frequency and crawl depth. The Default crawling environment is respectful, courteous and kind. Analysis Services arachnode.net comes with over 250 stored procedures, views and functions designed for use with Anaylsis Services and other business intelligence software. These procedures and views address trending, popularity, term ext