WebCrawler is a C# console application that recursively scans a website starting from a given URL, collects all discovered links, and saves them to a file. It’s useful for site mapping, link analysis, and content discovery.
github git url cli commandline website crawler opensource web grabber web-crawler web-crawling commandline-interface commandline-tool crawler-engine crawling-sites crawler-csharp
-
Updated
May 29, 2025 - C#