DotBot

Dotbot is a crawler of Dotnetdotcom.org^[1]. This crawling system is written in a mixture of C and Python.

Dotbot obeys the Robot Exclusion standard^[2], the user-agent string of Dotbot is DotBot/1.1^[3].

References

1. ^ DotbotDotcom.org
2. ^ A Standard for Robot Exclusion
3. ^ User-agent String of Dotbot