Spider trap

A spider trap (or crawler trap) is a set of web pages that may intentionally or unintentionally be used to cause a web crawler or search bot to make an infinite number of requests or cause a poorly constructed crawler to crash. Web crawlers are also called web spiders, from which the name is derived. Spider traps may be created to "catch" spambots or other crawlers that waste a website's bandwidth. They may also be created unintentionally by calendars that use dynamic pages with links that continually point to the next day or year.

Common techniques used are:

• creation of indefinitely deep directory structures like

  http://foo.com/bar/foo/bar/foo/bar/foo/bar/.....
  

• dynamic pages like calendars that produce an infinite number of pages for a web crawler to follow.
• pages filled with a large number of characters, crashing the lexical analyzer parsing the page.
• pages with session-id's based on required cookies.

There is no algorithm to detect all spider traps. Some classes of traps can be detected automatically, but new, unrecognized traps arise quickly.

Politeness

A spider trap causes a web crawler to enter something like an infinite loop, which wastes the spider's resources, lowers its productivity, and, in the case of a poorly written crawler, can crash the program. Polite spiders alternate requests between different hosts, and don't request documents from the same server more than once every several seconds, meaning that a "polite" web crawler is affected to a much lesser degree than an "impolite" crawler.

In addition, sites with spider traps usually have a robots.txt telling bots not to go to the trap, so a legitimate "polite" bot would not fall into the trap, whereas an "impolite" bot which disregards the robots.txt settings would be affected by the trap.

See also

• Robots exclusion standard
• Web crawler

This World Wide Web-related article is a stub. You can help Wikipedia by expanding it.v · Categories: World Wide Web World Wide Web stubs Wikimedia Foundation. 2010. Игры ⚽ Нужно решить контрольную? Batha River Number one modern rock hits of 1988 Look at other dictionaries: Spider trap — Eine Spider trap (wörtlich „Spinnen Falle“) ist eine Web Struktur, die unerwünschte Webcrawler an der Erfassung der Inhalte einer Website hindern soll. Das Ziel ist, unerwünschte Webcrawler, die Spam verbreiten oder Sicherheitslücken ausfindig… …   Deutsch Wikipedia trap-door spider — noun American spider that constructs a silk lined nest with a hinged lid • Hypernyms: ↑spider • Member Holonyms: ↑Ctenizidae, ↑family Ctenizidae * * * ˈ ̷ ̷ˌ ̷ ̷ noun : any of various often large burrowing spiders especially of the family… …   Useful english dictionary spider — I (Roget s IV) n. 1. arachnid, spinner, harvestman, daddy longlegs. Common spiders include: tarantula, black widow, garden, diadem, grass, trapdoor, crab, water, bird, brown, hermit, wolf, jumping, burrowing, hunting, violin back, brown recluse,… …   English dictionary for students Spider-Man Classics — is a line produced by Toy Biz focusing on Spider Man and his allies and enemies. The line has over 15 series, with dozens of Spider Man incarnations and many of his famous enemies. Series One came out in 2001. HistorySpider Man Classics is a term …   Wikipedia Spider-Woman (Jessica Drew) — Spider Woman Cover to Spider Woman #1. Art by Joe Sinnott. Publication information Publisher Marvel Comi …   Wikipedia SPIDER — (Heb. עַכָּבִישׁ, akkavish). Isaiah (59:5–6) compares the evil designs of those who plot against the righteous to the webs which the spider spins to trap insects, while Job (8:14–15) compares the house of the wicked to the spider s fragile web.… …   Encyclopedia of Judaism trap-door spider — trap′ door spi′der n. ivt any of several burrowing spiders, esp. of the family Ctenizidae, that build a tubular, lidded nest • Etymology: 1820–30 …   From formal English to slang Spider — Taxobox name = Spiders image width = 250px image caption = an Orb weaver spider, Family: Araneidae regnum = Animalia phylum = Arthropoda subphylum = Chelicerata classis = Arachnida ordo = Araneae ordo authority = Clerck, 1757 diversity link =… …   Wikipedia spider — spiderless, adj. spiderlike, adj. /spuy deuhr/, n. 1. any of numerous predaceous arachnids of the order Araneae, most of which spin webs that serve as nests and as traps for prey. 2. (loosely) any of various other arachnids resembling or… …   Universalium trap-door spider — any of several burrowing spiders, of the family Ctenizidae, that construct a tubular nest with a hinged lid. [1820 30] * * *       any member of the spider family Ctenizidae and certain members of the families Antrodiaetidae, Actinopodidae, and… …   Universalium 18+ © Academic, 2000-2024 Contact us: Technical Support, Advertising Dictionaries export, created on PHP, Joomla, Drupal, WordPress, MODx. Mark and share Search through all dictionaries Translate… Search Internet Share the article and excerpts Direct link … Do a right-click on the link above and select “Copy Link”