Alcuni di questi robot sono progettati per copiare tutto siti web, che possono utilizzare una grande quantità di larghezza di banda. Altri raccolto gli indirizzi e-mail per gli spammer. La maggior parte dei benefici dei rispettivi proprietari, offrendo al tempo stesso assolutamente alcun beneficio per il proprietario del sito web. Si deve decidere per te quello che si qualifica come una cattiva web robot.
Ecco una lista dei robot che si può desiderare di escludere, in User-agent: SiteSnagger
User-agent: SlySearch
User-agent: SmartDownload
User-agent: SmartDownload/1.2.76 (Win32; 1 aprile 1999)
User-agent: SmartDownload/1.2.77 (Win32; 17 agosto 1999)
User-agent: SmartDownload/1.2.77 (Win32; 1 febbraio 2000)
User-agent: SmartDownload/1.2.77 (Win32; 19 giugno 2001)
User-agent: SpankBot
User-agent: Sqworm/2.9.85-BETA (beta_release; 20011115-775; i686-pc-linux
User-agent: SuperBot
User-agent: SuperBot/3.0 (Win32)
User-agent: SuperBot/3.1 (Win32)
User-agent: SuperHTTP
User-agent: SuperHTTP/1.0
User-agent: Surfbot
User-agent: Szukacz/1.4
User-agent: Teleport
User-agent: Teleport Pro
User-agent: Teleport Pro/1.29
User-agent: Teleport Pro/1.29.1590
User-agent: Teleport Pro/1.29.1634
User-agent: Teleport Pro/1.29.1718
User-agent: Teleport Pro/1.29.1820
User-agent: Teleport Pro/1.29.1847
User-agent: TeleportPro
User-agent: TELESOFT
User-agent: Il Intraformant
User-agent: The.Intraformant
User-agent: TheNomad
User-agent: TightTwatBot
User-agent: Titan
User-agent: True_Robot
User-agent: True_Robot/1.0
User-agent: TurnitinBot
User-agent: TurnitinBot/1.5
User-agent: URL di controllo
User-agent: URL_Spider_Pro
User-agent: URLy Avvertenza
User-agent: URLy.Warning
User-agent: VCI
User-agent: VCI WebViewer VCI WebViewer Win32
User-agent: VoidEYE
User-agent: WWW-Collector-E
User-agent: WWWOFFLE
User-agent: Web Image Collector
User-agent: Web Sucker
User-agent: Web.Image.Collector
User-agent: WebAuto
User-agent: WebAuto/3.40 (Win98; I)
User-agent: WebBandit
User-agent: WebBandit/3.50
User-agent: WebCapture 2,0
User-agent: WebCopier
User-agent: WebCopier V.2.2
User-agent: WebCopier v2.5
User-agent: WebCopier v2.6
User-agent: WebCopier v2.7a
User-agent: WebCopier v2.8
User-agent: WebCopier v3.0
User-agent: WebCopier v3.0.1
User-agent: WebCopier v3.2
User-agent: WebCopier v3.2a
User-agent: WebEMailExtrac .*
User-agent: WebEnhancer
User-agent: WebFetch
User-agent: WebGo IS
User-agent: WebLeacher
User-agent: WebReaper
User-agent: WebReaper [info@webreaper.net]
User-agent: WebReaper [webreaper@otway.com]
User-agent: WebReaper v9.1 - www.otway.com / webreaper
User-agent: WebReaper v9.7 - www.webreaper.net
User-agent: WebReaper v9.8 - www.webreaper.net
User-agent: WebReaper vWebReaper v7.3 - www, otway.com / webreaper
User-agent: WebSauger
User-agent: WebSauger 1.20b
User-agent: WebSauger 1.20j
User-agent: WebSauger 1.20k
User-agent: WebStripper
User-agent: WebStripper/2.03
User-agent: WebStripper/2.10
User-agent: WebStripper/2.12
User-agent: WebStripper/2.13
User-agent: WebStripper/2.15
User-agent: WebStripper/2.16
User-agent: WebStripper/2.19
User-agent: WebWhacker
User-agent: WebZIP
User-agent: WebZIP/2.75 (http://www.spidersoft.com)
User-agent: WebZIP/3.65 (http://www.spidersoft.com)
User-agent: WebZIP/3.80 (http://www.spidersoft.com)
User-agent: WebZIP/4.0 (http://www.spidersoft.com)
User-agent: WebZIP/4.1 (http://www.spidersoft.com)
User-agent: WebZIP/4.21
User-agent: WebZIP/4.21 (http://www.spidersoft.com)
User-agent: WebZIP/5.0
User-agent: WebZIP/5.0 (http://www.spidersoft.com)
User-agent: WebZIP/5.0 PR1 (http://www.spidersoft.com)
User-agent: WebZIP
User-agent: WebZip/4.0
User-agent: WebmasterWorldForumBot
User-agent: Sito web Quester
User-agent: Sito web Quester - www.asona.org
User-agent: Sito web Quester - www.esalesbiz.com/extra/
User-agent: Sito web estrattore
User-agent: Sito web Extractor (http://www.asona.org)
User-agent: Website.Quester
User-agent: Webster Pro
User-agent: Webster.Pro
User-agent: wget
User-agent: Wget/1.5.2
User-agent: Wget/1.5.3
User-agent: Wget/1.6
User-agent: Wget/1.7
User-agent: Wget/1.8
User-agent: Wget/1.8.1
User-agent: Wget/1.8.1 + cvs
User-agent: Wget/1.8.2
User-agent: Wget/1.9-beta
User-agent: Widow
User-agent: Xaldon WebSpider
User-agent: Xaldon WebSpider 2.5.b3
User-agent: Xenu's
User-agent: Xenu's Link Sleuth 1.1c
User-agent: Zeus
User-agent: Zeus 11389 Webster Pro V2.9 Win32
User-agent: Zeus 11652 Webster Pro V2.9 Win32
User-agent: Zeus 18018 Webster Pro V2.9 Win32
User-agent: Zeus 26378 Webster Pro V2.9 Win32
User-agent: Zeus 30747 Webster Pro V2.9 Win32
User-agent: Zeus 32297 Webster Pro V2.9 Win32
User-agent: Zeus 39206 Webster Pro V2.9 Win32
User-agent: Zeus 41641 Webster Pro V2.9 Win32
User-agent: Zeus 44238 Webster Pro V2.9 Win32
User-agent: Zeus 51070 Webster Pro V2.9 Win32
User-agent: Zeus 51674 Webster Pro V2.9 Win32
User-agent: Zeus 51837 Webster Pro V2.9 Win32
User-agent: Zeus 63567 Webster Pro V2.9 Win32
User-agent: Zeus 6694 Webster Pro V2.9 Win32
User-agent: Zeus 71129 Webster Pro V2.9 Win32
User-agent: Zeus 82016 Webster Pro V2.9 Win32
User-agent: Zeus 82900 Webster Pro V2.9 Win32
User-agent: Zeus 84842 Webster Pro V2.9 Win32
User-agent: Zeus 90872 Webster Pro V2.9 Win32
User-agent: Zeus 94934 Webster Pro V2.9 Win32
User-agent: Zeus 95245 Webster Pro V2.9 Win32
User-agent: Zeus 95351 Webster Pro V2.9 Win32
User-agent: Zeus 97371 Webster Pro V2.9 Win32
User-agent: Zeus Link Scout
User-agent: asterias
User-agent: b2w/0.1
User-agent: cosmo
User-agent: eCatch
User-agent: eCatch/3.0
User-agent: hloader
User-agent: httplib
User-agent: humanlinks
User-agent: ia_archiver
User-agent: larbin
User-agent: larbin (samualt9@bigfoot.com)
User-agent: larbin samualt9@bigfoot.com
User-agent: larbin_2.6.2 (kabura@sushi.com)
User-agent: larbin_2.6.2 (larbin2.6.2 @ unspecified.mail)
User-agent: larbin_2.6.2 (listonATccDOTgatechDOTedu)
User-agent: larbin_2.6.2 (vitalbox1@hotmail.com)
User-agent: larbin_2.6.2 kabura@sushi.com
User-agent: larbin_2.6.2 larbin2.6.2 @ unspecified.mail
User-agent: larbin_2.6.2 larbin@correa.org
User-agent: larbin_2.6.2 listonATccDOTgatechDOTedu
User-agent: larbin_2.6.2 vitalbox1@hotmail.com
User-agent: libWeb / clsHTTP
User-agent: lwp banale
User-agent: lwp-trivial/1.34
User-agent: moget
User-agent: moget/2.1
User-agent: pavuk
User-agent: pcBrowser
User-agent: psbot
User-agent: searchpreview
User-agent: spanner
User-agent: suzuran
User-agent: takeout
User-agent: toCrawl / UrlDispatcher
User-agent: turingos
User-agent: webfetch/2.1.0
User-agent: wget
Disallow: /
Bookmark Quali sono alcuni Web Bad Robot? Ultimi post sul blog