User-agent: Twitterbot Disallow: /en-temps-real.html User-Agent: trendkite-akashic-crawler Disallow: / # Horrible bandwidth eating robots User-Agent: AiHitBot User-Agent: AlphaSeoBot User-Agent: AlphaSeoBot-SA User-Agent: BLEXBot User-Agent: CareerBot User-Agent: Cliqzbot User-Agent: MegaIndex User-Agent: Owlin User-Agent: Owlin Bot User-Agent: Owlin Bot v. 3.0 User-Agent: QuerySeekerSpider User-Agent: SEOkicks-Robot User-Agent: Semrush User-Agent: SemrushBot User-Agent: Sogou User-Agent: TkBot User-Agent: Vagabondo User-Agent: WBSearchBot User-Agent: dotbot User-Agent: grapeshot User-Agent: iCjobs User-Agent: linkdexbot User-Agent: magpie-crawler User-Agent: proximic User-Agent: scrapy User-Agent: scrapybot User-Agent: sentibot User-agent: AdIdxBot User-agent: AhrefsBot User-agent: BLP_bbot User-agent: BLP_bbot/0.1 User-agent: Baiduspider User-agent: Baiduspider+ User-agent: Baiduspider-image+ User-agent: BingPreview User-agent: Exabot/3.0 User-agent: MJ12bot User-agent: NaverBot User-agent: SemrushBot User-agent: Snapbot/1.0 User-agent: Yandex User-agent: YandexBot User-agent: Yeti User-agent: YoudaoBot User-agent: dtSearchSpider User-agent: ichiro User-agent: magpie-crawler User-agent: moget User-agent: sogou spider User-agent: Screaming Frog SEO Spider Disallow: / User-agent: * Disallow: /administrator/ Disallow: /cache/ Disallow: /components/ Disallow: /images/ Disallow: /includes/ Disallow: /installation/ Disallow: /language/ Disallow: /libraries/ Disallow: /media2/ Disallow: /modules/ Disallow: /plugins/ Disallow: /templates2/ Disallow: /tmp/ Disallow: /xmlrpc/ Disallow: /cgi-bin/ Disallow: /serveirestringit.html Disallow: /*.raw Disallow: /edicio/ Disallow: /*/*.raw Disallow: /*/*.raw?* Disallow: *.jpg?ESBORRAM*