• To: "Paul T" <pault12@p...>,"K. Ari Krupnikov" <ari@c...>
• Subject: Re: Re: web crawling (was: Re: HGRAB. Syndication. Google. Grey area.)
• From: "Dare Obasanjo" <kpako@y...>
• Date: Wed, 9 Jan 2002 00:22:48 -0800
• Cc: <xml-dev@l...>
• References: <02e301c19889$99cf3680$2cd0a340@paultx> <3C3B7134.D418F69B@c...> <034801c1989c$21f77470$2cd0a340@paultx> <3C3BC2F3.80B93425@c...> <03fe01c198cc$589dee20$2cd0a340@paultx>

----- Original Message -----
From: "Paul T" <pault12@p...>
To: "K. Ari Krupnikov" <ari@c...>
Cc: <xml-dev@l...>
Sent: Tuesday, January 08, 2002 9:13 PM
Subject:  Re: web crawling (was: Re:  HGRAB. Syndication.
Google. Grey area.)

> <aside>
> I've looked at many robots.txt files and nobody
> disallows the /. Maybe there are some especial websites
> that *do* that, but http://www.metasystema.org/terms.mhtml
> looks like a  *very* rare example to me. But that's
> irrelevant, because you make a stronger point.
> </aside>


http://www.kpmg.com/robots.txt

--
THINGS TO DO IF I BECOME AN EVIL OVERLORD #71
If I decide to test a lieutenant's loyalty and see if he/she should be made
a trusted lieutenant, I will have a crack squad of marksmen standing by
in case the answer is no.


_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com

• References:
  • HGRAB. Syndication. Google. Grey area.
    • From: Paul T <pault12@p...>
  • Re: HGRAB. Syndication. Google. Grey area.
    • From: "K. Ari Krupnikov" <ari@c...>
  • Re: HGRAB. Syndication. Google. Grey area.
    • From: Paul T <pault12@p...>
  • OT: web crawling (was: Re: HGRAB. Syndication. Google. Grey area.)
    • From: "K. Ari Krupnikov" <ari@c...>
  • Re: web crawling (was: Re: HGRAB. Syndication. Google. Greyarea.)
    • From: Paul T <pault12@p...>

• Prev by Date: Re: RE: [ANN] XML Limerick Competition
• Next by Date: RE: [ANN] XML Limerick Competition
• Previous by thread: Re: web crawling (was: Re: HGRAB. Syndication. Google. Greyarea.)
• Next by thread: Re: HGRAB. Syndication. Google. Grey area.
• Index(es):
  • Date
  • Thread