[XML-DEV Mailing List Archive Home] [By Thread] [By Date] [Recent Entries] [Reply To This Message] Re: Re: web crawling (was: Re: HGRAB. Syndication. G
----- Original Message ----- From: "Paul T" <pault12@p...> To: "K. Ari Krupnikov" <ari@c...> Cc: <xml-dev@l...> Sent: Tuesday, January 08, 2002 9:13 PM Subject: Re: web crawling (was: Re: HGRAB. Syndication. Google. Grey area.) > <aside> > I've looked at many robots.txt files and nobody > disallows the /. Maybe there are some especial websites > that *do* that, but http://www.metasystema.org/terms.mhtml > looks like a *very* rare example to me. But that's > irrelevant, because you make a stronger point. > </aside> http://www.kpmg.com/robots.txt -- THINGS TO DO IF I BECOME AN EVIL OVERLORD #71 If I decide to test a lieutenant's loyalty and see if he/she should be made a trusted lieutenant, I will have a crack squad of marksmen standing by in case the answer is no. _________________________________________________________ Do You Yahoo!? Get your free @yahoo.com address at http://mail.yahoo.com
|
PURCHASE STYLUS STUDIO ONLINE TODAY!Purchasing Stylus Studio from our online shop is Easy, Secure and Value Priced! Download The World's Best XML IDE!Accelerate XML development with our award-winning XML IDE - Download a free trial today! Subscribe in XML format
|