The search for the leecher!
KingOfMyCastle.com had some strange web stats last month. They said that human visitors had used up nearly half of my webspace but an Unknown robot (identified by ‘crawl’) had used up even more. These robots are created by companies like Google, Ask Jeeves and MSN, and you need them to visit your site so that it appears on search engines. But, there was one robot that used up 55% of my complete webspace! And he was unknown, unlike the Google ones. Webspace costs money so I decide to hunt the fella down (with some help from Kerry).
First I had a look at the last 300 visitors to the site. Around 250 of them were the same robot. Its identity was Agent: gsa-crawler (Enterprise; GIX-01873; *followed by three email address.) – I was getting closer, as I now had their eMail address.
Then Kerry showed me the program known as Webaliser. This gives you even more detailed statistics about who has been using up all my bandwidth. The program highlighted one place as responsible for all the use, Hostname IP address: 217.154.230.114 – You can then type this into your explorer bar and it will take you to p2b.net – a new search engine for Brighton and Hove (read the story here). It also contains a name which is the same as one of those eMail address above – and a phone number. Gotcha!
Kerry suggests that they are leeching my entire site. And not only once. The company is called www.scip.org.uk. Looks like I’ll have to give them a call in the morning. I wont be too angry though as it appears that they’re recruiting! ;)
They _could_ be leeching the site, but my guess is that there is some kind of misconfiguration going on causing the Google Search Appliance to get stuck in a bad loop… Calling them leechers makes for a much better headline though! Happy hunting, and remember to apply lots of salt.
I left a message for one of the guys on the phone. And have sent all three of them an eMail explaining the problem. Hopefully I’ll get some answers soon.
Looks like its sorted now. For the last four days I’m back to usual bandwidth levels. The search engine is still looking at my site, I think, but just the way all the others do. Thanks for sorting.