1 2 Jump to
- Jeremy
- Administrator
Offline 
- From: United States
- Registered: August 30, 2009
- Posts: 2,774
Re: Exclude unknown or certain browsers/operating systems
We are looking into some options for this. Most spiders should obey a robots.txt file, including Baidu, which provides instructions here:
Flag Counter Developer
Boardhost.com, Inc.
- begordon
- Flag Collector
Offline
- Registered: March 16, 2013
- Posts: 57
Re: Exclude unknown or certain browsers/operating systems
Jeremy wrote:
We are looking into some options for this. Most spiders should obey a robots.txt file, including Baidu, which provides instructions here:
I've tried; robots.txt doesn't work at all, or even slow them down, no matter where I put it.
I read the link you posted, and it just occurred to me that maybe robots.txt isn't working because the counters aren't technically on my sites at all; they're actually on s##.flagcounter.com -- and so maybe they wouldn't be protected by a robots.txt that only refers to files located within the sites.
Just a thought...
- •
Thread Starter- begordon
- Flag Collector
Offline
- Registered: March 16, 2013
- Posts: 57
Re: Exclude unknown or certain browsers/operating systems
Also, not long after this started happening, I decided to watch the counter at the top of this forum, and Beijing Municipality has soared from about 2,600 to over 3,100 (a 20% increase) within just the past two months, even though the counter was first put up (apparently) in 2012... so it would appear that at least some other counters are suffering the same sort of mass Baiduspider attack that mine are.
- •
Thread Starter- begordon
- Flag Collector
Offline
- Registered: March 16, 2013
- Posts: 57
Re: Exclude unknown or certain browsers/operating systems
I figured out a possible solution.
The robots.txt won't block a spider from hitting the counter if you just use this:
User-agent: Baiduspider
Disallow: /
...because the disallow only covers files on the site. Instead, it seems that you have to do this:
User-agent: Baiduspider
Disallow:
(where s## represents whatever number server that your flagcounter is on; for example, one of my counters is on s09.flagcounter.com.)
So far, it's been a week and I haven't had any bot hits.
Last edited by begordon (March 29, 2015 5:45 am)
- •
Thread Starter- Jens
- Flag Master
Offline
- From: Dierhagen / Germany
- Registered: January 12, 2010
- Posts: 6,880
Re: Exclude unknown or certain browsers/operating systems
Cool. Thanks for that very valuable information and for the efforts you put in to get that issue resolved.
The views expressed in this statement are in no way intended to represent the views of boardhost.com. The views are my own as moderator of this forum. Since 31 Dec 2016 I'm retired and no longer Moderator of this forum.
- begordon
- Flag Collector
Offline
- Registered: March 16, 2013
- Posts: 57
Re: Exclude unknown or certain browsers/operating systems
Update: it definitely seems to cut down on bot hits, although not all the way. Occasionally (about a week ago, for a few days, for example) a spurt of hits do get through, possibly due to naive servers sending out bots.
Last edited by begordon (April 26, 2015 2:50 pm)
- •
Thread Starter
1 2 Jump to