Flag Counter Forum / Exclude unknown or certain browsers/operating systems

**Posted by Jeremy** March 2, 2015 12:54 am

We are looking into some options for this. Most spiders should obey a robots.txt file, including Baidu, which provides instructions here: http://www.baidu.com/search/robots_english.html

**Posted by begordon** March 4, 2015 12:51 am

Jeremy wrote:
We are looking into some options for this. Most spiders should obey a robots.txt file, including Baidu, which provides instructions here: http://www.baidu.com/search/robots_english.html

I've tried; robots.txt doesn't work at all, or even slow them down, no matter where I put it.

I read the link you posted, and it just occurred to me that maybe robots.txt isn't working because the counters aren't technically on my sites at all; they're actually on s##.flagcounter.com -- and so maybe they wouldn't be protected by a robots.txt that only refers to files located within the sites.

Just a thought...

**Posted by begordon** March 7, 2015 5:00 am

Also, not long after this started happening, I decided to watch the counter at the top of this forum, and Beijing Municipality has soared from about 2,600 to over 3,100 (a 20% increase) within just the past two months, even though the counter was first put up (apparently) in 2012... so it would appear that at least some other counters are suffering the same sort of mass Baiduspider attack that mine are.

**Posted by begordon** March 29, 2015 5:44 am

I figured out a possible solution.

The robots.txt won't block a spider from hitting the counter if you just use this:

User-agent: Baiduspider
Disallow: /

...because the disallow only covers files on the site. Instead, it seems that you have to do this:

User-agent: Baiduspider
Disallow: http://s##.flagcounter.com/

(where s## represents whatever number server that your flagcounter is on; for example, one of my counters is on s09.flagcounter.com.)

So far, it's been a week and I haven't had any bot hits.

Last edited by begordon (March 29, 2015 5:45 am)

**Posted by Jens** March 29, 2015 8:14 am

Cool. Thanks for that very valuable information and for the efforts you put in to get that issue resolved.

**Posted by begordon** April 21, 2015 12:29 am

Update: it definitely seems to cut down on bot hits, although not all the way. Occasionally (about a week ago, for a few days, for example) a spurt of hits do get through, possibly due to naive servers sending out bots.

Last edited by begordon (April 26, 2015 2:50 pm)