Blocking IPs and Visitors by Country

Before we start, one word about orientation. On the right, you will notice links to the multiple pages in this series about website optimization. If you haven't done so yet, I suggest you also visit my main page about website optimization. 

Blocking IPs and Visitors by Country

If you inspect your web traffic, you may have noticed a surprising number of visitors from countries such as China, Russia, Poland, India and Brazil. Maybe your website only offers local information about your small town in Louisiana. Can that many people from overseas can be interested in your local news? 

The answers hides in your website's error logs. There, you will see that many of these visitors are trying to access files and folders that do not exist, hoping to hack into your site, perhaps to turn your server into a spamming zombie. 

Some webmasters decide to block all traffic from a number of handpicked countries, sacrificing the few legitimate visitors from China who were looking at your site because they planned to fly in next week to catch a movie at the community hall. 

There are several ways to deny traffic from a particular country. To my knowledge, all these methods rely on matching the visitor's IP address to some kind of IP geolocation table. 

None of the solutions are perfect: determined visitors will be able to access your site through proxies. 

Method 1: Maintaining a List on Your Server

To preface, let me state that this is by far my least favorite method. 

There are several sites with free lists of IPs for specific countries, ready to download, sites such as Okean for Chinese and Korean IPs, Wizcrafts for Chinese, Russian and Nigerian IPs, Country IP Blocks for a full list of countries, and BlockACountry for another huge list of countries. I don't know how well any of these lists is maintained. 

Once you have a list, you can block the IPs in htaccess or httpd.conf. In my view, this is a cumbersome way of blocking visitors from certain countries, because IPs change all the time. Do you want to update your IP country table all the time? 

If you did want to use this method, you might add lines like these to your .htaccess:

Order allow,deny Allow from all Deny from 58.14.0.0/15 Deny from 58.16.0.0/16 … etc. (huge list of IP addresses or ranges)

Method 2: Custom Rule in CloudFlare

Update: I do not recommend this method anymore. Click the link to read why I no longer use CloudFlare. I will leave this material here for reference. 

On my page about how to block traffic from certain IPs, I explain the free CloudFlare service, in Beta as of Q3 2010. 

On a CloudFlare-enabled site, you can add custom rules to block specific IPs or even specific countries. It is fast to implement, and it is extremely convenient because you don't need to update the list of IPs for the countries that you want to block. CloudFlare does that for you. 

This method is perfect if you want to use the same rules across all sites. However, it may be that you want to block Russia across most sites, but that one of your sites happens to have many legitimate visitors from Russia. For such situations, you can use Method 3. 

Method 3: CloudFlare plus Apache Configuration

If you have several sites and need greater control over which countries you block for each site, you can take advantage of a feature of CloudFlare: the CF-IPCountry variable it inserts in request headers, a variable that contains the geolocation data looked up by CloudFlare. 

You can then tweak Apache configuration files to block visitors based on CF-IPCountry. 

Flavor 1: Simplest Technique on Shared Hosts
For this technique, you only edit the ".htaccess" file at the root of your web pages. Paste the following at the top of htaccess, just below the "AddHandler" line if any: 

SetEnvIf CF-IPCountry CN BuzzOff=1
SetEnvIf CF-IPCountry RU BuzzOff=1
SetEnvIf CF-IPCountry IN BuzzOff=1
Order allow,deny
Allow from all
Deny from env=BuzzOff

In this example, notice "CN", "RU" and "IN" in the first three lines. These stand for China, Russia and India. Edit those to suit your needs (and add or remove as many lines as you like) by consulting the list of internet country abbreviations. 

Depending on your needs, you can create different htaccess code for each of your sites. 

I suggest you test the implementation by blocking your own country on one of your sites (but make sure you have FTP access to remove the block from your htaccess!) 

Flavor 2: Simple Technique for Many Countries on Multiple Sites (Private Hosts only)
Here is an implementation that works great when you maintain a large list of countries that you want to block on multiple sites. For this implementation, you need access to httpd.conf, which is rarely the case on shared hosts. 

First, in httpd.conf, paste something like the following before the VirtualHost section, editing the example to add whichever countries you want to block and remove whichever countries you do not want to block. To choose which countries to ban, visit a list of country abbreviations. 

SetEnvIf CF-IPCountry BD BuzzOff=1
SetEnvIf CF-IPCountry CN BuzzOff=1
SetEnvIf CF-IPCountry HR BuzzOff=1
SetEnvIf CF-IPCountry HU BuzzOff=1
SetEnvIf CF-IPCountry ID BuzzOff=1
SetEnvIf CF-IPCountry IN BuzzOff=1
SetEnvIf CF-IPCountry LU BuzzOff=1
SetEnvIf CF-IPCountry LV BuzzOff=1
SetEnvIf CF-IPCountry PH BuzzOff=1
SetEnvIf CF-IPCountry PK BuzzOff=1
SetEnvIf CF-IPCountry PL BuzzOff=1
SetEnvIf CF-IPCountry RO BuzzOff=1
SetEnvIf CF-IPCountry RU BuzzOff=1
SetEnvIf CF-IPCountry SI BuzzOff=1
SetEnvIf CF-IPCountry SK BuzzOff=1
SetEnvIf CF-IPCountry TH BuzzOff=1
SetEnvIf CF-IPCountry TW BuzzOff=1
SetEnvIf CF-IPCountry UA BuzzOff=1
SetEnvIf CF-IPCountry VN BuzzOff=1

Then, for each of the websites for which you want to block these countries, open your .htaccess and paste these lines near the top, right below the "AddHandler" line if any: 

Order allow,deny
Allow from all
Deny from env=BuzzOff

Below, there is a more complex example for situations where a country is not always blocked.

Don't forget to restart Apache! Here's a tutorial on SetEnvIf if you'd like to tweak the code. And here's some reading on the Order allow,deny directive. 

Flavor 3: Blocking a Country on All Sites Except One
Let's say you want to block Russia on all sites except one. 

Using "Flavor 2" from above, in httpd.conf, instead of the above, you would have something like: 

SetEnvIf CF-IPCountry RU IsRussia=1
SetEnvIf CF-IPCountry CN BuzzOff=1
SetEnvIf CF-IPCountry IN BuzzOff=1
… (other countries to be blocked)

For sites where you don't want to block Russia, your .htaccess would look like this: 

Order allow,deny
Allow from all
Deny from env=BuzzOff

For sites where you do want to block Russia, your .htaccess would look like this: 

Order allow,deny
Allow from all
Deny from env=IsRussia
Deny from env=BuzzOff

This example should give you an idea of how to further customize country blocking for each site. 

China (Russia, etc) is Still Showing in my CloudFlare Threats!
If you have used Method 3, it is normal that your blocked countries will still appear in the list of threats of your CloudFlare dashboard. Why? CloudFlare is your first line of defense. Visitors from China first go to the CloudFlare DNS. There, they may be challenged (and show in your Threats Panel).

The visitors that CloudFlare doesn't block are routed to your server, where they should be blocked by the techniques shown above. Therefore, such visitors should not show in stats reports of pages actually seen, such as those produced by Google Analytics.

The dmesg Command

The dmesg command is used to write the kernel messages in Linux and other Unix-like operating systems to standard output(which by default is the display screen).

A kernel is the core of an operating system. It is the first part of the operating system that is loaded into memory when a computerboots up (i.e., starts up), and it controls virtually everything on a system. The numerous messages generated by the kernel that appear on the display screen as a computer boots up show the hardware devices that the kernel detects and indicate whether it is able to configure them.

dmesg obtains its data by reading the kernel ring buffer. A buffer is a portion of a computer's memory that is set aside as a temporary holding place for data that is being sent to or received from an external device, such as a hard disk drive (HDD), printer or keyboard. A ring buffer is a buffer of fixed size for which any new data added to it overwrites the oldest data in it.

dmesg can be very useful when troubleshooting or just trying to obtain information about the hardware on a system. Its basic syntax is

dmesg [options]

Invoking dmesg without any of its options (which are rarely used) causes it to write all the kernel messages to standard output. This usually produces far too many lines to fit into the display screen all at once, and thus only the final messages are visible. However, the output can be redirected to the less command through the use of a pipe (designated by the vertical bar character), thereby allowing the startup messages to be viewed one screenful at a time:

dmesg | less

less allows the output to be moved forward one screenful at a time by pressing the SPACE bar, backward by pressing the b key and removed by pressing the q key. (The more command could have been used here instead of the less command; however, lessis newer than more and has additional functions, including the ability to return to previous pages of the output.)

When a user encounters a problem with the system, it can be convenient to write the output of dmesg to a file and then send that file by e-mail to a system administrator or other knowledgeable person for assistance. For example, the output could be redirected to a file named boot_messages using the output redirection operator (designated by a rightward facing angle bracket) as follows:

dmesg > boot_messages

Because of the length of the output of dmesg, it can be convenient to pipe its output to grep, a filter which searches for any lines that contain the string (i.e., sequence of characters) following it. The -i option can be used to tell grep to ignore the case (i.e.,lower case or upper case) of the letters in the string. For example, the following command lists all references to USB (universal serial bus) devices in the kernel messages:

dmesg | grep -i usb

And the following tells dmesg to show all serial ports (which are represented by the string tty):

dmesg | grep -i tty

The dmesg and grep combination can also be used to show how much physical memory (i.e., RAM) is available on the system:

dmesg | grep -i memory

The following command checks to confirm that the HDD(s) is running in DMA (direct memory access) mode:

dmesg | grep -i dma

The output of dmesg is maintained in the log file /var/log/dmesg, and it can thus also be easily viewed by reading that file with atext editor, such as vi or gedit, or with a command such as cat, e.g.,

cat /var/log/dmesg | less

How to fix the jftp error in joomla JFTP::store: Bad response – Warning! – Failed to move file.

While trying to upload/install some components or modules  in Joomla , you may get the following error!

ERROR:

JFTP::store: Bad response. Warning! – Failed to move

Dont get panic!. There is a simple work around for this issue. This happens when the ftp of Joomla doesn’t work properly. You may fix this issue by just disabling the ftp in the file ‘configuration.php’

Open the configuration.php file using  an editor like VI

root@ssages [/home/indisage/www]# vi configuration.php

Then search for the following line :

var $ftp_enable = ’1?;

change 1 to 0 to fix the issue.

Alternatevily you can disable it from the global configuration option–> server in Joomla

Clamscan Script

Often with a compromised web server it’s always nice to set the infected files to be immutable.   Here’s my way of doing things.

Starting with an install of clamAV and running an update

yum install clamscan -y

freshclam

Scan to file and send to background

clamscan /var/www/vhosts/ -r > ~/clamscan &

Search for infected files (FOUND) and set them to be immutable

 chmod 000 `grep FOUND ~/clamscan.20130319 | awk ‘{print $1}’ | cut -d: -f1`

how to Enable wildcard subdomains

When you install WPMU by default the subdomain option is checked. This is for blogs in the format of username.yourdomain.com.

There are two steps that need to be done to your server in order for this to work.

In Apache, there needs to be a line in the domain’s virtual hosts section as follows:

ServerName yourdomain.com
ServerAlias yourdomain.com *.yourdomain.com
DocumentRoot /your/doc/root/
ServerAdmin webmaster@yourdomain.com

** If you’re running an Nginx server, you’d use:

server {
listen 80;
server_name domain.com *.yourdomain.com;
}

The Server Alias line with the *.yourdomain.com is what controls this. If you do not have access to change this, ask your webhost for support. More and more hosts are enabling this by default.

The second thing that needs to be done is adding the DNS record. Where you add it depends on your domain hosting. Basically, you need to add a line like this:

*. in A YOURIPADDRESS

These wildcards mean that any subdomain request will fall through to the WPMU install, where MU will look up the name in the database. If it finds it, it serves up the blog.

Note: if you choose the subfolder option on installation, you do not have to do either one of these steps.

Resolving Domain Park Wrapper Errors

If you run a more populated shared hosting server, sooner or later you’ll have received complaints about the dreaded park wrapper errors in cPanel that occur when a user tries to add a parked or addon domain to their cPanel. The errors may look like this:

Error from domain wrapper: domain.com is owned by another user.

Error from domain wrapper: Domain already exists, it was not added.

I’ve never really been able to attribute that error to a specific action, but my assumption is that it occurs as a result of the end user not completely removing the domain from their cPanel (i.e. hitting esc or closing the browser during removal), therefore not allowing cPanel to remove the domain’s entries to allow that domain to be re-added when certain security settings are enabled in WHM > Tweak Settings.

The easy solution to the first error is to enable the option for users to add domains owned by other users via WHM > Tweak Settings. But this is a very bad idea as it essentially allows users to repoint domains that you’re already hosting.

If you’re attempting to re-add a domain to a cPanel account and are getting one of the above errors, first check that the error needs to be corrected. Meaning, make sure that the domain in question isn’t already set up elsewhere. If it is, you would need to remove it from that account before being able to add it to another.

If the error is actually occurring due to an improperly removed domain, follow the below steps until you are able to add the domain back to the server:

1. run /scripts/killdns  on the server to remove the DNS records from the DNS cluster
2. do grep -r /var/cpanel/users to see if it exists in a user file, and if so, delete the entry and run /scripts/updateuserdomains, make sure it’s remove from /etc/userdomains
3. grep -r /var/cpanel/userdata to see if the domain appears in a user’s template. If so, remove any files based on the domain name (including .cache files), and remove any subdomain/parked/addon domain entries for that domain from the ‘main’ file located in that user’s folder (i.e. /var/cpanel/templates/username/main), then /scripts/rebuildhttpconf to remove it from httpd.conf .

This should allow you to re-add the domain name to the user’s cPanel without getting the park wrapper error.

Updating to WP 3.2 generates a fatal error code

Fatal error: Call to undefined method Arras_Widget_Tag_Cloud::WP_Widget_Tag_Cloud() in /home/XXXXX/public_html/wp-content/themes/arras/library/widgets.php on line 404/405

To fix this, simply replace line 404 of /library/widgets.php from:

function Arras_Widget_Tag_Cloud() {
    $this->WP_Widget_Tag_Cloud();
}

to:

function Arras_Widget_Tag_Cloud() {
    parent::__construct();
}