I was recently helping someone troubleshoot some problems at SearchEngineWatch forums and it struck me that I should write an article about this and other issues.
Because sometimes when you run into a brick wall and can not figure out why your site is not getting listed or even crawled in the search engines it is not always a site issue.
In fact, sometimes it is a server issue which you may or may not have control over. In that case, it does not matter what you do to your site, you just aren’t going to get it listed. This article covers some things to look at and how you can go about fixing them.
Let me start by giving you an idea of what happened with the person on the forum. His site had been up for some time, in fact he recently put up 3 sites. Two of which were getting crawled and indexed and one wasn’t showing up in the engines.
We started by looking at the usual suspects – such as improperly coded robots.txt, poor navigation, banned domain and so on, yet none of these were an issue. The site was very search engine friendly, easily navigable and didn’t use a robots.txt file.
So the next thing I checked was the WHOIS information and on one site I found improper information and on another WHOIS lookup I found no information at all. This lead me to believe that perhaps there was a hosting/server problem at the root of the issue.
So I contacted the site owner and got him to verify with the web host that the DNS information was properly configured. It turns out that it wasn’t. Since the site wasn’t properly resolving via DNS it wasn’t getting crawled by the search engines.
This is just one of the many server issues which can affect the positioning and ranking of your site in the search engines. While DNS resolution and IP configuration are generally the two most common culprits there are several other server issues which can harm your search engine ranking and even cause indexing problems.
DNS issues
Let’s take a closer look at what happened with this site: What I found is that I could connect to the domain name through my browser. Further, the owner had enough foresight to register his site as the only site on the IP, so I was also able to connect to the site via IP in the browser.
I could also ping both the site and IP and receive proper results, indicating that at least at some level there was DNS resolution happening.
However, when I went to perform the WHOIS lookup at the different site I wasn’t seen the results I should have, which means the DNS for the site hadn’t been properly configured.
I like to use whois.webhosting.info to perform these lookups as you can also perform IP searches to see if the site resolves properly to the IP. Further, you can also see how many and what types of sites may also be hosted on a domain. This can give an indication of other issues which could cause a site to not be indexed.
For example, had I performed a WHOIS lookup and found a bunch of porn or casino sites, I could have assumed (probably quite rightly) that Google had greylisted or blacklisted that IP address because of the sites hosted on it. Then, any legitimate site would also suffer the flag, causing indexing problems by the engine.
But that wasn’t the case here. As I said, it was a single site on a unique IP address.
I gave the information about the DNS to the site owner, indicating my concerns that the site wasn’t properly resolving and I found out a few days later that one of the hosts name servers was indeed acting up. I don’t know what “acting up” means, but it could indicate that the resolver wasn’t always resolving the IP to the site for some visitors, or it was improperly resolving it. In either case, should a search engine come across such inconsistency it will, over time, quit coming to the site.
More issues which could affect rank
There are other issues, not DNS related, but server related which could affect your indexing and ranking possibilities. These include improperly configured servers, or overloaded servers among other things.
Improperly configured servers
As you read above, there was an improperly configured (or failing) name server causing problems with the site, but there are other server issues which could affect the site as well.
Once the site is resolved via a name server, the request is then forwarded to the webserver hosting the site. If the webserver isn’t properly configured however to accept the request, then it will deny the request to the site, even if the site exists on the server.
While this is a rare occurrence, it does happen. If the webmaster mistypes the domain name for example, or attaches the site to the wrong IP attached to a different server, this could cause issues.
Another example of an improperly configured server is one which denies requests to domains without the “www.” Your webserver should be configured to accept all requests for a domain whether they have the www or not.
In fact, the ideal situation is for the server to automatically issue a “301” server site redirect to any request made for a non-www domain which then forwards users to the www domain. This is because sometimes search engines will see the www and non-www domains as 2 separate sites. Therefore link popularity and rankings could be split among the domains. In a worst case scenario, one or both of these domains could be banned by the search engines for duplicate content.
In addition, if you are fortunate enough to have your own IP address, you should check with your web host to ensure that the IP address resolves to the site. This way if a user (or search engine) tries to connect to the site via IP they will be served the site properly, and not receive a “page not found” error.
Alias issues
As mentioned above, it is best to have an non-www requests 301 redirected to the www domain. However, for ease of use (and because its quicker and easier to configure) most web hosts will allow all requests to a site to return that site, so that any request to the www, or non-www domain will return the same site. As I eluded to earlier, this could get the site penalized or banned. At the very least, the site won’t properly propagate link popularity, which will ultimately affect its ability to rank competitively in the search engines.
With some servers, the only way to combat this is to set up a unique site for each alias and have the non-www alias 301 to the www domain. Sometimes that requires the appearance of 2 sites per domain on the web server. While most web hosts won’t agree to this, perhaps they can help you come up with a solution that will work for you.
Overloaded servers
Sometimes, when you perform a WHOIS lookup you will find hundreds and hundreds of sites hosted on a single IP.
While in many cases this isn’t an issue, you can still run into problems in the search engines.
This is because, to save time, many engines will cache the IP of a domain and simply request the domain by IP rather than a full DNS request each time it wishes to index the site.
That means, every other site hosted on that IP has the potential to negatively impact your search engine rankings.
This is because every site which is associated with that IP will by default be associated with your site, because of the common IP. If one of the sites has done something to spam the engine in the past, there is a chance that your site could suffer the consequences.
Go back to the example I gave earlier – a legitimate site sharing an IP with porn or gambling sites. I can almost guarantee you that the legitimate site will not rank competitively.
Consider it guilt by association. In this case the association is the sharing of an IP address.
While search engines are getting much smarter at identifying these types of things, some issues still arise. Therefore it’s safer to take precautions now.
If you do find such a site on your shared IP, you could always request that your site be moved to a new IP so that it doesn’t suffer the consequences of a penalized IP.
Another problem with overloaded hosts is that sometimes there is a site (or a few sites) which use so much of a servers resources that other sites suffer. If this is the case, then your site may not respond in a timely manner which means search engine crawlers could stop coming to the site and it will fall from the index.
Again, the solution here is to move to a different webserver or change hosting providers altogether.
As you may see, there could be many issues outside of the site itself which can affect search engine rankings.
The best way you can identify these issues is to become a web detective and perform a little searching of your own using WHOIS servers and PING commands to test out your site to make sure it resolves properly to the web host and IP it is supposed to.
Many times when you’ve ensured that there isn’t an issue on the site itself, then its likely a hosting issue.
And in some cases, you may be able to get these resolved quickly while others take a little longer. If you find its taking too long to get hosting issues solved, might I suggest shopping around for a new host? One that will treat you like a customer and not a number or pay check? That way you can ensure that your site receives the attention it deserves, and problems are dealt with quickly.
Rob Sullivan is a SEO Consultant and Writer for http://www.textlinkbrokers.com. Textlinkbrokers is the trusted leader in building long term rankings through safe and effective link building. Please provide a link directly to Textlinkbrokers when syndicating this article.
