Knowledge Base - AYKsolutions

Search

Search results

General Server Questions

Googlebots are no longer able to crawl my site, what happened?

This may be attributed to your MTU setting on your main eth interface (usually eth1 or eth0). The below example applies to an RH-based OS.

When you are logged into shell as root or a privileged user, type in:

ifconfig ethX mtu X

Where ethX is either eth0, eth1, etc.. depending on your configuration and mtu X is a number in seconds. To see what you currently have it set at, you can run the ifconfig command all by itself and look at the MTU value returned. A good number for googlebot to properly crawl your site is around 500. So, as a full example, if your main interface is eth1, you'd run the following command from the prompt:

ifconfig eth1 mtu 500

To have this take effect on reboot/startup, add the following line to your /etc/rc.local file:

/sbin/ifconfig eth1 mtu 500;