Harnessing the Power of Robots.txt
Harnessing the Power of Robots.txt
Once we have a website up and running, we need to make sure that all visiting search engines can access all the pages we want them to look at.
Sometimes, we may want search engines to not index certain parts of the site, жди меня национальная служба or even ban other Посейдон Search Engines from the site all together.
This is where a simple, little 2 line text file called robots.txt comes in.
Robots.txt resides in your websites main Turbo MOLE/60A LS WHITE MASS.GREZZO directory (on LINUX systems this is your /public_html/ directory), and looks something like the following:
User-agent: *
Disal
low:
The first line controls the “bot” that will be visiting your site, the second line controls if they are allowed in, or which parts of the site they are not allowed to visit…
If you want to handle multiple “bots”, then simple repeat the above Elica MAYA INOX A/60 lines. So an example:
User-agent: googlebot
Disallow:
User-agent: askjeeves
Disallow: /
This will ELICA WAVE MW REMOTE IX/F 51 allow Goggle (user-agent name GoogleBot) to visit every page and directory, while at the same time banning Ask Jeeves from the site completely.
To find a “reasonably” up to date list of robot user Александр. Невская битва (2008 г.) names this visit http://www.robotstxt.org/wc/active/html/index.html
Even if you want to allow every robot to index every page of your site, it’s still very advisable to put Toughbook статьи a robots.txt file on your site. It will stop your error logs filling up with entries from Toshiba Satellite A200-1YW search engines trying to access your robots.txt file that doesn’t Novex CT1471 exist.
For more information on robots.txt see, the full list of resources about robots.txt at http://www.websitesecrets101.com/robotstxt-further-reading-resources/
Bruce Hearder owns and run http://www.WebsiteSecrets101.com a site packed full with tips and tricks to getting the most out микроволновые печи Kuppersbusch of your existing website. Sign up for the WebSiteSecrets101 newsletter today and squeeze more out of your website.
Google Hijacks Press Clipping Services
A few years back, I wanted to find out if my press releases got any media coverage. Media folk don’t have the time to tell you if they’re publishing a story about you. So I was excited when I heard about services that monitor publications and notify you when you make the press. The problem was it cost way too much money for the average unsigned band.
Enter Google.
Google has a couple o’killer services that do the same thing, but they notify you by email… as often as you want… and for free!
Google Alert is the primary service. You can set it up to do a search for “Brobdingnagian Bards” and will notify you daily or clearwater fl beachfront vacation rentals weekly whenever that term is added to a webpage online. Google News Acer Aspire 6935G-934G32Bn Alert does the same thing, but focuses o