htaccess and robots.txt

(Submitted Mon, 2006-09-18 04:10)

robots

The robots.txt file tells various spidering engines, like those used by search engines, what content to index and what content to leave alone.

You don't strictly need a robots.txt file in your root Drupal directory if you are running a public site. Without one, however, your admin log will start filling up with "robots.txt not found" warnings.

A quick solution is to create an empty robots.txt file. Search engine spiders will find the file, will not encounter any disallow rules, and - hopefully! - go about their business of indexing your website.

Yet there is an even better approach. Why not actually list the directories that you don't want the spider indexing or wasting its time on? Think of the printer friendly pages of, for example, book pages. Duplicate content. Duplicate content bad (just ask Google).

Here's a sample robots.txt that will help keep the spiders where you want them ... in your main content. This one courtesy of twohills.

User-agent: *
Crawl-Delay: 10
Disallow: /aggregator/
Disallow: /tracker/
Disallow: /comment/reply/
Disallow: /node/add/
Disallow: /taxonomy/
Disallow: /user/
Disallow: /files/
Disallow: /search/
Disallow: /book/print/
Disallow: /database/
Disallow: /includes/
Disallow: /misc/
Disallow: /modules/
Disallow: /sites/
Disallow: /themes/
Disallow: /admin/

htaccess

There is reportedly a problem with the stock Drupal 4.7 .htaccess file. It attempts to redirect accesses to yoursite.com/somenode to www.yoursite.com/somenode. But instead it actually redirects to just the frontpage.

This replacement rewrite condition supplied by alliax fixes that oversight.

# This is the better way to do it:
RewriteCond %{HTTP_HOST} !^example\.com
RewriteRule (.*) http://example.com/$1 [R=301,L]

Node Reference: 64780

Submitted by Tips (not verified) on Fri, 2006-09-29 19:24.

I think there is a typo. It should be:

RewriteCond %{HTTP_HOST} !^example\.com
RewriteRule (.*) http://example.com/$1 [R=301,L]

("If not RewriteCond, then RewriteRule")

Submitted by annewity (not verified) on Tue, 2009-03-17 04:49.

I was banging my head as to why the code wasn't working. Thanks.

Submitted by admin on Sat, 2006-09-30 18:29.

Whoops. Good catch, thanks.

Corrected above.

Submitted by Roman Novak (not verified) on Mon, 2006-12-11 00:15.

Hello, I think it is easy to upload the robots.txt file on the server instead of try to set it up by Drupal. And also why to put for example "Disallow: /includes/" etc. into a robots.txt? Because if does not exist links to these folders (/includes/...) I would not be worry about search robots. ok Roman

Submitted by OP Tech Works (not verified) on Sat, 2007-02-24 16:25.

I like the module because I have a multiple sites running off the same installation, with the module the robots.txt is backed up in the databases too.

Submitted by Frontpage Templates (not verified) on Sun, 2009-01-25 00:41.

I was curious if the module does all the editing of the robots.txt file for you, or can you also edit it yourself and upload changes without Drupal overwriting your altered robots.txt. I've encountered search engine problems with some of the robots.txt files automatically output by Drupal, Joomla, and others. I like to know I can always edit it manually and not have my manual changes overwritten.

Submitted by Arizona SEO (not verified) on Fri, 2009-02-06 19:04.

Thanks for the tips!! Just learning Drupal so any little bit helps!! At Arizona SEO we far prefer Joomla, the CMS is so much easier to use. But knowing both is essential.

Submitted by craps strategies (not verified) on Tue, 2009-02-17 10:30.

I am a newbie in the field of seo and and now i am very much inspired and have a clear vision for my future in this arena known as SEO. I think the most important thing to make it big here is "patience", if you have that you are a winner here.

Submitted by rhinestone (not verified) on Fri, 2009-03-20 12:15.

You are true buddy. Actually I am also new in this field and I am now trying to be a professional in this field. SEO is a new concept for me . But I am keeping patience to learn more.

Submitted by продвижение сайтов (not verified) on Thu, 2009-03-12 11:40.

Good, informative review.

As for me It;s simple way to start advance the robots.txt file on the server instead of advance it by drupal.

Thanks for useful tips.

Submitted by Siladitya Banerjee (not verified) on Sat, 2009-03-21 11:10.

I love ur all posts which is mainly written about drupal.I like drupal.Thats why i'm commenting here.Thankx.
wholesale clothing

Submitted by neckties (not verified) on Mon, 2009-03-23 18:02.

Thanks for such a nice information. Very informative blog. I like this one . Please keep posting such a nice information, it makes us visit you again and again.
Thanks

Submitted by youth Hotels (not verified) on Tue, 2009-03-24 16:13.

Yes, this blog is really informative. I came to know about the robot files today. I don't have any idea about robot files ever before.

Submitted by IT services (not verified) on Wed, 2009-03-25 19:22.

Hey there thanks for the information some really great content thanks really helpfull x

Submitted by directory (not verified) on Thu, 2009-03-26 11:55.

This is a very useful information thanks

Submitted by seo company (not verified) on Sat, 2009-03-28 04:41.

maybe drupal is good for some community but, honestly i my self never learn about drupal, i concentrate learning about seo company,seo
marketing,seo service,search engine optimization

Submitted by fulltilt (not verified) on Sun, 2009-03-29 14:23.

Hey !!! robot files are the file used in search engines to make the search faster. Actually SEO is a vast field to explore.We are still learning.

Submitted by Gatwick airport (not verified) on Sun, 2009-03-29 19:54.

I dont thinks so the module will do any thing with the robots.txt.

Submitted by youth Hotel (not verified) on Sun, 2009-03-29 19:57.

Really great one . Please keep posting such a nice information, it makes us visit you again and again.

Submitted by full tilt (not verified) on Mon, 2009-03-30 11:23.

Hey !!! thanks for providing this information to common people. Well this kind of efforts never go useless. Your blog must be visited by lots of people.

Submitted by Siladitya (not verified) on Mon, 2009-03-30 19:19.

u submitted a marvellous comment.it helps me a lot.Thankx.
fulltilt

Submitted by motocross helmets (not verified) on Wed, 2009-04-01 21:02.

Great tips on the Robots.txt files. Definitely clarified a few issues. Thanks for a great post!

Submitted by Danies (not verified) on Thu, 2009-04-02 00:19.

nice tips, now i know how to use robot.txt

------------------------------------------
unlock iphone 3g

Submitted by cacau (not verified) on Thu, 2009-04-02 00:25.

hey i've read somehow it can be install on iphone? both the official and the unlock iphone 3g, any one can confirm this?

Submitted by theOne (not verified) on Thu, 2009-04-02 00:31.

hey the one should be install is drupal right? i think the iphone, both the official and the unlock iphone 3g is still a phone you know.. not a linux os

Submitted by Siladitya Banerjee (not verified) on Fri, 2009-04-03 20:13.

Thankx for this valuable code which u posted here.i'm very glad to see this here.Thankx.
notebook batteries

Submitted by خليجي (not verified) on Wed, 2009-04-15 04:39.

Submitted by Promotional Gifts (not verified) on Thu, 2009-04-16 16:29.

oh yes , i finally got it.

thanks for this great helpful post, you don't know you made my day.

Submitted by Anonymous (not verified) on Thu, 2009-04-16 20:20.

very usefull info at this post thanks!!

Affiliate Promotion

Submitted by ladies clothes (not verified) on Fri, 2009-04-17 17:30.

Robots and htaccess files are important for any web sites. Somtimes we ignore it.

Submitted by Freelance sales agents (not verified) on Tue, 2009-04-21 11:40.

Well, Finally i could gain the knowledge about Robot text and Htaccess files, This must be newbie guide for that.

Submitted by Anonymous (not verified) on Tue, 2009-04-21 17:29.

العاب طبخ -
اصنع الكعك -
اصنعي البرغر -
اصنعي الايس كريم -
تصفيف الشعر -
تلبيس -
ميك اب -
ملابس اميرات ديزني -
صور ملابس
- فساتين -
ازياء-
موضه -
جمال -
العين -
الشعر
photoshop cs 4 _ فوتوشوب 11 -
صور قلوب -
دروس فوتوشوب
- ستايلات -
سكرابز-
خامات -
صور تصميم -
فوتوشوب 9

Submitted by dlu3at.com (not verified) on Wed, 2009-04-22 18:20.

ستار اكاديمي 6 صور نانسي صور اليسا صور هيفاء وهبي العاب بنات بنات الجافا صور تصاميم مناحي صور msn شعر عن الصداقة
فساتين سهرات 2009
فساتين حليمه بولند
افلام عربى 2008
تعب قلبي
اسماء بنات
فساتين سهرات
صور بنات كول
عبد محفوظ 2009
دورة الفوتشوب
Prison Break 4 الحلقة 15 من بريزون بريك
Material Girls

فلم Milk

فلم Cadillac Records

فلم Gran Torino

فلم The Day The Earth Stopped

فلم Ghost Town PROPER

فلم The Rocker

فلم Changeling 2008

فلم Transporter 3

فلم High School Musical 3

فضائح النجوم والمشاهير

فيديو آخر اخبار روتانا والجرس

اليسا

دليل المواقع الاسلاميه

الغاز

العاب براتز
العاب دولز
خلفيات
خلفيات انمي وافلام
خلفيات ثلاثية الابعاد
خلفيات حيوانات وطيور
خلفيات رياضيه
خلفيات سيارات
خلفيات طبيعيه
خلفيات كمبيوتر وانترنت

العاب بنات
صور اليسا
ستار اكاديمي 6
صور نانسي
صور هيفاء وهبي
بنات
الجافا
مسجات عتاب
مسجات حب
مسجات اشتياق
مسجات وسائط mms
مسجات عتاب قوية
مسجات زعل
توبيكات للبنات
كندا
أستراليا
فندق شاطيء السلطان
فندق الحب
جزيرة الجبل الاخضر
مدينة فانكوفر
جزر تاهيتي
شلالات تروميل باخ
منتجع العز بالاحساء
الطبيعه في المانيا
الجزائر
تركيا
جزر الساندوتش
جزيره على شكل جيتار
شرم الشيخ
جزر الباسفيك
مضيق جبل طارق
شنط رالف لورين
احزمه رالف لورين
بوتات 2009
احذيه رالف لورين
ميريل داغر
فساتين باللون الابيض
ازياء 2009
ازياء جيوفاني
ازياء عبيد محفوظ
سواريهات 2009
ازياء بلانكا
صور فساتين سهرات
فساتين سهره للدلوعات
سواريهات باللون الاسود
جيبات وتنانير
بلايز وتوبات شتويه
احذيه من dior
عبايات 2009
شبيه تامر حسني
صور بيت نانسي عجرم
مهند والوسمي
صور حليمه بولند مع ابوها
قبل وبعد عمليات التجميل
الخضار بالبشاميل
ميلك شيك
مكرونه مكسيكيه
افخاذ الدجاج
حلى الجواهر
غرف جلوس
ديكورات يابانيه
غرف نوم
سيراميك 2009
مسرحية البيت بيتك
مسلسل ناصر
اخويا هايص وانا لايص
راجل وست ستات
محمد سعد الدبابير
مسلسل تامر وشوقيه
مسلسل قلب ميت
مسلسل شرف فتح الباب
فلم شيكامارا
فلم الداده دودي
مجموعه صور انمي
صور انمي
فيروسات autorun
نظف جهازك من الفيروسات وملفات التجسس
السيرفر الشخصي
اظهار ملفات الفلاش المخفيه
اظهار الاي بي الثابت
ماسنجر بلس متوافق مع ماسنجر 2009
برنامج سلم الرواتب
حذف كلمة السر في الويندوز
تحويل الصوتيات الاحترافي
برنامج استخراج الصور والاصوات
برنامج WS_FTP Pro 2008 كامل مع الكراك
تحسين اداء الرام وذاكرة النظام
مثلث برمودا
جبل فوجي
منتجع قصر اشبيليا في الاحساء
قصص مرعبه
قصص رعب
خريف سويسرا
شوارع باريس ليلا
مسلسل وتمضي الايام
مسلسل الحلم الضائع
مسلسل حد السكين
كلمة فصل
تاراتاتا
The Manage
نكت
ازياء
زهير مراد 2009
ايلي صعب 2009
احمد البدوي 2009

فلم Cadillac Records
فلم Gran Torino
فلم The Day The Earth Stopped
فلم Ghost Town PROPER
فلم The Rocker
فلم Changeling 2008
فلم Transporter 3
فلم High School Musical 3
العاب مكياج
مكياج عرائس
العاب تلبيس
العاب طبخ
فساتين زفاف
فضائح النجوم والمشاهير
فيديو آخر اخبار روتانا والجرس
اليسا
دليل المواقع الاسلاميه
الغاز
أميرات ديزني
نينا ريتشي
فساتين سهره من ايلي صعب

Submitted by трактор (not verified) on Thu, 2009-04-23 10:03.

Robots.txt - my important helper! Thanks for informative post, I consider these tips will be useful for my site! Goodluck!

Submitted by Prestiti Online (not verified) on Fri, 2009-04-24 07:54.

"Drupal is the best software.
If u dont believe than for try it just search this site "

Prestiti

Prestiti Online

Cessione del quinto Prestiti

Submitted by Carta credito (not verified) on Fri, 2009-04-24 10:03.

"I use drupal and its work very good"

"Thanks
i will give advice to visitors"

"My advice here to all of you is, Please post only topic related content.Otherwise there is enough space to create your own blog with your own topic.

Yeah,comments are great. It is very helpful in socializing in blogosphere. Also useful in building links. Fight spam, not blogs."

Carte di credito

Carta credito

Carta Revolving

Submitted by chicago Realtor (not verified) on Sun, 2009-04-26 05:40.

I use word press for my new websites, but I am learning about Drupal and would like to switch soon, thanks for the nice post!

Submitted by jordan shoes (not verified) on Sun, 2009-05-03 12:40.

Submitted by Tgool (not verified) on Wed, 2009-05-06 05:00.

New Balance shoes have great color! These shoes are perfect for walking around. Quite comfortable, good grip on wet surfaces.
If you have an interest in this shoes, you can visit http://www.tgool.com/
If you have an interest in it, you can contact us by email or MSN.
Email:tgool@shopkiss.com
MSN:joeyselina@hotmail.com

Submitted by Somnath (not verified) on Tue, 2009-05-19 11:15.

I liked the segment.......and have a wish to know more about this topic.....hair loss treatment products

Judah Hertz
EMF Financial Products
Best DRUPAL Web Hosting & 30% off coupon

What Would Drupal Do?

htaccess and robots.txt

New Modules