| Author |
robots.txt |
|
the_demigod
Home away from home
Joined: 05-Sep-2001
Posts: 170
From: Various Godforsaken places
|
Posted: 2002-04-27 12:59
what does it do??
I recently received some search engine submissions rejections blaming robots.txt as the cause....
|
|
Gareth
Just can't stay away
Joined: 26-Oct-2001
Posts: 88
From: Wales, United Kingdom
|
Posted: 2002-04-27 13:26
robots.txt files tells search engine robots (spiders) which pages they can or can not index.
The robots.txt file supplied with myPHPNuke tells the robots NOT to index any pages, hence your problem.
I would suggest changing it to this
Quote:
| User-agent: *
Disallow: /EventsConfig.php
Disallow: /GPL License.htm
Disallow: /admin.php
Disallow: /admin/
Disallow: /admin/admin/
Disallow: /admin/adminfaq.php
Disallow: /admin/authors.php
Disallow: /admin/automated.php
Disallow: /admin/banners.php
Disallow: /admin/blockcenter/
Disallow: /admin/blocks/
Disallow: /admin/comments.php
Disallow: /admin/download.php
Disallow: /admin/download/
Disallow: /admin/emailuser/
Disallow: /admin/ephemerids.php
Disallow: /admin/everyonenet/
Disallow: /admin/filemanager.php
Disallow: /admin/gallery/
Disallow: /admin/guestbook/
Disallow: /admin/hardencode.php
Disallow: /admin/headlines.php
Disallow: /admin/index.html
Disallow: /admin/links.php
Disallow: /admin/mpnSQLrun/
Disallow: /admin/mymsg2/
Disallow: /admin/myphpnuke/
Disallow: /admin/myyp/
Disallow: /admin/original/config.php
Disallow: /admin/original/index.html
Disallow: /admin/original/item.php
Disallow: /admin/original/main.php
Disallow: /admin/original/original/
Disallow: /admin/original/tools.php
Disallow: /admin/partners/
Disallow: /admin/phpbbaccess.php
Disallow: /admin/phpbbconfig.php
Disallow: /admin/phpbbforum.php
Disallow: /admin/phpbbrank.php
Disallow: /admin/polls.php
Disallow: /admin/qotd/
Disallow: /admin/rand_banners/
Disallow: /admin/referers.php
Disallow: /admin/reviews.php
Disallow: /admin/savedb/
Disallow: /admin/sections.php
Disallow: /admin/settings.php
Disallow: /admin/smilies.php
Disallow: /admin/stories.php
Disallow: /admin/submissions.php
Disallow: /admin/sysinfo/
Disallow: /admin/topics.php
Disallow: /admin/users.php
Disallow: /admin/who/
Disallow: /auth.inc.php
Disallow: /avantgo.php
Disallow: /banners.php
Disallow: /bb_smilies.php
Disallow: /bbcode_ref.php
Disallow: /blocksconst.inc.php
Disallow: /cache/
Disallow: /chat.php
Disallow: /chatboxII.php
Disallow: /chatheader.php
Disallow: /chatinput.php
Disallow: /chattop.php
Disallow: /chatuser.php
Disallow: /clanmail.php
Disallow: /class.Guestbook.php
Disallow: /class.Util.php
Disallow: /class.myfunctions.php
Disallow: /class.phpmailer.php
Disallow: /class.rc4crypt.php
Disallow: /class.smtp.php
Disallow: /click.php
Disallow: /comments.php
Disallow: /compose.php
Disallow: /config.php
Disallow: /contactbook.php
Disallow: /contactmailer.php
Disallow: /count_disp.php
Disallow: /counter.php
Disallow: /crawler.php
Disallow: /dbconf.php
Disallow: /decodemessage.inc
Disallow: /dhtmllib.js
Disallow: /docs/
Disallow: /editpost.php
Disallow: /emailconfig.php
Disallow: /emailhelp.php
Disallow: /eventcaladmin.php
Disallow: /eventfunctions.php
Disallow: /extention.inc
Disallow: /footer.php
Disallow: /functions.php
Disallow: /gallery/
Disallow: /header.php
Disallow: /image.php
Disallow: /images/
Disallow: /inbox.php
Disallow: /include.php
Disallow: /install.php
Disallow: /install.txt
Disallow: /killpost.php
Disallow: /language/
Disallow: /libmail.php
Disallow: /mailattach.php
Disallow: /mailbox.php
Disallow: /mailfooter.php
Disallow: /mailheader.php
Disallow: /mainfile.php
Disallow: /manual/
Disallow: /module.errorhandler.php
Disallow: /module.guestbook.php
Disallow: /module.textsanitizer.php
Disallow: /mymsg2_functions.php
Disallow: /newimage.js
Disallow: /newsapplet.php
Disallow: /overlib.js
Disallow: /phpsyndication.lib.php
Disallow: /phptimer.php
Disallow: /plugtest.html
Disallow: /pollBooth.php
Disallow: /pollcomments.php
Disallow: /pop3.inc
Disallow: /readmail.php
Disallow: /readpmsg.php
Disallow: /reply.php
Disallow: /replypmsg.php
Disallow: /scroller.html
Disallow: /scroller.js
Disallow: /settings.php
Disallow: /snoopy.class.inc
Disallow: /sql/
Disallow: /themes/
Disallow: /topicadmin.php
Disallow: /upgrade.txt
Disallow: /yavs.class
|
|
This will allow the robots to index your site, excluding pages/files that should not be indexed. ----------------- a4dableweb.netInternet Hosting and Solutions.
Life is to short to make enemies
|
|
trini0
MPN Developer
Joined: 18-Jul-2001
Posts: 5096
From: FL USA
|
Posted: 2002-04-27 13:28
Gareth, you seem to know a bit about this. Is there an easier updated way to do this?? ----------------- trini0... mPN2 Test Bed
|
|
the_demigod
Home away from home
Joined: 05-Sep-2001
Posts: 170
From: Various Godforsaken places
|
Posted: 2002-04-27 13:37
Gareth: thanks.
the robots.txt file that came with my nuke was empty exept a single "disallow" line.
|
|
Gareth
Just can't stay away
Joined: 26-Oct-2001
Posts: 88
From: Wales, United Kingdom
|
Posted: 2002-04-27 13:39
There is a different way to do it that is to put an instruction in the metatags. I am not too familiar with this method but I will look it up.
Basically it is something like:
But as I said I'll have to look it up to get the correct format. Not all search engines will obey this.
The other way is to delete the robots.txt file altogether, but the big problem here is that the search engines with then index every page, including admin sections ----------------- a4dableweb.netInternet Hosting and Solutions.
Life is to short to make enemies
|
|
the_demigod
Home away from home
Joined: 05-Sep-2001
Posts: 170
From: Various Godforsaken places
|
Posted: 2002-04-27 13:57
|
|
trini0
MPN Developer
Joined: 18-Jul-2001
Posts: 5096
From: FL USA
|
Posted: 2002-04-27 14:22
I read up on robots.txt last week. From what I saw, its either the txt file or the meta tags. I was wondering about completely different methods to doing it. It seems the robots method was old, thought something newer better was out...
Quote:
|
On 2002-04-27 13:39, Gareth wrote:
There is a different way to do it that is to put an instruction in the metatags. I am not too familiar with this method but I will look it up.
Basically it is something like:
But as I said I'll have to look it up to get the correct format. Not all search engines will obey this.
The other way is to delete the robots.txt file altogether, but the big problem here is that the search engines with then index every page, including admin sections
-----------------
trini0...
mPN2 Test Bed
|
|
----------------- trini0... mPN2 Test Bed
|
|
Garret
MPN Developer
Joined: 23-Mar-2001
Posts: 1262
From: Colorado, USA
|
Posted: 2002-04-27 14:24
They will index *ALL* files they find, including any that are in your admin folder.
|
|
the_demigod
Home away from home
Joined: 05-Sep-2001
Posts: 170
From: Various Godforsaken places
|
Posted: 2002-04-28 12:33
so... the default setting in all PHP nuke version is:
User-agent: *
Disallow: /
meaning that the robots don't index ANYTHING... right??
my site was never indexed by any search engines despite BEING in many of them- all we got was just the domain, no meta's, no info nothing... and the above "disallow" would explain why...
|
|
Garret
MPN Developer
Joined: 23-Mar-2001
Posts: 1262
From: Colorado, USA
|
Posted: 2002-04-28 16:44
correct.
|
Profile
Quote
