Support Forums › Categories › Life Support › Omni's Secret Hidden Inside Tricks
GoogleBot Loves My Forum So Much...

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Threaded Mode
GoogleBot Loves My Forum So Much...
Solidus Offline
Double Helix
****
Posts: 644
Threads: 16
Joined: Oct 2009
Reputation: 43
#11
10-15-2010, 05:31 PM
This is why you only give bots the same permissions as guests, otherwise it will index private links such as moderator sections and the administrator panel.
[Image: di-Y0TG.png]
GFX Forums | RandomForum | Twitter
Website Find
Reply
DAMINK™ Away
Moderation is Overrated
*****
Posts: 1,782
Threads: 167
Joined: Oct 2009
Reputation: 71
#12
10-15-2010, 05:38 PM
(10-15-2010, 05:31 PM)Jay Wrote: This is why you only give bots the same permissions as guests, otherwise it will index private links such as moderator sections and the administrator panel.

In theory you could block the admin links etc in robots.txt and still have a bot as registered or god forbid higher!
However i think its considered bad by google if it can see content others can not.
For example if you made google a member and your forum was only visible to logged in users.
This could get your forum blacklisted even i suspect.

So yes i agree. Bots have to see exactly what users not logged in can see.
The Rules!
FTW Forum <-- Home of the Damned! --> Join me On MM


Website Find
Reply
deantehnoob Offline
Learner
*
Posts: 32
Threads: 0
Joined: Sep 2010
Reputation: 0
#13
10-15-2010, 09:30 PM
yea googlebot is allways online going through forums, googlebot is everywhere at the same time. god od the interwebz
[Image: denassig.png]
Find
Reply
Resistance Away
Mr. Straight Forward
*****
Posts: 1,106
Threads: 195
Joined: Sep 2010
Reputation: 17
#14
10-16-2010, 12:01 AM
Hahaha, @DAMINK...

Lol "Google and all spiders scrape up all links on your forum. (unless in robots.txt file as deny)"
[Image: t5BWm.png]
Website Find
Reply
Fremo Offline
Learner
*
Posts: 28
Threads: 0
Joined: Oct 2010
Reputation: 0
#15
10-17-2010, 06:05 PM
maybe its stalking people :O
Find
Reply
DAMINK™ Away
Moderation is Overrated
*****
Posts: 1,782
Threads: 167
Joined: Oct 2009
Reputation: 71
#16
10-17-2010, 06:27 PM
(10-16-2010, 12:01 AM)L3g1tWa5te Wrote: Hahaha, @DAMINK...

Lol "Google and all spiders scrape up all links on your forum. (unless in robots.txt file as deny)"

Yep exactly what i said in my 2 posts in this thread.
However some bots dont obey this rule. Many actually.
Some actually look for the deny and scrape those.
However that really is getting a little off topic i guess.
The Rules!
FTW Forum <-- Home of the Damned! --> Join me On MM


Website Find
Reply
Kaleb Offline
Regular Member
***
Posts: 158
Threads: 18
Joined: Feb 2010
Reputation: 1
#17
10-17-2010, 06:28 PM
My googlebot is always viewing the no permission page, i have no freakin idea what its trying to do.
Find
Reply
NekoChan Offline
Emotional Support, Yo'
*****
Posts: 1,059
Threads: 81
Joined: Oct 2009
Reputation: 21
#18
10-18-2010, 05:06 AM
(10-17-2010, 06:28 PM)Kaleb Wrote: My googlebot is always viewing the no permission page, i have no freakin idea what its trying to do.

It is haxxoring your forum right now, go and ban it!
[Image: Nyan.jpg]
Website Find
Reply
DAMINK™ Away
Moderation is Overrated
*****
Posts: 1,782
Threads: 167
Joined: Oct 2009
Reputation: 71
#19
10-23-2010, 09:07 PM
[Image: 12878930704218_download.jpg]

Noticed this today.
Damn google bot!
The Rules!
FTW Forum <-- Home of the Damned! --> Join me On MM


Website Find
Reply
Positive Offline
HJT Helper
***
Posts: 121
Threads: 14
Joined: Oct 2010
Reputation: 5
#20
10-23-2010, 10:25 PM
(08-21-2010, 01:32 PM)Omniscient Wrote: As the biggest search engine online it does the most indexing. It's normal to see Googlebot on your site often.

You would be the best person to talk to about that. Wouldn't you. Everytime i go on HF, i look at the Home page and see a new member then refresh and you have a new sign up. Big Grin
Website Find
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)