10-15-2010, 05:38 PM
(10-15-2010, 05:31 PM)Jay Wrote: This is why you only give bots the same permissions as guests, otherwise it will index private links such as moderator sections and the administrator panel.
In theory you could block the admin links etc in robots.txt and still have a bot as registered or god forbid higher!
However i think its considered bad by google if it can see content others can not.
For example if you made google a member and your forum was only visible to logged in users.
This could get your forum blacklisted even i suspect.
So yes i agree. Bots have to see exactly what users not logged in can see.