Search Engine Optimisation Web Log (BLOG)
Editor: Matt Paines

| This News Blog is dedicated to bringing news, information and innovations targeting the Search Engine Optimisation industry.

Google to index hidden pages

Google's crawl agent Googlebot has started to delve deeper into content hidden behind web forms in a bid to index content previously invisible to the search engine.
 
The internet giant has told webmasters that it is experimenting with technology that is able to fill in HTML forms and then index any of the results it thinks may be useful to searchers.
 
Google says that the technology is capable of submitting forms with either select menus and radio buttons or with user-entered text boxes.
 
Googlebot will generate queries from select menus and radio buttons by selecting values from the HTML. For text box inputs, the program will seek out query terms by looking throughout the site hosting the form.
 
Some webmasters have already been voicing their concerns about the potential for increased server load and intrusions into content that they don't want to appear in search results.
 
However, Google is reassuring webmasters that it is limiting the number of fetches for any individual site. It also says that it is maintaining good crawl practices such as only retrieving GET forms, adhering to robots.txt instructions and avoiding forms that request user information or passwords.

Comment: