Twitter makes an interesting choice.

With the recent update of Twitter’s robots.txt file, the floodgates have been opened to millions of pages to be crawled, however this is no guarantee that that results offered shall always be wanted.

The Sociable was the first to be the first to actively see a difference with Twitter’s robot.txt, this allows Google, Bing, Yahoo, Yandex and the bots of others to crawl into Twitter’s search results pages.

Twitter has now taken to opening up content search results such as tweets and hashtags, yet simultaneously keeping bots from crawling through the results for by any user’s, videos and images. Tweet and hashtag search results are displayed using the/search/path; searches for users of Twitter show using the/search/users path; and video and photo searches are presented using /search/(keyword)/grid. Twitter is as of now permitting Google to crawl the tweet/hashtag results of searches, yet holding back the two other kinds; Twitter sent the same message to Bing and other search engines as well.
The tweet/hashtag of Twitter search results sometimes posses an/or lead to content of all kinds — An example of a search for #Chuck’s during last weekend’s Chuck’s Awards or a search for #Best place to eat Italian food. The latter search led to Twitter’s Italian food hashtag page, partnerships with Food Network that involved a mix of human- and algorithmically-curated content about the Italian food.

Check us out at Joint Effort Marketing or give us call at 513-474-1158 for more information on targeting mobile devices to get you more website traffic.