Setting Up Blogger Crawler and Index

Now Blogger has great feature for search engine visibility.Now we can visible our blog in search engines with robot.txt. To active this feature go to Settings | Search Preferences, click Edit next to "Custom robots.txt" and select Yes. After this selection text box will be visible to give the crawler code for your site.

Note: which was mention in blogger site
Warning! Use with caution. Incorrect use of these features can result in your blog being ignored by search engines.

by default the robot.txt has following code,

User-agent: Mediapartners-Google
Disallow: 

User-agent: *
Disallow: /search
Allow: /
Sitemap: http://www.example.com/feeds/posts/default?orderby=updated

User-agent: Mediapartners-Google is a robot from Google Adsense, leave it as is because if you mistakenly change that than the ads served will not fit with your content.

The next line is for all the robots and marked with an asterisk (*). On the default configuration, it is clear that the label of our blog is not indexed Disallow: /search.

Keep in mind that a slash (/) is as your homepage, so for example if you want the label to get indexed, do not just fill up with a slash like this Disallow: / because that would be you do not allow the robot tracing your blog

Finally your code can be like bellow,

User-agent: Mediapartners-Google 

Disallow:

User-agent: * 

Disallow: /search?updated-min= 

Disallow: /search?updated-max= 

Disallow: /search/label/*?updated-min= 

Disallow: /search/label/*?updated-max= 

Allow: / 

Sitemap: http://www.youtdomain.blogspot.com/feeds/posts/default?orderby=updated

After the changes, make sure everything is fit like what we want by visiting www.example.com/robots.txt. Replace the Example.com with your domain name.