The robots.txt file tells search engine crawlers which pages they can and cannot access on your website.
Understanding Robots.txt
Located at yoursite.com/robots.txt, this file contains directives for web crawlers:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yoursite.com/sitemap_index.xmlTamRank’s Robots.txt Editor
Navigate to TamRank → Settings → Robots.txt
Default Rules
TamRank creates sensible defaults:
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /trackback/
Disallow: /feed/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yoursite.com/sitemap_index.xmlCommon Directives
| Directive | Meaning |
|---|---|
User-agent: * |
Applies to all crawlers |
User-agent: Googlebot |
Applies only to Google |
Disallow: /folder/ |
Block entire folder |
Disallow: /page.html |
Block specific page |
Allow: /folder/file.html |
Allow specific file in blocked folder |
Sitemap: |
Location of your sitemap |
Best Practices
Do Block:
/wp-admin/(except admin-ajax.php)/wp-includes/- Staging environments
- Internal search results
- Duplicate content paths
Don’t Block:
- CSS and JavaScript files (Google needs these)
- Images you want indexed
- Important content pages
Testing Your Robots.txt
Google Search Console
- Go to Search Console
- Use the robots.txt Tester tool
- Enter URLs to check if they’re blocked
TamRank Preview
The TamRank editor shows a live preview of your robots.txt file.
Important Notes
- Robots.txt is a suggestion, not a security measure
- Malicious bots may ignore it
- For sensitive content, use proper authentication
- Changes take effect immediately
