Your robots.txt tells crawlers which parts of your site they may fetch. TamRank lets you write your own without touching a file on the server: you switch on robots.txt management in the settings, type the rules in a text box, and TamRank serves them. This page shows where that editor is, what to put in it, and the two traps that catch people out.
In short
- Go to TamRank → Settings → General and find the Robots.txt card.
- Switch on Activate robots.txt management. The editor appears.
- Type your rules, one per line, and save.
- Click View current robots.txt to see what your site is actually serving.
Where the editor is
The editor is on the Robots.txt card under TamRank → Settings → General. It stays collapsed until you turn on Activate robots.txt management, because until you do, TamRank leaves your robots.txt alone.
What you get is a plain text box, not a rule builder. You write robots.txt the way robots.txt is written, one directive per line. The placeholder shows a sensible starting point: a rule blocking /wp-admin/, an exception allowing admin-ajax.php, and a Sitemap: line pointing at your sitemap.
Underneath sits a View current robots.txt button that opens the live file. That is the honest check, because it shows what a crawler gets rather than what you meant to write.
There is no file on your server
This is the part that confuses people, so it is worth stating plainly. TamRank does not write a robots.txt file to your web root. WordPress serves robots.txt virtually, generating it on request, and TamRank hooks into that and replaces the contents with yours.
Two consequences follow, and both are useful to know:
- Emptying the editor gives you WordPress’ default back. Your override is only active while the box has content in it. There is nothing to delete and nothing to restore.
- A real robots.txt file on your server wins, silently. If a file physically exists in your web root, your web server hands that over and WordPress is never asked, so TamRank never gets a say. Nothing in the admin will tell you this is happening. If your edits appear to do nothing at all, that file is the first thing to check, and it is the usual leftover from a plugin that did write files.
What to put in it
Most sites need very little. A short, correct robots.txt beats a long clever one.
Reasonable to block: your admin area, staging environments, internal search result pages, and paths that generate endless near-duplicate URLs.
Do not block: your CSS and JavaScript, because Google renders your pages and needs them to judge the result; images you want found; and anything you actually want ranked.
The common directives:
User-agent: *applies the rules below it to every crawlerDisallow: /folder/asks crawlers to stay out of a pathAllow: /folder/file.phpcarves an exception out of a blocked pathSitemap:points crawlers at your sitemap, and belongs at the end
Tip: robots.txt is a request, not a lock
Well-behaved crawlers honour it. Anything malicious ignores it, and the file is public, so listing a path you want hidden is closer to advertising it than protecting it. Anything genuinely private needs a login or a server rule.
There is a second misunderstanding worth clearing up: blocking a URL here does not remove it from Google. A blocked page can still appear in results, just without a useful description, because you told Google not to look at it. If you want a page out of the index, let crawlers fetch it and set it to noindex instead. Blocking and de-indexing pull in opposite directions, and doing both means the noindex is never read.
Troubleshooting
My edits do not show up at all. Either robots.txt management is switched off, or a physical robots.txt file exists in your web root and is being served instead. Check the toggle first, then the file.
The Sitemap line points at the wrong domain. TamRank writes that line once, at activation, using the site address at that moment. It is never updated afterwards, so a site built on staging and then moved live keeps the staging URL. Correct it by hand in the editor.
Google Search Console says a page is blocked. Read your rules from the top down and look for a Disallow that covers that path. Remember that Disallow: / blocks the entire site.
I want to check a specific URL. Search Console has a robots.txt report under Settings that shows the file Google last fetched and when. Google retired the old standalone Tester tool, so if you are looking for that, this is where it went.
I emptied the box and now the defaults are back. That is intended. An empty editor means no override.
Frequently asked questions
Where is the robots.txt editor?
TamRank → Settings → General, on the Robots.txt card, after switching on Activate robots.txt management.
Does TamRank create a robots.txt file?
No. It replaces the contents of the virtual robots.txt WordPress already serves. There is no file on your server.
What happens if I clear the editor?
Your override stops and WordPress’ own default robots.txt is served again.
Will blocking a page remove it from Google?
No. Use noindex for that. A blocked page can still be listed, just without a description.
Do I need a robots.txt at all?
Most small sites do not need a custom one. WordPress’ default is fine, and the main thing worth adding is the Sitemap line.
Related
- The file it points at: The XML sitemap in TamRank
- Keeping one page out of search: Writing meta titles and descriptions
- The bigger picture: The complete WordPress SEO guide
- Back to Documentation
