robots.txt and noindex are not the same tool

The two do different jobs

A robots.txt disallow rule asks a crawler not to fetch a URL. A noindex rule asks a search engine not to keep a page in its index. They sound like the same instruction phrased differently, so they are often applied together, on the assumption that two locks are better than one.

They are not. One prevents the other from working.

Why combining them backfires

A noindex rule lives in the page — either in a meta tag or in an HTTP response header. To act on it, the crawler has to fetch the page. If robots.txt has told it not to fetch the page, it never sees the rule.

Google states the consequence plainly:

“If the page is blocked by a robots.txt file or the crawler can’t access the page, the crawler will never see the noindex rule, and the page can still appear in search results.”

— Google Search Central, "Block Search indexing with noindex"

What this looks like when it happens to you

The page keeps appearing in results, often with no description, because the crawler has been told about the URL by a link somewhere but has never been allowed to read what is on it. Meanwhile your own checks say the page is excluded, because the noindex tag is right there in the HTML where you put it.

Both things are true at once, which is what makes it hard to spot.

What to do instead

Decide which of the two jobs you actually need, and use only that one.

• To remove a page from search results: allow crawling, and serve noindex — either <meta name="robots" content="noindex"> or an X-Robots-Tag: noindex response header. The crawler must be able to fetch the page to obey it.
• To stop a crawler spending requests on a section that does not need indexing at all (faceted URLs, internal search results): disallow it in robots.txt, and accept that a URL may still surface if something links to it.
• To keep something genuinely private: neither of these. Require authentication, or return 404. robots.txt is a public file, and listing a path in it advertises that the path exists.
• Once a page has been removed from the index, you can add the robots.txt rule back if you want to save the crawl requests — but only after the removal has happened.