robots.txt

Each rule names a crawler, or all of them, and the paths it may or may not request. Well-behaved crawlers read the file before they fetch anything else; it cannot force a crawler to obey, and it keeps nothing private.

A path blocked here can still show up in results as a bare address if other sites link to it, because the search engine never reads the page. To keep a page out of results, use noindex and leave the page crawlable.

Mistakes are easy to make and expensive: one broad Disallow rule can hide a whole section, or the whole site. Checking the file after every release is cheap.