How-to · Intermediate · 15 minutes
How to Write a robots.txt
Steer crawlers properly, and avoid the self-inflicted wounds this file is famous for — a practical, jargon-free walkthrough with the classic mistake flagged before you make it.
Straight answer first
The job: steer crawlers properly, and avoid the self-inflicted wounds this file is famous for. Time to allow: 15 minutes. Skill needed: intermediate.
The walkthrough below covers every step, the mistake most people make, and the shortcut worth keeping. On Hosting Seller plans several of these steps are already done for you, and we say where.
Written by the Hosting Seller staff · Checked 1 August 2026
Intermediate
Difficulty level
15 minutes
Time to allow
5
Stages
24/7
Help at any hour
You do not need to be technical for this. The walkthrough is written for first-timers, tested on our own platform, and honest about which parts are genuinely fiddly and which are merely unfamiliar.
One promise before you start: nothing in this guide is irreversible. Where a step could bite, we say so and give you the undo.
The shape of the work
The job breaks into a few clear stages: know what it is not, start from almost nothing, never block a page you want ranked, point at the sitemap and test it before you trust it.
No stage needs code or a terminal unless the guide says so outright, and where it does, the exact commands are printed. The full step-by-step sits below; the sections around it give the context that makes it stick.
Where this goes wrong, and how not to
A Disallow: / left behind from development. One character blocks the entire site, nothing looks wrong on the page itself, and it gets discovered when traffic flatlines. This is the launch-checklist item that pays for the whole checklist.
It is worth learning because it is not some rare edge case — it is the single most common reason this task turns into a support ticket. Knowing it up front turns the whole job from risky into routine.
One habit that makes this easy for good
Do not block CSS and JavaScript paths. Google renders pages in full, so blocked assets make your pages look broken in its eyes — a ranking cost paid for a rule that gains you nothing whatsoever.
Small habits like this are the real difference between people who find hosting easy and people who find it stressful. The tools are identical; the working method is not.
How our platform shortens this
We have automated the steps that do not deserve your time: certificates issue and reissue themselves, the installer handles application setup, the daily backup covers the what-if, and per-site settings live in a panel instead of a config file. The guide above covers what remains — the part that is actually about your site.
And when a step misbehaves anyway, support answers at any hour with the actual fix, not a knowledge-base link and a shrug. Half our best guides began life as a pattern in the ticket queue.
The hosting this guide is written against
Every walkthrough on our how-to shelf is tested on the platform we actually run — cPanel, LiteSpeed, NVMe, one-click installs — so the instructions match your screen instead of gesturing vaguely at it.
Real people answer the counter at any hour, including the awkward questions other hosts wave off as out of scope.
- Step by step, tested exactly as printed
- The snag flagged before you reach it
- The dull steps are already automated
- People on hand at any hour if you stall
Why Hosting Seller
On every plan, as standard
5 steps, no filler
Each stage is a few minutes of steady clicking, with the fiddly parts marked as fiddly.
The snag, named early
The classic mistake for this exact job is flagged before step one, so 15 minutes stays 15 minutes.
Automation where it belongs
SSL, backups and installs run themselves here, so the guide covers only what is genuinely yours to do.
No jargon toll
Terms are explained where they appear or linked to the glossary — nothing assumes you already know.
Help on the counter
Stuck on step three at midnight? Support answers at any hour, mid-guide included.
Written from real tickets
Our guides come off the support desk, so the snags flagged here are the ones people genuinely hit.
First Steps
From choosing to live
- 1
Know what it is not
It is crawl guidance, not access control. robots.txt asks well-behaved crawlers to stay out of certain paths; it hides nothing from a browser or from anybody acting in bad faith.
- 2
Start from almost nothing
Most sites need very little: allow everything, point at the sitemap, and perhaps exclude genuine crawl waste such as internal search results or cart addresses.
- 3
Never block a page you want ranked
A disallowed path cannot be crawled, so anything you want indexed has to stay reachable. The file's power is exactly why its mistakes are so expensive.
- 4
Point at the sitemap
The Sitemap: line hands crawlers the map inside the first file they check. One line, and a standing invitation to come and look around.
- 5
Test it before you trust it
Search Console's robots tester, plus a direct read at /robots.txt, confirms the file says what you meant. Syntax errors fail quietly, and they fail toward blocking too much.
In the Box
Packed with every plan
- Site migration done for you by our staff, at no charge
- A backup taken daily, with restores you run yourself
- Spam and virus filtering fitted to every mailbox
- SSH, Git and Composer on the plans built for developers
- DDoS filtering handled out at the network edge
- Mailboxes that carry your own domain name
- A 30-day money-back guarantee on every hosting plan
- LiteSpeed caching running at the server, not bolted on by plugin
- Webmail in the browser plus IMAP, POP and SMTP for your own client
- A free SSL certificate on every plan, reissued before it lapses
Across the Counter
The questions we get asked most
Can I use robots.txt to hide private pages?
No. It is a public file politely asking crawlers to keep away, and it advertises every path it names. Real privacy needs authentication — a directory password or an application login. robots.txt is traffic management, not security.
Does every site need one?
Sites work perfectly well without one, since an absent file means allow everything. But a minimal file with a sitemap reference is worth having, and having it means you control the contents rather than leaving that address to chance.
Can I bring a domain I already own?
Yes, and transfers in are routine. Unlock the name at your present registrar, collect the auth code, then start the transfer from your client area. Whatever registration time is left comes across with it, and DNS keeps answering the whole way through.
Which control panel comes with the account?
cPanel, which the whole trade already knows. That means every tutorial online matches what you see, your backups restore onto any cPanel host anywhere, and what you learn here stays useful for life. Plesk and DirectAdmin are stocked on specific plans for anyone who prefers them.
Is the SSL certificate genuinely free?
On every plan, with nothing held back. The certificate is issued the moment your domain points here and reissues itself well before expiry. The encryption is the same as a paid DV certificate — paid tiers exist only for wildcard coverage or organisation validation, which most sites never need.
How do I read the mail when I am away from the desk?
Webmail opens in any browser, and every mailbox also speaks IMAP, POP and SMTP — so the mail app on your phone, the client on your desktop and webmail all show the same messages in the same order.
What happens to my files if I leave?
The site and the files stay yours. Pull a full backup from the panel whenever you like, before or during cancellation. Domains remain registered in your name for the term you paid for and can move to any registrar once the standard 60-day window has passed.
Read next
How to Create a Subdomain
Stand up blog.yourdomain or app.yourdomain before the kettle boils — beginner level, about 5 minutes.
How to Clean Up a Hacked Website
A proper recovery after a break-in, not just the visible mess swept away — advanced level, about half a day, done right.
Web Hosting
cPanel hosting on NVMe disks, with SSL, migration and year-one domain included.
Email Hosting
Proper mailboxes on your own domain, at one flat price.
Packing up and moving host? Take our checklist.
A plain running order for a move nobody notices: what to copy first, how to carry email across without dropping a single message, when to point DNS, and the two slips behind nearly every hour of downtime we get called about.
Get online, with no games at checkout.
From a first website to a rack of servers, moving up is a change to your account rather than a migration.
See the plans