Show Search Engines the Way: Sitemap & robots.txt Guide

No matter how good your website looks, if Google can't properly find your pages, they won't show up in search results. Two files help point search engines in the right direction: your sitemap (sitemap.xml) and robots.txt.

What is a sitemap (sitemap.xml)?

A sitemap is a file that lists all the page addresses on your site. When Google reads it, it crawls your pages faster and more completely.

• New pages get indexed quicker (e.g. a new menu or event page)
• Especially handy for large sites or sites with weak internal linking
• It won't directly boost rankings, but it helps ensure nothing gets missed

What is robots.txt?

The robots.txt file tells search engines "you can look here, but please don't look there."

• Blocks pages that don't need to appear, like admin pages, carts, or internal search results
• Focuses crawling effort on your important pages
• It's good practice to include your sitemap location inside this file

How to set it up

1. Create a sitemap: On WordPress, plugins like Yoast or Rank Math generate one automatically. You'll usually find it at yoursite.com/sitemap.xml.
2. Check robots.txt: Visit yoursite.com/robots.txt to review it. Accidentally blocking your whole site (Disallow: /) will make it disappear from search.
3. Submit to Google Search Console: Add the URL under the 'Sitemaps' menu to track your indexing status.

Common mistakes

• Leaving a full block in place after launch (often set during development)
• Keeping old or deleted pages in the sitemap
• Blocking a page in robots.txt and expecting it to vanish from search (blocking and de-indexing are different things)

These two small files are the foundation of good SEO — take a moment to check yours today.

At PICKTECH, we build websites for Korean businesses across Australia and handle sitemap and robots.txt setup, plus Search Console submission, as part of the job. If search visibility is on your mind, feel free to reach out.