How to Create an XML Sitemap for Your Website
If you've built a website, one of the best ways to ensure that it is easily crawled and indexed by Google and other search engines is to create a custom XML sitemap. An XML sitemap file is basically a list of your website's URLs, formatted in a way which is easily crawled and understood by search engines and other spiders crawling your site. Once your sitemap is created you can reference the file's location in your robots.txt file and submit it in Google Search Console to ensure that it's accessible to search engines.
There are numerous tools which you can use to create a custom XML sitemap file, and if you wanted to you could even write one yourself, but in today's article I'm going to walk through how to create an XML sitemap using Screaming Frog's SEO tool - in my opinion one of the best budget SEO Tools for any individual or agency to invest in.
There are numerous tools which you can use to create a custom XML sitemap file, and if you wanted to you could even write one yourself, but in today's article I'm going to walk through how to create an XML sitemap using Screaming Frog's SEO tool - in my opinion one of the best budget SEO Tools for any individual or agency to invest in.
Do I Need an XML Sitemap?
The short answer is that not really, but it helps to improve the SEO friendliness of your site, and if you're interested in organic web traffic, then this is an easy step to take to help improve indexation and make all of your website's content accessible to Google, Yahoo, and Bing.
If all of the pages on your website have links pointing to them from other pages, chances are good that Google's spider will find them on its own, but the first place any spider that comes to your website looks is in its robots.txt file, and if you've posted the location of a full list of your website's pages and/or blog posts there, it helps to ensure your pages are getting crawled as often as possible.
If all of the pages on your website have links pointing to them from other pages, chances are good that Google's spider will find them on its own, but the first place any spider that comes to your website looks is in its robots.txt file, and if you've posted the location of a full list of your website's pages and/or blog posts there, it helps to ensure your pages are getting crawled as often as possible.
How do I Reference my XML Sitemap in my Robots.txt File?
This couldn't be simpler. At the bottom or top of your robots.txt file simply add a line which reads sitemap: http://www.mysite.com/sitemap.xml (just update with the actual URL/location of your sitemap is). For an example take a look at my site's robots.txt by navigating to joehessert.com/robots.txt in a new browser window.
Is Screaming Frog the Best Tool to Build an XML Sitemap?
If you're building a sitemap for a small site you can use Screaming Frog's free download of their SEO Tool and follow the steps below to create it, or you can buy the full version which is a bit over $120 for a year's license. You can also opt for a less expensive, but equally good option from a site like www.xml-sitemaps.com.
I like the different options provided by Screaming Frog, and recommend it for larger websites with a more complex site structure where you plan to add more detail to the sitemap like priority, change frequency, etc. (this is discussed below). For small sites who want to spend less, you can get a great tool with the "Unlimited Sitemap Generator" standalone option from xml-sitemaps.com which will cost under $20.
Without further delay, here's the guide:
I like the different options provided by Screaming Frog, and recommend it for larger websites with a more complex site structure where you plan to add more detail to the sitemap like priority, change frequency, etc. (this is discussed below). For small sites who want to spend less, you can get a great tool with the "Unlimited Sitemap Generator" standalone option from xml-sitemaps.com which will cost under $20.
Without further delay, here's the guide:
How to Create a Custom XML Sitemap with Screaming Frog
Step 1: Crawl the site you want to generate a sitemap for. This is simply a matter of entering the site's homepage address and clicking "Start" at the top center of the Screaming Frog Window. Wait for the crawl to complete before beginning the next step.
Once your site has been crawled by Screaming Frog click Sitemaps > Create XML Sitemap in the Screaming Frog menu.
Once your site has been crawled by Screaming Frog click Sitemaps > Create XML Sitemap in the Screaming Frog menu.
Step 2: Update the settings in each tab of the Sitemap Export Configuration window as desired.
Here's an overview of each window, what the settings mean and how I would recommend that you configure your XML sitemap.
Here's an overview of each window, what the settings mean and how I would recommend that you configure your XML sitemap.
XML Sitemap Pages Tab
I'm of the mindset that you shouldn't send mixed-messages to Google, and including pages which have the meta noindex tag (instructing Google not to index them) in your sitemap just seems like a waste of time. The same is true for paginated URLs, etc. I recommend that you include PDF files in your sitemap, but leave the other options un-checked.
XML Sitemap Last Modified Tag
On this tab you have the option to include the last modified or <lastmod> tag in your sitemap or not. Essentially this is a tag which is associated with each URL, indicating to Google when the last time the page was modified. Including this tag helps Google's visits to your site be more informed/efficient, as spiders can clearly identify pages which were recently added or modified and spend time crawling those first and skipping over those which it has been over recently and haven't changed since its last visit.
If you choose to include this tag you can either use a custom date (if you update your sitemaps manually, it would be today's time each time you generate a new file), or you can use the server response time.
In general I recommend using the server response time if the CMS you've chosen to build your website allows you to upload a file once and have additional pages and modifications auto-update that file. If you aren't so lucky and you're tasked with regularly updating the file yourself and re-submitting, then using a custom date is a good option as well, as it has the added benefit of allowing you to keep track of how recently you've updated your sitemap and it can serve as a good reminder that it might be time to refresh your on page SEO, or update your XML Sitemap file to include recently added pages or blog posts.
If you choose to include this tag you can either use a custom date (if you update your sitemaps manually, it would be today's time each time you generate a new file), or you can use the server response time.
In general I recommend using the server response time if the CMS you've chosen to build your website allows you to upload a file once and have additional pages and modifications auto-update that file. If you aren't so lucky and you're tasked with regularly updating the file yourself and re-submitting, then using a custom date is a good option as well, as it has the added benefit of allowing you to keep track of how recently you've updated your sitemap and it can serve as a good reminder that it might be time to refresh your on page SEO, or update your XML Sitemap file to include recently added pages or blog posts.
XML Sitemap Priority Tag Settings
The priority tag in XML Sitemaps is a way to help search engines understand your site's structure and the relative priority of each URL listed. Priority is assigned a numeric value on a scale from 0.1 to 1 with 1 being the highest priority page (usually the homepage). Typically level 1 pages (major category pages) will then be assigned the 0.9 priority, sub-category pages will be assigned 0.8, product pages within the sub-categories would be 0.7, etc. The cool thing about Screaming Frog is it will assign and apply all of these for you based on the level of each page identified during the crawl.
I should add that there's no hard-and-fast rule saying you have to directly follow your site content structure when assigning priority, but it's best practice to be consistent. So for example if you equally weigh the level 1 and level 2 pages in terms of their priority on your website, you can assign the homepage a priority level 1 and all level 1 and level 2 pages on your site could be assigned a priority level of 0.9 via the XML sitemap priority tag.
You can also uncheck the include option in Screaming Frog if you aren't interested in having this included in your custom XML sitemap. In general, small sites don't need this, but it's helpful if you're managing a large e-commerce website with thousands of products and category pages.
I should add that there's no hard-and-fast rule saying you have to directly follow your site content structure when assigning priority, but it's best practice to be consistent. So for example if you equally weigh the level 1 and level 2 pages in terms of their priority on your website, you can assign the homepage a priority level 1 and all level 1 and level 2 pages on your site could be assigned a priority level of 0.9 via the XML sitemap priority tag.
You can also uncheck the include option in Screaming Frog if you aren't interested in having this included in your custom XML sitemap. In general, small sites don't need this, but it's helpful if you're managing a large e-commerce website with thousands of products and category pages.
XML Sitemap Change Frequency Tag
The <changefreq> tag in your XML sitemap is an optional tag you can add to enhance Google's understanding of how frequently your site's content is updated. Again, for smaller sites you may choose to not include this tag, but for larger websites I recommend it.
Here you can use your content tree levels to assign the typical change frequency of different content sections of your website. If your product pages aren't updated except for monthly pricing updates, then you can indicate that by setting them to a monthly change frequency, and if your homepage and major category pages are updated every day with special offers, you can indicate that too. The settings you choose for the XML sitemap change frequency tag will be determined by your site maintenance regimen.
Here you can use your content tree levels to assign the typical change frequency of different content sections of your website. If your product pages aren't updated except for monthly pricing updates, then you can indicate that by setting them to a monthly change frequency, and if your homepage and major category pages are updated every day with special offers, you can indicate that too. The settings you choose for the XML sitemap change frequency tag will be determined by your site maintenance regimen.
XML Images Sitemap
Finally, you can include the URLs of images on your website within your main XML sitemap with Screaming Frog, but I recommend leaving this tab un-checked. If you are interested in creating an image sitemap, you can do so separately (that's an option in the Sitemaps menu), and referencing that in your robots.txt file as an additional sitemap. This will help keep your primary webpage/blogpost sitemap clean and focused.
Once Your XML Sitemap is Generated What's Next?
After you have created your custom XML Sitemap file, the next step is to upload it to your website. Finally, there are two steps you should take to ensure that it is error-free and properly found and utilized by search engines.
Once you've tested and submit your sitemap within Google's Search Console, you can use Webmaster Tools to monitor the indexation of your site and identify any errors which may result from updating your website's URLs, etc. which require you to update your sitemap.
- Reference the file's location in your robots.txt file (mentioned above).
- Upload it to Google Search Console under Crawl > Sitemaps.
Once you've tested and submit your sitemap within Google's Search Console, you can use Webmaster Tools to monitor the indexation of your site and identify any errors which may result from updating your website's URLs, etc. which require you to update your sitemap.
Final Thoughts: Creating Multiple XML Sitemaps for a Large Website
One final option which is worth considering if you're generating an XML sitemap for a large website, is to segment your website, and create sitemaps for each directory.
An example of this is a site like RentalCars.com - a massive website which includes numerous location-specific landing pages in multiple languages. They segment their sitemap by language, so that each language directory of their site lives in a separate sitemap. If you look at that website's robots.txt file you'll see that they reference a single sitemap XML file which is an index of all of their individual sitemaps.
The advantage of taking this tiered sitemap approach is that you can submit each sitemap individually in Search Console and closely monitor indexation so that it's very easy to identify sections of your site which aren't as well represented in Google's search index, and take steps to improve those sections.
Even if you don't maintain a website as large as this, creating smaller focused sitemaps for different directories of your site can offer you the same ability to make important decisions while maintaining your website.
I hope this post has been helpful to you and answered some of your questions about how to create an XML sitemap with Screaming Frog. If you have any questions or thoughts on something I've missed please feel free to leave a comment below.
An example of this is a site like RentalCars.com - a massive website which includes numerous location-specific landing pages in multiple languages. They segment their sitemap by language, so that each language directory of their site lives in a separate sitemap. If you look at that website's robots.txt file you'll see that they reference a single sitemap XML file which is an index of all of their individual sitemaps.
The advantage of taking this tiered sitemap approach is that you can submit each sitemap individually in Search Console and closely monitor indexation so that it's very easy to identify sections of your site which aren't as well represented in Google's search index, and take steps to improve those sections.
Even if you don't maintain a website as large as this, creating smaller focused sitemaps for different directories of your site can offer you the same ability to make important decisions while maintaining your website.
I hope this post has been helpful to you and answered some of your questions about how to create an XML sitemap with Screaming Frog. If you have any questions or thoughts on something I've missed please feel free to leave a comment below.
You Might Also Like: My Keyword Research Guide or My Guide to Improving Your Site's SEO CTR