There are numerous tools which you can use to create a custom XML sitemap file, and if you wanted to you could even write one yourself, but in today's article I'm going to walk through how to create an XML sitemap using Screaming Frog's SEO tool - in my opinion one of the best budget SEO Tools for any individual or agency to invest in.
Do I Need an XML Sitemap?
If all of the pages on your website have links pointing to them from other pages, chances are good that Google's spider will find them on its own, but the first place any spider that comes to your website looks is in its robots.txt file, and if you've posted the location of a full list of your website's pages and/or blog posts there, it helps to ensure your pages are getting crawled as often as possible.
How do I Reference my XML Sitemap in my Robots.txt File?
Is Screaming Frog the Best Tool to Build an XML Sitemap?
I like the different options provided by Screaming Frog, and recommend it for larger websites with a more complex site structure where you plan to add more detail to the sitemap like priority, change frequency, etc. (this is discussed below). For small sites who want to spend less, you can get a great tool with the "Unlimited Sitemap Generator" standalone option from xml-sitemaps.com which will cost under $20.
Without further delay, here's the guide:
How to Create a Custom XML Sitemap with Screaming Frog
Once your site has been crawled by Screaming Frog click Sitemaps > Create XML Sitemap in the Screaming Frog menu.
Here's an overview of each window, what the settings mean and how I would recommend that you configure your XML sitemap.
XML Sitemap Pages Tab
XML Sitemap Last Modified Tag
If you choose to include this tag you can either use a custom date (if you update your sitemaps manually, it would be today's time each time you generate a new file), or you can use the server response time.
In general I recommend using the server response time if the CMS you've chosen to build your website allows you to upload a file once and have additional pages and modifications auto-update that file. If you aren't so lucky and you're tasked with regularly updating the file yourself and re-submitting, then using a custom date is a good option as well, as it has the added benefit of allowing you to keep track of how recently you've updated your sitemap and it can serve as a good reminder that it might be time to refresh your on page SEO, or update your XML Sitemap file to include recently added pages or blog posts.
XML Sitemap Priority Tag Settings
I should add that there's no hard-and-fast rule saying you have to directly follow your site content structure when assigning priority, but it's best practice to be consistent. So for example if you equally weigh the level 1 and level 2 pages in terms of their priority on your website, you can assign the homepage a priority level 1 and all level 1 and level 2 pages on your site could be assigned a priority level of 0.9 via the XML sitemap priority tag.
You can also uncheck the include option in Screaming Frog if you aren't interested in having this included in your custom XML sitemap. In general, small sites don't need this, but it's helpful if you're managing a large e-commerce website with thousands of products and category pages.
XML Sitemap Change Frequency Tag
Here you can use your content tree levels to assign the typical change frequency of different content sections of your website. If your product pages aren't updated except for monthly pricing updates, then you can indicate that by setting them to a monthly change frequency, and if your homepage and major category pages are updated every day with special offers, you can indicate that too. The settings you choose for the XML sitemap change frequency tag will be determined by your site maintenance regimen.
XML Images Sitemap
Once Your XML Sitemap is Generated What's Next?
- Reference the file's location in your robots.txt file (mentioned above).
- Upload it to Google Search Console under Crawl > Sitemaps.
Once you've tested and submit your sitemap within Google's Search Console, you can use Webmaster Tools to monitor the indexation of your site and identify any errors which may result from updating your website's URLs, etc. which require you to update your sitemap.
Final Thoughts: Creating Multiple XML Sitemaps for a Large Website
An example of this is a site like RentalCars.com - a massive website which includes numerous location-specific landing pages in multiple languages. They segment their sitemap by language, so that each language directory of their site lives in a separate sitemap. If you look at that website's robots.txt file you'll see that they reference a single sitemap XML file which is an index of all of their individual sitemaps.
The advantage of taking this tiered sitemap approach is that you can submit each sitemap individually in Search Console and closely monitor indexation so that it's very easy to identify sections of your site which aren't as well represented in Google's search index, and take steps to improve those sections.
Even if you don't maintain a website as large as this, creating smaller focused sitemaps for different directories of your site can offer you the same ability to make important decisions while maintaining your website.
I hope this post has been helpful to you and answered some of your questions about how to create an XML sitemap with Screaming Frog. If you have any questions or thoughts on something I've missed please feel free to leave a comment below.