Have you ever wondered how your business website or blog actually gets indexed on Google or Bing so it can display for a searcher? Come on, you know you’ve thought of it at least once...
The short answer; is they do this through two primary methods (overly simplified here)
They crawl and harvest the sitemap on your website, which lists every page and every URL to things like pages, posts, images, blog categories & tags, and files. This is the preferred method.
Or...they actually visit your website via a coded web crawler (often called a bot, a spider, or [surprise] a spiderbot - who names this stuff?). These crawlers navigate your website just like a human would by clicking and entering every page that is listed in your visible navigation and the inner pages of your website as they are linked to each other.
This is where things either work, or fail.
In the first scenario(sitemap crawl), if your website doesn't have a sitemap, bots will default to step-two. If your website does have a sitemap (congrats🥳), pull it up and read it…right now, go ahead - do it. Do you see any pages on it that would not answer a lead or prospect’s question or that do not generate leads or sales? If you have login, policy, conversion, or shipping pages - these will not drive sales for your business, and you need to optimize your sitemap so it only contains the pages that either inform a visitor, or create leads & sales. This is referred to as sitemap pruning. Everything else, probably belongs on your site - but not on your sitemap.
The second scenario(site crawl) is probably the worst option for search engines - and your business. Unless every page of your website is inter-linked from another page of your website, or through your navigation, you’re only going to get partially indexed, and therefore will not show up on search engines. Add to this, the same issue of having login, shipping, conversion, and policy pages on your site without being properly coded to prevent them from showing up on search engines; and you’re certain to be limiting your branding, market share, and revenue potential.
So what to do?
Be certain to have proper sitemaps (pages, posts, & image ones are most common)
Add all these sitemaps into your robots.txt file - if you are able
Prune your sitemap of any pages that are not informational, or lead/sale generating
Finally, add your sitemaps to both Google Search Console and Bing Webmaster Tools so they can find all your awesome website pages, blog posts, and images!
And lastly, get in the habit of thinking about your sitemap as you add or change content