I was speaking with a colleague this week who was having trouble with an eCommerce website they were working on. 

They knew there were hundreds of pages on the website, but Screaming Frog was only showing 50! The sitemap was included in the crawl, and hundreds of pages were flagged as orphaned pages.  

This didn't make sense, as they were sure the pages were linked internally! 

So why weren't they being found in the Screaming Frog crawl?

This was becoming a real headache!

I had a look at the site and realised that it was built with JavaScript, and the category pages were using JS widgets to share the products.

When JS is turned off in the browser, these widgets disappear. The same goes for the crawler; if JavaScript isn't turned on, it's hard for them to know the links and content are there.  

My colleague and I discussed this, and after altering some settings in the crawl configuration to turn on JavaScript rendering, we re-ran the crawl and the hundreds of pages that were previously "missing" were now being found. 

Always check to see how the website you are working on is built, and make sure you adjust your crawl settings accordingly. Whether using Screaming Frog, Sitebulb, or another crawler, they all have options for crawling JS websites.  

Check the technology and stop getting frustrated and scratching your head thinking that hundreds or thousands of pages are missing from the site! 

So, yes, JavaScript can be a b**ch, but there's usually a way around it!