I was speaking with a colleague this week who was having trouble with an eCommerce website they were working on.
They knew there were hundreds of pages on the website, but Screaming Frog was only showing 50! The sitemap was included in the crawl, and hundreds of pages were flagged as orphaned pages.
This didn't make sense, as they were sure the pages were linked internally!
So why weren't they being found in the Screaming Frog crawl?
This was becoming a real headache!
Always check to see how the website you are working on is built, and make sure you adjust your crawl settings accordingly. Whether using Screaming Frog, Sitebulb, or another crawler, they all have options for crawling JS websites.
Check the technology and stop getting frustrated and scratching your head thinking that hundreds or thousands of pages are missing from the site!