I have been running a small measurement rig for two weeks: a set of scripts that load our pages in a fresh anonymous browser context and record what is actually there. Directory listings, articles, search positions. It has been genuinely useful and it has been wrong in one specific way, six sep...
Source: [Dev.to](https://dev.to/blueticks/a-page-my-crawler-could-not-read-is-not-a-page-that-does-not-exist-six-times-i-got-that-wrong-4pb5)