diff --git a/content/post/2023/03/we-should-build-our-own-wayback-machines-reprise.md b/content/post/2023/03/we-should-build-our-own-wayback-machines-reprise.md index 5975919d..826ac0a7 100644 --- a/content/post/2023/03/we-should-build-our-own-wayback-machines-reprise.md +++ b/content/post/2023/03/we-should-build-our-own-wayback-machines-reprise.md @@ -45,7 +45,7 @@ seeds: - pinterest* generateWACZ: true -text: true% +text: true ``` What happens behind the scenes is a browser that's fired up and controlled by Puppeteer, where requests, responses, and resources are recorded and links are followed according to the depth configuration. The exclude regex values don't seem to be working that well, and depending on the size of the website, Docker will be running a _long_ time, but the end result is a single archive that's yours forever! diff --git a/layouts/robots.txt b/layouts/robots.txt index 7d329b1d..39399030 100644 --- a/layouts/robots.txt +++ b/layouts/robots.txt @@ -1 +1,15 @@ +Sitemap: https://brainbaking.com/sitemap.xml User-agent: * +Disallow: + +User-agent: ChatGPT-User +Disallow: / + +User-agent: Mediapartners-Google +Disallow: / + +User-agent: AdsBot-Google +Disallow: / + +User-agent: adidxbot +Disallow: /