From 854351610ce189576583671bca335ffcaf0aa56e Mon Sep 17 00:00:00 2001 From: wgroeneveld Date: Wed, 29 Mar 2023 14:46:07 +0200 Subject: [PATCH] fuck off ChatGPT. --- ...hould-build-our-own-wayback-machines-reprise.md | 2 +- layouts/robots.txt | 14 ++++++++++++++ 2 files changed, 15 insertions(+), 1 deletion(-) diff --git a/content/post/2023/03/we-should-build-our-own-wayback-machines-reprise.md b/content/post/2023/03/we-should-build-our-own-wayback-machines-reprise.md index 5975919d..826ac0a7 100644 --- a/content/post/2023/03/we-should-build-our-own-wayback-machines-reprise.md +++ b/content/post/2023/03/we-should-build-our-own-wayback-machines-reprise.md @@ -45,7 +45,7 @@ seeds: - pinterest* generateWACZ: true -text: true% +text: true ``` What happens behind the scenes is a browser that's fired up and controlled by Puppeteer, where requests, responses, and resources are recorded and links are followed according to the depth configuration. The exclude regex values don't seem to be working that well, and depending on the size of the website, Docker will be running a _long_ time, but the end result is a single archive that's yours forever! diff --git a/layouts/robots.txt b/layouts/robots.txt index 7d329b1d..39399030 100644 --- a/layouts/robots.txt +++ b/layouts/robots.txt @@ -1 +1,15 @@ +Sitemap: https://brainbaking.com/sitemap.xml User-agent: * +Disallow: + +User-agent: ChatGPT-User +Disallow: / + +User-agent: Mediapartners-Google +Disallow: / + +User-agent: AdsBot-Google +Disallow: / + +User-agent: adidxbot +Disallow: /