wgroeneveld
/
brainbaking
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

fuck off ChatGPT.

This commit is contained in:
Wouter Groeneveld 2023-03-29 14:46:07 +02:00
parent 407ac62a63
commit 854351610c
2 changed files with 15 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
content/post/2023/03/we-should-build-our-own-wayback-machines-reprise.md
View File

@ -45,7 +45,7 @@ seeds:
- pinterest*
generateWACZ: true
text: true%
text: true
```
What happens behind the scenes is a browser that's fired up and controlled by Puppeteer, where requests, responses, and resources are recorded and links are followed according to the depth configuration. The exclude regex values don't seem to be working that well, and depending on the size of the website, Docker will be running a _long_ time, but the end result is a single archive that's yours forever!

14
layouts/robots.txt
View File

@ -1 +1,15 @@
Sitemap: https://brainbaking.com/sitemap.xml
User-agent: *
Disallow:
User-agent: ChatGPT-User
Disallow: /
User-agent: Mediapartners-Google
Disallow: /
User-agent: AdsBot-Google
Disallow: /
User-agent: adidxbot
Disallow: /