sync generators js post

This commit is contained in:
wgroeneveld 2020-06-11 17:15:26 +02:00
parent eb43f4ec8b
commit af25508940
4 changed files with 163 additions and 5 deletions

View File

@ -13,6 +13,7 @@ enableGitInfo = true
[params]
fathomid = "WNLTO"
favicon = "/img/avatar-icon.png"
# disabled to get rid of cloudflare cookies
MathJax = false
@ -64,7 +65,7 @@ enableGitInfo = true
[[menu.main]]
name = "Archives"
pre = "<svg width='24' height='24'><use xlink:href='#tag'></use></svg> "
url = "/tags"
url = "/archives"
weight = 3
[[menu.mainright]]

View File

@ -0,0 +1,148 @@
---
title: "Combining async with generators in Node 11"
date: '2020-06-11'
subtitle: "Feel like blowing up your colleagues heads? Use async function*!"
tags:
- javascript
- node
---
Whizbang time! So, I was in need of a simple resursive Javascript function that iterates over all directories to look for `.md` Markdown files, to feed it to a search indexer. This is a node script my deploy script utilizes when updating this website. My Hugo content directory looks like this nowadays:
```
content/
post/
2020/
06/
article1.md
article2.md
05/
article3.md
2019/
12/
article4.md
```
This means that the URL structure of articles on Brain Baking changed (for the better) to [/post/2020/06/combining-async-with-generators-in-node/](/post/2020/06/combining-async-with-generators-in-node/). Instead of simply using `fs.readdir()` in the `content/` folder, I had to use a bit of recursion because of the slightly more complex directory structure.
And then I found an interesting [Stackoverflow post](https://stackoverflow.com/questions/5827612/node-js-fs-readdir-recursive-directory-search), explaining how to do it in node 8, 10.10+, and 11+. And my god, things did not stand still, did they. As is usually the case in the JS world.
### Node 8: Using await/async
Since most filesystem related node calls are asynchronous, you use a good amount of `await` and `async` when writing the code:
```js
const { promisify } = require('util');
const { resolve } = require('path');
const fs = require('fs');
const readdir = promisify(fs.readdir);
const stat = promisify(fs.stat);
async function getFiles(dir) {
const subdirs = await readdir(dir);
const files = await Promise.all(subdirs.map(async (subdir) => {
const res = resolve(dir, subdir);
return (await stat(res)).isDirectory() ? getFiles(res) : res;
}));
return files.reduce((a, f) => a.concat(f), []);
}
```
What's happening? Nothing special, once you get the hang of Javascript's [async functions](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function). Usage is also simple: `getFiles('post').then(doStuffWithFiles)`. In fact, `async` is just syntactic sugar for `Promise.resolve()`. According to MDN, this:
```js
async function sup() {
return "yo";
}
```
is equivalent to:
```js
function sup() {
return Promise.resolve("yo");
}
```
since nothing asynchronous is happening at all, the `Promise` object can immediately resolve, and you can carry on with doing whatever it is you plan on doing. Oh, and `util.promisify()` is a Node 8 trinket that converts callback-based functions into Promise-based ones. The last function of `readdir()` is a callback function, and otherwise, we would end up nesting functions in functions in functions in ... - the only requirement is that you follow Node's callback style. A function template like `const myfn = (delay, callback) => { };` fits.
And yes, there's a [polyfill for that](https://github.com/ljharb/util.promisify).
### Node 10.10: Spreading it out
```js
const { resolve } = require('path');
const { readdir } = require('fs').promises;
async function getFiles(dir) {
const dirents = await readdir(dir, { withFileTypes: true });
const files = await Promise.all(dirents.map((dirent) => {
const res = resolve(dir, dirent.name);
return dirent.isDirectory() ? getFiles(res) : res;
}));
return Array.prototype.concat(...files);
}
```
The spread operator `...` makes our `reduce()` redundant, but the most important change here is `requie('fs').promises` instead of `promisify()` and a slightly changed API usage of that. Things are starting to get a bit woozy here.
For instance, did you know that the [Bluebird Promise library](http://bluebirdjs.com/docs/why-bluebird.html) claims to be significantly faster than native ES6 Promise implementations? Of course, there are other reasons to resort to Bluebird, such as compatibility and utility functions. It should work even in Netscape! Yay - who still cares about that?
### Node 11: Combining async with generator functions
```js
const { resolve } = require('path');
const { readdir } = require('fs').promises;
async function* getFiles(dir) {
const dirents = await readdir(dir, { withFileTypes: true });
for (const dirent of dirents) {
const res = resolve(dir, dirent.name);
if (dirent.isDirectory()) {
yield* getFiles(res);
} else {
yield res;
}
}
}
```
Head = blown. Don't forget to change it's usage, since we're returning item per item using `yield`, we should iterate over the results:
```js
(async () => {
for await (const f of getFiles('.')) {
console.log(f);
}
})()
```
Let's take a few steps back. What's a `function*`? A really, _really_ badly chosen syntax for a _generator function_, that, according to MDN, is:
> Generators are functions that can be exited and later re-entered. Their context will be saved across re-entrances.
You simply "exit" the function using `yield` and get access to the returned object or primitive using `fn.next().value`. As long as `yield` has indeed something to yield, it will not return `undefined`. This is mostly handy to create so-called unlimited functions that seemingly never exit, except that they do. A simple generator, inside an ES6 class (now that we're at it we might as well go all out) could look like this:
```js
class Ids {
*nextId () {
let index = 0
while(true) yield index++
}
}
const id = new Ids()
const gen = id.nextId()
console.log(gen.next().value) // prints 0
console.log(gen.next().value) // prints 1
console.log(gen.next().value) // ... you know the drill
```
(Did you see what I did there with the `;` semicolons?)
Inside generator functions, you can yield other generator function return values using `yield*`. Inception! Now what If I want to call some `async` code in between different `yield`s? Then it's time for some Asperine and an in-depth blog post qwtel wrote about [async generators and it's usage](https://qwtel.com/posts/software/async-generators-in-the-wild/).
Lastly, what is that strange `for await` syntax? That's the "idiomatic" way to consume async generator functions. Do I even want to know? Here's the [MDN documentation](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for-await...of), it would take me too far to go into detail. Async iterators are part of ECMAScript 2018 (ES9), that is only 100% supported by the `V8` engine. For those of you who cannot wait, there are always polyfills...
I have to admit that [keeping up with the ECMA-262 standard](https://itnext.io/status-of-javascript-ecmascript-2019-beyond-5efca6a2d233) is getting extremely challenging...

View File

@ -1,5 +1,8 @@
---
title: Archives
url: /archives/
aliases:
- /tags/
bigimg: Archive.jpg
icontag: tag
---
@ -18,10 +21,10 @@ Not finding what you're looking for? Try browsing the archives:
### By year
- [2020](/post/2020) ... when Hugo 0.7 was released and I started paying attention to webdesign
- [2019](/post/2019) ... when I started taking computing education blogging seriously
- [2019](/post/2019) ... when computing education articles started appearing
- [2018](/post/2018) ... when my PhD work started and I tried writing essays in Dutch
- [2017](/post/2017) ... when self-improvement meta-posts started popping up more often
- [2016](/post/2016) ... when legacy code convinced me to successfully unit test Visual Basic 6
- [2016](/post/2016) ... when legacy software convinced me to successfully unit test Visual Basic 6 code
- [2015](/post/2015) ... when I completely forgot about blogging at all
- [2014](/post/2014) ... when I switched from mostly programming in Java to C#
- [2013](/post/2013) ... when this site was a wiki running on pmWiki, and then DokuWiki

View File

@ -2,8 +2,14 @@
<html lang="{{ .Site.LanguageCode }}">
<head>
{{ partial "head-meta" . }}
<title>{{ .Title }}</title>
{{ if eq .Title .Site.Title }}
<title>{{ .Title }}</title>
{{ else }}
<title>{{ .Site.Title }}: {{ .Title }}</title>
{{ end }}
{{ if .Site.Params.favicon }}
<link rel="apple-touch-icon" href="{{ .Site.Params.favicon | absURL }}" />
<link rel="image_src" href="{{ .Site.Params.favicon | absURL }}">
<link rel="icon" href="{{ .Site.Params.favicon | absURL }}">
{{ end }}
{{ partial "css" . }} {{ partial "js" . }} {{ hugo.Generator }}
@ -20,7 +26,7 @@
<nav class="navbar">
<div class="container">
<div class="navbar-header">
<a class="navbar-brand visible-xs" href="#">{{ .Title }}</a>
<a class="navbar-brand visible-xs" href="#">{{ .Site.Title }}</a>
<button class="navbar-toggle" aria-label="navigation toggle">
<span class="hidden" aria-hidden="true">navigation toggle</span>