read a directory and write the output into a file using streams

Hello,
As a small personal challenge i am trying to use streams for:
1.read a directory (__dirname in here)
2.transform the output as upperCase
3.write the transfomed output into a file
I found this solution, but i doubt that it is the best practice... May someone confirm ? or correct ?
const fs = require('fs') const { Transform, pipeline, Readable } = require('stream') const streamUpper = () => { return new Transform({ transform(chunk, enc, next) { next(null, chunk.toString().toUpperCase()) } }) } async function readDirStream() { const dir = await fs.promises.opendir(__dirname) for await (const dirent of dir) { await require('util').promisify(pipeline)( Readable.from(dirent.name+'\n'), streamUpper(), fs.createWriteStream('./fileList.txt', {flags:'a'}) ) } } readDirStream().catch(e => console.error(e))
Comments
-
I just notice that the Readable.from() (line 17) is useless, that is enought
await require('util').promisify(pipeline)( dirent.name+'\n', streamUpper(), fs.createWriteStream('./fileList.txt', {flags:'a'}) )
So it makes me try the course exemple "Reading Directories (.cont)", as there is the same Readable.from() consuming the output from opendir(). And it works without (Readable.from()) as well. I guess it makes sense as opendir() return already a stream, am i right ?
exemple from the course without Readable.from()
opendir(__dirname, (err, dir) => { if (err) { res.statusCode = 500 res.end('Server Error') return } // const dirStream = Readable.from(dir) const entryStream = createEntryStream() res.setHeader('Content-Type', 'application/json') pipeline(dir, entryStream, res, (err) => { if (err) console.error(err) }) })
0 -
- Avoid doing inline
require
- always putrequire
at the top Readable.from
is a convenient way to turn a data structure (string, array) into a readable stream- It's not needed in your directory processing example, and really neither is the transform stream
- A better example to use with your transform stream would be a TCP socket:
const { Transform, pipeline } = require('stream') const streamUpper = () => { return new Transform({ transform(chunk, enc, next) { next(null, chunk.toString().toUpperCase()) } }) } const { createServer } = require('net') createServer((socket) => { pipeline(socket, streamUpper(), socket, (err) => { if (err) console.error(err) else console.log('socket closed') }) }).listen(9999)
Readable.from
would be useful for testing your stream:const { Transform, Readable } = require('stream') const assert = require('assert').strict const streamUpper = () => { return new Transform({ transform(chunk, enc, next) { next(null, chunk.toString().toUpperCase()) } }) } const upper = streamUpper() Readable.from(['test-a', 'test-b']).pipe(upper) upper.once('data', (first) => { assert.equal(first, 'TEST-A') upper.once('data', (second) { assert.equal(first, 'TEST-B') }) })
1 - Avoid doing inline
-
Hello David,
Thank you for the answer, i will work on that. Thank you.
1
Categories
- All Categories
- 50 LFX Mentorship
- 103 LFX Mentorship: Linux Kernel
- 553 Linux Foundation Boot Camps
- 296 Cloud Engineer Boot Camp
- 119 Advanced Cloud Engineer Boot Camp
- 52 DevOps Engineer Boot Camp
- 53 Cloud Native Developer Boot Camp
- 4 Express Training Courses
- 4 Express Courses - Discussion Forum
- 1.9K Training Courses
- 18 LFC110 Class Forum
- 7 LFC131 Class Forum
- 25 LFD102 Class Forum
- 150 LFD103 Class Forum
- 17 LFD121 Class Forum
- LFD137 Class Forum
- 61 LFD201 Class Forum
- LFD210 Class Forum
- LFD210-CN Class Forum
- 1 LFD213 Class Forum - Discontinued
- 128 LFD232 Class Forum
- LFD237 Class Forum
- 23 LFD254 Class Forum
- 598 LFD259 Class Forum
- 102 LFD272 Class Forum
- 1 LFD272-JP クラス フォーラム
- LFD273 Class Forum
- 2 LFS145 Class Forum
- 24 LFS200 Class Forum
- 739 LFS201 Class Forum
- 1 LFS201-JP クラス フォーラム
- 3 LFS203 Class Forum
- 69 LFS207 Class Forum
- 300 LFS211 Class Forum
- 54 LFS216 Class Forum
- 47 LFS241 Class Forum
- 41 LFS242 Class Forum
- 37 LFS243 Class Forum
- 11 LFS244 Class Forum
- 34 LFS250 Class Forum
- 1 LFS250-JP クラス フォーラム
- LFS251 Class Forum
- 140 LFS253 Class Forum
- LFS254 Class Forum
- 1K LFS258 Class Forum
- 10 LFS258-JP クラス フォーラム
- 92 LFS260 Class Forum
- 130 LFS261 Class Forum
- 32 LFS262 Class Forum
- 79 LFS263 Class Forum
- 15 LFS264 Class Forum
- 11 LFS266 Class Forum
- 17 LFS267 Class Forum
- 17 LFS268 Class Forum
- 23 LFS269 Class Forum
- 203 LFS272 Class Forum
- 1 LFS272-JP クラス フォーラム
- LFS281 Class Forum
- 221 LFW211 Class Forum
- 167 LFW212 Class Forum
- SKF100 Class Forum
- 902 Hardware
- 219 Drivers
- 74 I/O Devices
- 44 Monitors
- 115 Multimedia
- 209 Networking
- 101 Printers & Scanners
- 85 Storage
- 761 Linux Distributions
- 88 Debian
- 66 Fedora
- 15 Linux Mint
- 13 Mageia
- 24 openSUSE
- 141 Red Hat Enterprise
- 33 Slackware
- 13 SUSE Enterprise
- 356 Ubuntu
- 477 Linux System Administration
- 41 Cloud Computing
- 69 Command Line/Scripting
- Github systems admin projects
- 95 Linux Security
- 77 Network Management
- 108 System Management
- 49 Web Management
- 66 Mobile Computing
- 23 Android
- 29 Development
- 1.2K New to Linux
- 1.1K Getting Started with Linux
- 536 Off Topic
- 131 Introductions
- 216 Small Talk
- 21 Study Material
- 817 Programming and Development
- 275 Kernel Development
- 508 Software Development
- 928 Software
- 260 Applications
- 184 Command Line
- 3 Compiling/Installing
- 76 Games
- 316 Installation
- 59 All In Program
- 59 All In Forum
Upcoming Training
-
August 20, 2018
Kubernetes Administration (LFS458)
-
August 20, 2018
Linux System Administration (LFS301)
-
August 27, 2018
Open Source Virtualization (LFS462)
-
August 27, 2018
Linux Kernel Debugging and Security (LFD440)