read a directory and write the output into a file using streams

Hello,
As a small personal challenge i am trying to use streams for:
1.read a directory (__dirname in here)
2.transform the output as upperCase
3.write the transfomed output into a file
I found this solution, but i doubt that it is the best practice... May someone confirm ? or correct ?
- const fs = require('fs')
- const { Transform, pipeline, Readable } = require('stream')
- const streamUpper = () => {
- return new Transform({
- transform(chunk, enc, next) {
- next(null, chunk.toString().toUpperCase())
- }
- })
- }
- async function readDirStream() {
- const dir = await fs.promises.opendir(__dirname)
- for await (const dirent of dir) {
- await require('util').promisify(pipeline)(
- Readable.from(dirent.name+'\n'),
- streamUpper(),
- fs.createWriteStream('./fileList.txt', {flags:'a'})
- )
- }
- }
- readDirStream().catch(e => console.error(e))
Comments
-
I just notice that the Readable.from() (line 17) is useless, that is enought
- await require('util').promisify(pipeline)(
- dirent.name+'\n',
- streamUpper(),
- fs.createWriteStream('./fileList.txt', {flags:'a'})
- )
So it makes me try the course exemple "Reading Directories (.cont)", as there is the same Readable.from() consuming the output from opendir(). And it works without (Readable.from()) as well. I guess it makes sense as opendir() return already a stream, am i right ?
exemple from the course without Readable.from()
- opendir(__dirname, (err, dir) => {
- if (err) {
- res.statusCode = 500
- res.end('Server Error')
- return
- }
- // const dirStream = Readable.from(dir)
- const entryStream = createEntryStream()
- res.setHeader('Content-Type', 'application/json')
- pipeline(dir, entryStream, res, (err) => {
- if (err) console.error(err)
- })
- })
0 -
- Avoid doing inline
require
- always putrequire
at the top Readable.from
is a convenient way to turn a data structure (string, array) into a readable stream- It's not needed in your directory processing example, and really neither is the transform stream
- A better example to use with your transform stream would be a TCP socket:
- const { Transform, pipeline } = require('stream')
- const streamUpper = () => {
- return new Transform({
- transform(chunk, enc, next) {
- next(null, chunk.toString().toUpperCase())
- }
- })
- }
- const { createServer } = require('net')
- createServer((socket) => {
- pipeline(socket, streamUpper(), socket, (err) => {
- if (err) console.error(err)
- else console.log('socket closed')
- })
- }).listen(9999)
Readable.from
would be useful for testing your stream:- const { Transform, Readable } = require('stream')
- const assert = require('assert').strict
- const streamUpper = () => {
- return new Transform({
- transform(chunk, enc, next) {
- next(null, chunk.toString().toUpperCase())
- }
- })
- }
- const upper = streamUpper()
- Readable.from(['test-a', 'test-b']).pipe(upper)
- upper.once('data', (first) => {
- assert.equal(first, 'TEST-A')
- upper.once('data', (second) {
- assert.equal(first, 'TEST-B')
- })
- })
1 - Avoid doing inline
-
Hello David,
Thank you for the answer, i will work on that. Thank you.
1
Categories
- All Categories
- 143 LFX Mentorship
- 143 LFX Mentorship: Linux Kernel
- 817 Linux Foundation IT Professional Programs
- 368 Cloud Engineer IT Professional Program
- 167 Advanced Cloud Engineer IT Professional Program
- 83 DevOps Engineer IT Professional Program
- 132 Cloud Native Developer IT Professional Program
- 122 Express Training Courses
- 122 Express Courses - Discussion Forum
- Microlearning - Discussion Forum
- 6.6K Training Courses
- 40 LFC110 Class Forum - Discontinued
- 66 LFC131 Class Forum
- 39 LFD102 Class Forum
- 237 LFD103 Class Forum
- 22 LFD110 Class Forum
- 44 LFD121 Class Forum
- 1 LFD123 Class Forum
- LFD125 Class Forum
- 17 LFD133 Class Forum
- 6 LFD134 Class Forum
- 17 LFD137 Class Forum
- 70 LFD201 Class Forum
- 3 LFD210 Class Forum
- 2 LFD210-CN Class Forum
- 2 LFD213 Class Forum - Discontinued
- 128 LFD232 Class Forum - Discontinued
- 1 LFD233 Class Forum
- 3 LFD237 Class Forum
- 23 LFD254 Class Forum
- 689 LFD259 Class Forum
- 110 LFD272 Class Forum
- 3 LFD272-JP クラス フォーラム
- 10 LFD273 Class Forum
- 251 LFS101 Class Forum
- 2 LFS111 Class Forum
- 3 LFS112 Class Forum
- 3 LFS116 Class Forum
- 3 LFS118 Class Forum
- 1 LFS120 Class Forum
- 3 LFS142 Class Forum
- 3 LFS144 Class Forum
- 3 LFS145 Class Forum
- 1 LFS146 Class Forum
- 16 LFS148 Class Forum
- 8 LFS151 Class Forum
- 1 LFS157 Class Forum
- 70 LFS158 Class Forum
- LFS158-JP クラス フォーラム
- 5 LFS162 Class Forum
- 1 LFS166 Class Forum
- 3 LFS167 Class Forum
- 1 LFS170 Class Forum
- 1 LFS171 Class Forum
- 2 LFS178 Class Forum
- 2 LFS180 Class Forum
- 2 LFS182 Class Forum
- 4 LFS183 Class Forum
- 30 LFS200 Class Forum
- 737 LFS201 Class Forum - Discontinued
- 2 LFS201-JP クラス フォーラム
- 17 LFS203 Class Forum
- 118 LFS207 Class Forum
- 2 LFS207-DE-Klassenforum
- LFS207-JP クラス フォーラム
- 302 LFS211 Class Forum
- 55 LFS216 Class Forum
- 54 LFS241 Class Forum
- 43 LFS242 Class Forum
- 37 LFS243 Class Forum
- 13 LFS244 Class Forum
- 6 LFS245 Class Forum
- LFS246 Class Forum
- LFS248 Class Forum
- 108 LFS250 Class Forum
- 1 LFS250-JP クラス フォーラム
- LFS251 Class Forum
- 145 LFS253 Class Forum
- LFS254 Class Forum
- 2 LFS255 Class Forum
- 13 LFS256 Class Forum
- 1 LFS257 Class Forum
- 1.3K LFS258 Class Forum
- 11 LFS258-JP クラス フォーラム
- 116 LFS260 Class Forum
- 156 LFS261 Class Forum
- 41 LFS262 Class Forum
- 82 LFS263 Class Forum - Discontinued
- 15 LFS264 Class Forum - Discontinued
- 11 LFS266 Class Forum - Discontinued
- 23 LFS267 Class Forum
- 25 LFS268 Class Forum
- 29 LFS269 Class Forum
- 7 LFS270 Class Forum
- 200 LFS272 Class Forum
- 1 LFS272-JP クラス フォーラム
- 2 LFS147 Class Forum
- LFS274 Class Forum
- 3 LFS281 Class Forum
- 18 LFW111 Class Forum
- 257 LFW211 Class Forum
- 179 LFW212 Class Forum
- 15 SKF100 Class Forum
- SKF200 Class Forum
- 2 SKF201 Class Forum
- 791 Hardware
- 199 Drivers
- 68 I/O Devices
- 37 Monitors
- 98 Multimedia
- 174 Networking
- 91 Printers & Scanners
- 85 Storage
- 754 Linux Distributions
- 82 Debian
- 67 Fedora
- 16 Linux Mint
- 13 Mageia
- 23 openSUSE
- 149 Red Hat Enterprise
- 31 Slackware
- 13 SUSE Enterprise
- 351 Ubuntu
- 465 Linux System Administration
- 39 Cloud Computing
- 71 Command Line/Scripting
- Github systems admin projects
- 95 Linux Security
- 78 Network Management
- 101 System Management
- 47 Web Management
- 56 Mobile Computing
- 18 Android
- 28 Development
- 1.2K New to Linux
- 1K Getting Started with Linux
- 366 Off Topic
- 114 Introductions
- 171 Small Talk
- 26 Study Material
- 534 Programming and Development
- 304 Kernel Development
- 223 Software Development
- 1.8K Software
- 212 Applications
- 182 Command Line
- 3 Compiling/Installing
- 405 Games
- 311 Installation
- 79 All In Program
- 79 All In Forum
Upcoming Training
-
August 20, 2018
Kubernetes Administration (LFS458)
-
August 20, 2018
Linux System Administration (LFS301)
-
August 27, 2018
Open Source Virtualization (LFS462)
-
August 27, 2018
Linux Kernel Debugging and Security (LFD440)