Welcome to the Linux Foundation Forum!

read a directory and write the output into a file using streams

Hello,

As a small personal challenge i am trying to use streams for:

1.read a directory (__dirname in here)
2.transform the output as upperCase
3.write the transfomed output into a file

I found this solution, but i doubt that it is the best practice... May someone confirm ? or correct ?

  1. const fs = require('fs')
  2. const { Transform, pipeline, Readable } = require('stream')
  3.  
  4. const streamUpper = () => {
  5. return new Transform({
  6. transform(chunk, enc, next) {
  7. next(null, chunk.toString().toUpperCase())
  8. }
  9. })
  10. }
  11.  
  12. async function readDirStream() {
  13. const dir = await fs.promises.opendir(__dirname)
  14.  
  15. for await (const dirent of dir) {
  16. await require('util').promisify(pipeline)(
  17. Readable.from(dirent.name+'\n'),
  18. streamUpper(),
  19. fs.createWriteStream('./fileList.txt', {flags:'a'})
  20. )
  21. }
  22. }
  23. readDirStream().catch(e => console.error(e))
  24.  

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Comments

  • Posts: 20

    I just notice that the Readable.from() (line 17) is useless, that is enought

    1. await require('util').promisify(pipeline)(
    2. dirent.name+'\n',
    3. streamUpper(),
    4. fs.createWriteStream('./fileList.txt', {flags:'a'})
    5. )

    So it makes me try the course exemple "Reading Directories (.cont)", as there is the same Readable.from() consuming the output from opendir(). And it works without (Readable.from()) as well. I guess it makes sense as opendir() return already a stream, am i right ?

    exemple from the course without Readable.from()

    1. opendir(__dirname, (err, dir) => {
    2. if (err) {
    3. res.statusCode = 500
    4. res.end('Server Error')
    5. return
    6. }
    7. // const dirStream = Readable.from(dir)
    8. const entryStream = createEntryStream()
    9. res.setHeader('Content-Type', 'application/json')
    10. pipeline(dir, entryStream, res, (err) => {
    11. if (err) console.error(err)
    12. })
    13. })
    • Avoid doing inline require - always put require at the top
    • Readable.from is a convenient way to turn a data structure (string, array) into a readable stream
    • It's not needed in your directory processing example, and really neither is the transform stream
    • A better example to use with your transform stream would be a TCP socket:
    1. const { Transform, pipeline } = require('stream')
    2.  
    3. const streamUpper = () => {
    4. return new Transform({
    5. transform(chunk, enc, next) {
    6. next(null, chunk.toString().toUpperCase())
    7. }
    8. })
    9. }
    10.  
    11. const { createServer } = require('net')
    12.  
    13. createServer((socket) => {
    14. pipeline(socket, streamUpper(), socket, (err) => {
    15. if (err) console.error(err)
    16. else console.log('socket closed')
    17. })
    18. }).listen(9999)

    Readable.from would be useful for testing your stream:

    1. const { Transform, Readable } = require('stream')
    2. const assert = require('assert').strict
    3. const streamUpper = () => {
    4. return new Transform({
    5. transform(chunk, enc, next) {
    6. next(null, chunk.toString().toUpperCase())
    7. }
    8. })
    9. }
    10.  
    11. const upper = streamUpper()
    12. Readable.from(['test-a', 'test-b']).pipe(upper)
    13. upper.once('data', (first) => {
    14. assert.equal(first, 'TEST-A')
    15. upper.once('data', (second) {
    16. assert.equal(first, 'TEST-B')
    17. })
    18. })
    19.  
  • Posts: 20

    Hello David,

    Thank you for the answer, i will work on that. Thank you.

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Welcome!

It looks like you're new here. Sign in or register to get started.
Sign In

Categories

Upcoming Training