How can I stream data dynamically with Node.js and Express over HTTPS?

Below is a list of things to check and change to get dynamic output with Node.js:

---

1. Use “No Buffer” Mode in Curl

By default, curl may buffer the output. Run curl with the --no-buffer (or -N) option:

BASHCopied
curl -N https://myurl.com/stream

---

2. Flush Headers and Response

Even though you set “Transfer-Encoding: chunked”, Node.js (and any middlewares) may buffer small chunks. Consider explicitly flushing headers and response. For example, after setting up the headers, call:

JAVASCRIPTCopied
res.flushHeaders();

This signals Node.js to send headers immediately.

If you have small chunks of data, you may try to force a flush after each write. Note that Express doesn’t provide flush by itself. You can use the npm module on-the-fly flush or ensure a middleware that adds a flush method. For example, if you add the compression middleware, it may already add a flush method (sometimes as res.flush).

Here’s a modified version of your code that flushes headers and data:

JAVASCRIPTCopied
app.get('/stream', (req, res) => {
    res.setHeader('Content-Type', 'text/plain; charset=utf-8');
    res.setHeader('Transfer-Encoding', 'chunked');

    // Immediately flush headers
    res.flushHeaders();

    const responseChunks = [
        "Hello,",
        "how are you",
        "...\n"
    ];

    let chunkIndex = 0;

    // Send data every 500ms
    const intervalId = setInterval(() => {
        if (chunkIndex < responseChunks.length) {
            res.write(responseChunks[chunkIndex]);
            // If a flush method is available, call it to force the chunk out
            if (typeof res.flush === 'function') {
                res.flush();
            }
            chunkIndex++;
        } else {
            clearInterval(intervalId);
            res.end();
        }
    }, 500);

    req.on('close', () => {
        clearInterval(intervalId);
        console.log('Stream stopped.');
    });
});

---

3. Disable Unwanted Middleware Effects

Sometimes middleware (such as compression middleware) may buffer output. Make sure that if you use compression or any response-modifying middleware, you either disable them for this route or configure them to allow streaming. For example, you can exclude the stream route from compression:

JAVASCRIPTCopied
app.get('/stream', (req, res, next) => {
    res.setHeader('Content-Type', 'text/plain; charset=utf-8');
    res.setHeader('Transfer-Encoding', 'chunked');
    
    // Disable compression for this route
    res.removeHeader("Content-Encoding");
    
    res.flushHeaders();

    // your streaming logic…
});

---

Summary

• Run curl with --no-buffer to avoid client buffering.
• Flush headers immediately with res.flushHeaders() and try to flush after each chunk (with res.flush() if available).
• Check middleware (like compression) that might inadvertently buffer the response.

Following these steps should help you stream data dynamically from your Node.js endpoint.