Why doesn’t my Node.js Puppeteer script download and save images?

Your issue is likely due to the busy-wait (the while loop) you’re using to delay iterations. Since Node.js is single-threaded, blocking the event loop prevents the asynchronous callbacks (such as the file's "finish" event) from ever executing.

Explanation

The code you posted uses a simple while loop:

JAVASCRIPTCopied
var milliseconds = 300;
var waitTill = new Date(new Date().getTime() + milliseconds);
while(waitTill > new Date()) {}

This loop isn’t yielding control back to Node’s event loop. As a result, when you call http.get and pipe the response to the file, Node never gets around to processing the response’s events, so you'll see no "File downloaded!" messages and no file writes actually occurring.

How to Fix It

Remove the blocking while loop. If you need to introduce a delay between downloads, consider using asynchronous patterns such as using promises with setTimeout, or using libraries that support asynchronous task sequencing.

Here’s a revised version without the blocking delay:

JAVASCRIPTCopied
const downloadImage = (key) => {
  return new Promise((resolve, reject) => {
    const filePath = gp_args['local_path_start'] + '/' + key + '.jpg';
    const file = fs.createWriteStream(filePath);
    
    console.log('getting ' + filePath);
    console.log('from ' + gp_args['url'] + 'img/' + key + '.jpg');

    http.get(gp_args['url'] + 'img/' + key + '.jpg', res => {
      res.pipe(file);
      file.on('finish', () => {
        file.close();
        console.log(`File downloaded! ${key}.jpg`);
        resolve();
      });
    })
    .on('error', err => {
      console.log('Error: ', err.message);
      reject(err);
    });
  });
};

const downloadImages = async () => {
  for(let x=0; x<card_arr.length; x++){
    try{
      await downloadImage(card_arr[x]);
      // Optionally add a delay between downloads
      await new Promise(resolve => setTimeout(resolve, 300));
    } catch(err) {
      console.error('Failed to download image: ', card_arr[x], err);
    }
  }
};

downloadImages();

Key Changes

• Removed the blocking while loop: Instead, asynchronous delays are introduced using setTimeout wrapped inside a promise.
• Structured with Promises/Async-Await: This ensures that each image is fully downloaded (or an error is handled) before moving on.
• Better error handling: Errors are caught and logged in the catch block.

This pattern keeps the Node.js event loop free to process asynchronous events and should allow your images to be saved properly.