How can I optimize parallel uploads and progress tracking for images and videos to Firebase Storage and Firestore?

Below is one approach to speed up your uploads and add progress tracking. The key ideas are:

1. Parallelize Work
   Instead of running a loop with sequential awaits, you can create a list of Futures for each asset. This lets you compress and start uploads simultaneously. Use Future.wait once all uploads are started.

2. Listen to Upload Progress
   Instead of waiting only for the final download URL, attach a snapshot listener to each upload. This way you can update a progress indicator as Firebase Storage’s task reports its state.

3. Batch Your Firestore Writes
   Once you have all download URLs and asset data, add them to a write batch and commit them in one go. This minimizes the number of network calls to Firestore.

Below is an example that combines these ideas:

---

DARTCopied
Future<void> uploadAssets({
  required List<AssetEntity> assets,
  required String bucketPath,
}) async {
  final firestore = FirebaseFirestore.instance;
  final List<Asset> assetModels = [];
  final WriteBatch batch = firestore.batch();
  final List<Future<void>> uploadFutures = [];

  // To track overall progress
  int totalAssets = assets.length;
  int completedAssets = 0;

  // Function to update the progress (you can hook this into your UI)
  void updateProgress() {
    double progress = completedAssets / totalAssets;
    print("Overall upload progress: ${(progress * 100).toStringAsFixed(1)}%");
  }

  for (final asset in assets) {
    uploadFutures.add(() async {
      late Uint8List? compressedData;
      late Uint8List? compressedThumbnailData;
      late String contentType;
      late String filename;
      late String downloadUrl;
      late String thumbnailUrl;

      // 1. Compress the asset (using VideoCompress/FlutterImageCompress)
      //    You can perform the compression either sequentially here (if each task is fast) or offload to isolates.
      if (asset.type == AssetType.video) {
        // Example: compress video and extract thumbnail data
        // compressedData = await compressVideo(asset);
        // compressedThumbnailData = await compressThumbnail(asset);
        contentType = "video/mp4";  // Adjust as needed
        filename = "${asset.id}.mp4";
      } else {
        // Example: compress image
        // compressedData = await FlutterImageCompress.compressWithList(...);
        contentType = "image/jpeg";
        filename = "${asset.id}.jpg";
      }

      // 2. Upload to Firebase Storage with progress tracking.
      //    Create a storage Reference as needed.
      final storageRef = FirebaseStorage.instance.ref().child(bucketPath).child(filename);

      // Upload raw data
      final uploadTask = storageRef.putData(
        compressedData!,
        SettableMetadata(contentType: contentType),
      );

      // Listen to progress changes for this upload
      uploadTask.snapshotEvents.listen((TaskSnapshot snapshot) {
        double progress = snapshot.bytesTransferred / snapshot.totalBytes;
        print("File $filename progress: ${(progress * 100).toStringAsFixed(1)}%");
        // You can update UI components here
      });

      // Wait for upload to finish then retrieve the download URL
      final TaskSnapshot snapshot = await uploadTask;
      downloadUrl = await snapshot.ref.getDownloadURL();

      // Optionally, upload the thumbnail and get its URL here (similar process)
      // thumbnailUrl = await uploadThumbnailIfNeeded(...);

      // 3. Create your Asset model with the obtained URLs and other data.
      //    Use getAssetGeoPoint() if needed.
      GeoPoint? location = await getAssetGeoPoint(asset);
      final assetModel = Asset(
        id: asset.id,
        downloadUrl: downloadUrl,
        thumbnailUrl: thumbnailUrl,
        contentType: contentType,
        filename: filename,
        location: location,
        // assign additional fields...
      );
      assetModels.add(assetModel);

      // 4. Prepare Firestore write
      final docRef = firestore.collection("assets").doc(asset.id);
      batch.set(docRef, assetModel.toMap());

      // Update overall progress counter
      completedAssets++;
      updateProgress();
    }());
  }

  // Wait for all uploads to finish.
  await Future.wait(uploadFutures);

  // 5. Commit your batch write to Firestore
  await batch.commit();
  print("All assets uploaded and Firestore updated");
}

---

Explanation

• Parallel Uploads:
  Each asset’s work (compression, upload, URL retrieval, and Firestore document creation) is wrapped inside a Future added to a list. Running them with Future.wait lets you perform uploads in parallel rather than sequentially.

• Progress Tracking:
  The snapshotEvents listener on each upload logs the individual file's progress. You can expand the updateProgress() function to combine these into an overall progress indicator.

• Batch Writing:
  Once each asset’s upload is complete and its static information is ready, you add a set operation on the Firestore WriteBatch. After all uploads, one commit saves everything.

Using this pattern should improve your overall throughput (thanks to parallel processing) and give you a more responsive progress tracking system. Adjust the compression calls and thumbnail processing as needed for your project.