How can I paginate beyond 10,000 documents in Firebase without exceeding Firestore's limit?

When you paginate with Firestore by trying to "skip" (e.g., using an offset equal to (page‑1) × limit), you quickly hit the hard limit Firestore enforces on offsets (10,000 in your case). This isn’t an issue with using startAfter per se—it’s the large offset query used to locate the starting point that causes the problem.

Why the Error Occurs

Even if you’re using startAfter for pagination, both of your approaches are first fetching a large batch of documents (using .limit((page‑1) × limit)) to determine the last document of the previous page. When page 11 is requested with limit 1000, this results in an offset of 10,000 documents—a limit that Firestore does not support.

Recommended Solutions

1. Cache Pagination Cursors:
   Instead of recalculating the offset by fetching previous documents every time, store the end cursor (last document snapshot) for each page as you navigate through your results. Then, when the user requests page 11, use the cached cursor from page 10 to start the next query. This eliminates having to fetch 10,000 records just to determine the starting point.

   Example Approach:

   • When fetching page 1, store the last document snapshot in an array (e.g., cursors[1]).
   • When fetching page 2, use cursors[1] as the starting point and then store the new last document snapshot as cursors[2].
   • Continue this way so that moving to any subsequent page uses a quickly retrievable cursor without a heavy offset query.

2. Adopt “Infinite Scroll” or “Load More” Patterns:
   Instead of letting users jump directly to arbitrarily high pages, consider a user experience that loads additional documents as needed. This prevents users from hitting the 10,000 document barrier while still providing seamless scrolling through the dataset.

3. Restructure Data Querying:
   If you need random access into the dataset (for example, direct page jumps), you may need to redesign the data structure or maintain an index that maps page numbers to cursors so that you don’t have to compute the offset via a query on the fly.

Updated Code Example Using a Cached Cursor

Below is an example sketch of how you might modify your action to use a cached cursors object. (This assumes you manage the cursors somewhere in your state.)

JAVASCRIPTCopied
// Assume cursors is an object: { 1: lastDocSnapshotPage1, 2: lastDocSnapshotPage2, ... }

rebindConv: firestoreAction(async ({ bindFirestoreRef, state, commit }, { uid, page, limit }) => {
  if (!uid) {
    throw new Error("UID is required for rebindConvosNew");
  }
  
  const conversationsRef = db.collection('Conversations')
      .where('Uid', '==', uid)
      .orderBy('DateUpdated', 'desc');
  
  let query;
  
  if (page === 1) {
    // For the first page, no starting cursor needed
    query = conversationsRef.limit(limit);
  } else {
    // Retrieve cached cursor for the previous page
    const lastDoc = state.cursors && state.cursors[page - 1];
    if (!lastDoc) {
      throw new Error('Missing pagination cursor for the requested page. Please load previous pages first.');
    }
    query = conversationsRef.startAfter(lastDoc).limit(limit);
  }
  
  // Bind the query to the 'convos' store state
  const snapshot = await query.get();
  
  // Cache the last document snapshot for future pages if available
  if (!snapshot.empty) {
    const lastSnapshot = snapshot.docs[snapshot.docs.length - 1];
    commit('setCursor', { page, cursor: lastSnapshot });
  }
  
  return bindFirestoreRef('convos', query);
})

And in your Vuex mutations (for example):

JAVASCRIPTCopied
mutations: {
  setCursor(state, { page, cursor }) {
    // Initialize the cursors object if needed
    if (!state.cursors) {
      state.cursors = {};
    }
    state.cursors[page] = cursor;
  }
}

Summary

• Avoid Large Offsets: Don’t try to compute the starting document by fetching a huge offset; this causes you to hit Firestore’s limit.
• Cache Cursors: Use the last document snapshot from each page as a cursor and store it for future pagination needs.
• Consider UX Changes: Evaluate whether a “load more” or infinite scroll pattern might work better for your needs, as it naturally avoids this problem.

By implementing these changes, you can bypass the 10,000 document limit and efficiently paginate large collections in Firestore.