Clarifying ProgressiveDataSet Streaming and Performance in Kusto REST API V2 Using .NET SDK

Below is a summary of current understanding and discussion points for your questions. Note that official documentation (at the provided link) leaves many details unspecified. The points below are based on observed behavior, inferences from the available API surface, and community experience:

---

1. How Does the ProgressiveDataSet Work?

SDK Data Retrieval (HTTP):
• The SDK initiates an HTTP request to the endpoint and establishes a streaming connection.
• Instead of waiting for the entire result set to be available (as with a traditional full-response payload), the response is sent in “chunks” or segments.
• Data is received progressively over a single long-lived request (often implemented using HTTP chunked transfer encoding), which allows immediate processing of the first available results.

Frames Explanation:
• In a ProgressiveDataSet, result content is delivered in units referred to as “frames.”
• A frame generally represents a fragment of a table (i.e. a partial result set) that includes both metadata (such as schema information) and a subset of records.
• While specifics (like maximum size) aren’t documented, frames are designed to be small enough to arrive quickly yet large enough to reduce the overhead of too many network calls.
• Advantages of this approach include lower perceived latency (since some data is available immediately) and the ability to process data “on the fly.”

Overhead Considerations:
• There is some added overhead—each frame may include extra protocol or framing metadata, and the client must manage the state of an ongoing stream.
• Additionally, if the server sends many small frames rather than fewer larger ones, the extra framing and potential HTTP chunking overhead may contribute to a noticeable delay (e.g., the ~500 ms difference you’ve observed under load).

---

2. Differences in HTTP Communication (v1 vs. v2)

HTTP Traffic for v2 Queries:
• With the v2 (streaming) endpoint, the HTTP response is designed to stream multiple “frames” in a single connection rather than returning one monolithic payload.
• This means once the query processing begins, the client receives a series of smaller HTTP chunks (frames) rather than waiting for one bulk result.

Differences Compared to v1:
• v1: Typically returns all data together when the query has fully executed. This might result in lower per-frame overhead, but also higher latency before any data is available.
• v2: Supports progressive delivery. The protocol is optimized for continuous streaming where data can be processed as soon as it arrives.
• The differences in underlying HTTP traffic (multiple chunks vs. a single response) can affect latency: while v2 provides earlier access to data, the management of multiple frames/chunks introduces an overhead that might explain the slower average response times in some scenarios.

---

3. Performance Issues

Load Test Observations:
• Your test requesting up to 100,000 rows shows that the streaming (v2) approach is around 500 ms slower in aggregate measurements.
• This extra delay could stem from:

• The overhead of processing multiple frames.
• Multiple, smaller HTTP chunks that each incur networking and client-side processing overhead.

Practical Perspective:
• For scenarios that benefit from early availability of first results, v2 may still be advantageous despite some overhead.
• In cases where the full result set is required immediately (and initial latency is less critical), the traditional (v1) full-response approach may perform better.

---

4. Use-Cases for v1 vs. v2

When to Use v2 (ProgressiveDataSet / Streaming):
• When your application benefits from low latency by processing data as soon as it arrives.
• Use cases where you want to “pipe” data from the backend (ADX) to the frontend continuously.
• Situations where the user experience is improved by displaying partial results immediately rather than waiting for the full data load.

Considerations Regarding Data Size:
• The streaming approach is particularly effective when rows are processed incrementally.
• If your records are large or if frames become too numerous and small, the overhead might overcome the benefits.
• Conversely, if you can balance frame size and frequency, the v2 streaming model may reduce overall latency.

---

5. Logging and Ensuring Data Streams in Batches

Logging As Data Arrives:
• In your code sample (using an async iterator to process frames), you’re already handling logging of the first frame to measure TTFB (Time To First Byte).
• To ensure that data is not received in one huge batch, you can log each frame or even individual records as they’re processed:

CSHARPCopied
private async IAsyncEnumerable<CompleteModel> StreamResultsAsync(Task<ProgressiveDataSet> queryTask)
{
    using (var dataSet = await queryTask)
    {
        TrackFirstResponse("S-Performance-Fixed-20VUs"); // Log first frame received
        using (var frames = dataSet.GetFrames())
        {
            while (frames.MoveNext())
            {
                var frame = frames.Current;
                // Log the arrival of each frame along with metadata like size or timestamp
                LogFrameArrival(frame);
                
                if (frame.FrameType == FrameType.TableFragment)
                {
                    var frameFragment = frame as ProgressiveDataSetDataTableFragmentFrame;
                    var record = new object[frameFragment.FieldCount];
                    while (frameFragment.GetNextRecord(record))
                    {
                        // Log each record if required (be mindful of performance and log volume)
                        LogRecord(record);
                        
                        yield return new Model
                        {
                           // Map object array to your domain model properties
                        };
                    }
                }
            }
        }
    }
}

Overhead Discussion:
• The approach of multiple HTTP requests (or chunks within a long-lived HTTP connection) does introduce extra network and parsing overhead.
• The algorithm is optimized for progressive consumption and may therefore sacrifice some raw throughput for lower first-item latency.

---

Final Thoughts

With the current (sparse) documentation, much of the behavior of ProgressiveDataSet remains largely “internal” and subject to change. The trade-offs—between early delivery via streaming (v2) and bulk transfer (v1)—should be evaluated based on your specific use case. For continuous UI updates, v2 is engineered to reduce perceived latency, even if it comes at the cost of some additional overhead. For batch queries, the older approach might deliver slightly faster overall response times when latency isn’t crucial.

If further details are required, consider reaching out to Microsoft support channels or product teams who may offer more internal insights.