Namespace Hi.Common.ParallelBulkUtils

ParallelBulkWriter<TData>: Parallel bulk writer that efficiently processes and writes data in parallel. The writing data is buffered and processed on a separate task to improve performance. A write that throws is reported and the next batch is written as usual; whatever ends the working task opens the gate Wait() waits on and keeps it open, so no caller waits for a task that is gone, and Dispose() reports instead of throwing.

SegmentedReadUtil: Reads keyed rows from a store one index segment at a time, with no cache in front of the store. A segment is segmentSize consecutive indices starting at a multiple of it; a consumer that scans many rows groups the indices it needs into their segments (AlignedSegments(IEnumerable<int>, int)), reads each segment once into a dictionary and works on that segment before it reads the next, so its working set is bounded by one segment and a row written after its segment was read is not seen until the segment is read again. What the store answers for an index is authoritative: a key that is absent is a row the store does not hold, there is no stale cached absence to retry.

SequentialBulkReader<TData>: Sequential bulk reader that caches whole buckets of rows in front of a store; one lock, every call queues behind a bucket read. Kept for the retiring HardNc NC optimization only – every other consumer of the step cache reads its own kind of row segment by segment (SegmentedReadUtil) with no cache in front of the store.