Class SegmentedReadUtil

Namespace

Hi.Common.ParallelBulkUtils

Assembly

HiGeom.dll

Reads keyed rows from a store one index segment at a time, with no cache in front of the store. A segment is segmentSize consecutive indices starting at a multiple of it; a consumer that scans many rows groups the indices it needs into their segments (AlignedSegments(IEnumerable<int>, int)), reads each segment once into a dictionary and works on that segment before it reads the next, so its working set is bounded by one segment and a row written after its segment was read is not seen until the segment is read again. What the store answers for an index is authoritative: a key that is absent is a row the store does not hold, there is no stale cached absence to retry.

public static class SegmentedReadUtil

Inheritance

object

SegmentedReadUtil

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Methods

AlignedSegments(IEnumerable<int>, int)

Groups the given indices into the segments that hold them: segment k covers [k * segmentSize, (k + 1) * segmentSize). The segments come in ascending order, each with the indices it holds in ascending order; a segment no index falls in is not produced, so a sparse index set costs as many reads as it touches segments.

public static IEnumerable<(int Begin, int End, List<int> Indices)> AlignedSegments(IEnumerable<int> indices, int segmentSize)

Parameters

indices IEnumerable<int>

The indices to group, in any order; non-negative.

segmentSize int

Indices per segment; a segment starts at a multiple of it.

Returns

IEnumerable<(int Begin, int End, List<int> Indices)>

Each segment's bounds (begin inclusive, end exclusive) and its indices.

CreateOrderedGetter<TData>(int, Func<int, int, Dictionary<int, TData>>)

Builds a getter of one row at a time for a walk asked in ascending index order. The segment holding the asked index is read when the walk enters it and kept until the walk leaves it, so a walk in index order reads each segment once; a walk that goes back re-reads. A segment the store answers with null (a faulted read, reported by readSegment itself) counts as empty for its whole span.

public static Func<int, TData> CreateOrderedGetter<TData>(int segmentSize, Func<int, int, Dictionary<int, TData>> readSegment) where TData : class

Parameters

segmentSize int

Indices per store read; a segment starts at a multiple of it.

readSegment Func<int, int, Dictionary<int, TData>>

Reads the rows of [begin, end): index to row, with whatever null means to the store (a NULL column, say); absent for no row; null for a faulted read.

Returns

Func<int, TData>

The row of an index, null when the store holds none or holds null for it.

Type Parameters

TData

The row type.

ReadSegment<TData>(ReadBulkDelegate<TData>, Func<TData, int>, int, int)

Reads the rows of [begin, end) through a ReadBulkDelegate<TData> into a dictionary keyed by the index each row names. A faulted read – the delegate reports it itself and answers null – is an empty dictionary: every index of the segment then reads as a row the store does not hold.

public static Dictionary<int, TData> ReadSegment<TData>(ReadBulkDelegate<TData> readBulk, Func<TData, int> getIndex, int begin, int end)

Parameters

readBulk ReadBulkDelegate<TData>

Reads the rows of a range.

getIndex Func<TData, int>

The index a row names.

begin int

The starting index (inclusive).

end int

The ending index (exclusive).

Returns

Dictionary<int, TData>

Index to row for the rows the store holds in the range.

Type Parameters

TData

The row type.