Iterator Pattern Real-World Example in C#: Complete Implementation
Most iterator pattern tutorials show you a custom collection with MoveNext() and Current, iterate over a handful of strings, and call it done. That's helpful for understanding the mechanics, but it doesn't address the kind of problem you'll actually face in production code -- like iterating over thousands of database records without loading them all into memory at once. This article builds a complete iterator pattern real world example in C# from the ground up: a paginated data access layer where client code iterates seamlessly with foreach while the iterator handles page fetching, lazy evaluation, and end-of-data detection behind the scenes.
By the end, you'll have a full set of compilable classes covering the synchronous iterator, an async version using IAsyncEnumerable<T>, integration with the repository pattern and dependency injection, and xUnit tests that verify the whole thing works. If you want a practical example of the iterator pattern that goes beyond toy collections and solves a real engineering problem, this is the article for you.
The Problem: Paginated Data Access
Consider an application that processes thousands of customer records from a database or external API. Loading all records into a single list would consume significant memory -- potentially hundreds of megabytes for large datasets. The data source returns results in pages of 100 records at a time, and your application needs to process each record individually.
Without the iterator pattern, every piece of code that consumes this data needs to manage pagination itself. Here's what that looks like:
public class OrderReportGenerator
{
private readonly ICustomerDataSource _dataSource;
public OrderReportGenerator(
ICustomerDataSource dataSource)
{
_dataSource = dataSource;
}
public void GenerateReport()
{
int pageNumber = 0;
const int pageSize = 100;
while (true)
{
var page = _dataSource.FetchPage(
pageNumber, pageSize);
if (page.Count == 0)
{
break;
}
foreach (var customer in page)
{
ProcessCustomer(customer);
}
pageNumber++;
}
}
private void ProcessCustomer(Customer customer)
{
// Report generation logic
}
}
This code has several problems that compound as your codebase grows. The pagination loop is interleaved with business logic, making both harder to read and test. Every consumer that needs paginated data -- report generators, data exporters, batch processors -- duplicates this same while-true-fetch-break pattern.
The real pain shows up when requirements change. If the page size needs to be configurable, you're updating every consumer. If the data source switches from offset-based to cursor-based pagination, every consumer needs rewriting. And LINQ queries? Forget about it -- you can't chain .Where() or .Select() over a manual pagination loop.
The iterator pattern solves all of this by encapsulating the pagination mechanics inside a class that implements IEnumerable<T>. Client code writes a plain foreach loop and never knows that pages are being fetched lazily underneath.
Designing the Iterator: PaginatedEnumerable
The core idea is straightforward. We need a class that implements IEnumerable<T> and accepts a function that knows how to fetch a single page of data. The class itself doesn't know anything about databases, APIs, or page sizes -- it just calls the function and yields results until the function returns an empty page.
Here's the design for PaginatedEnumerable<T>:
using System.Collections;
public sealed class PaginatedEnumerable<T>
: IEnumerable<T>
{
private readonly Func<int, IReadOnlyList<T>> _pageFetcher;
public PaginatedEnumerable(
Func<int, IReadOnlyList<T>> pageFetcher)
{
_pageFetcher = pageFetcher
?? throw new ArgumentNullException(
nameof(pageFetcher));
}
public IEnumerator<T> GetEnumerator()
{
return new PaginatedEnumerator<T>(
_pageFetcher);
}
IEnumerator IEnumerable.GetEnumerator()
{
return GetEnumerator();
}
}
The pageFetcher delegate takes a page number (zero-based) and returns a read-only list of items for that page. When the list comes back empty, iteration is complete. This delegate-based approach keeps the iterator completely decoupled from the data source -- you can pass in a function that queries a database, calls an HTTP API, or reads from a file. The iterator pattern real world example here shines because the same PaginatedEnumerable<T> works for any paginated data source without modification.
Notice we're using a Func<int, IReadOnlyList<T>> rather than requiring callers to implement a specific interface. This makes the API flexible. You can pass a lambda, a method group, or a closure that captures connection strings, HTTP clients, or whatever context the page-fetching logic needs. If you've worked with patterns like the facade pattern, you'll recognize this idea of hiding complex subsystem details behind a simple entry point.
Implementing the Enumerator: PaginatedEnumerator
The enumerator is where the real work happens. It tracks the current page number, the current position within that page, and handles the transition between pages transparently:
using System.Collections;
public sealed class PaginatedEnumerator<T>
: IEnumerator<T>
{
private readonly Func<int, IReadOnlyList<T>> _pageFetcher;
private IReadOnlyList<T> _currentPage;
private int _pageNumber;
private int _indexInPage;
private bool _finished;
public PaginatedEnumerator(
Func<int, IReadOnlyList<T>> pageFetcher)
{
_pageFetcher = pageFetcher;
_currentPage = Array.Empty<T>();
_pageNumber = 0;
_indexInPage = -1;
_finished = false;
}
public T Current
{
get
{
if (_indexInPage < 0
|| _indexInPage >= _currentPage.Count)
{
throw new InvalidOperationException(
"Enumerator is not positioned " +
"on a valid element.");
}
return _currentPage[_indexInPage];
}
}
object? IEnumerator.Current => Current;
public bool MoveNext()
{
if (_finished)
{
return false;
}
_indexInPage++;
if (_indexInPage < _currentPage.Count)
{
return true;
}
_currentPage = _pageFetcher(_pageNumber);
_pageNumber++;
_indexInPage = 0;
if (_currentPage.Count == 0)
{
_finished = true;
return false;
}
return true;
}
public void Reset()
{
_currentPage = Array.Empty<T>();
_pageNumber = 0;
_indexInPage = -1;
_finished = false;
}
public void Dispose()
{
_currentPage = Array.Empty<T>();
}
}
Let's walk through the key decisions in this implementation. The MoveNext() method first increments the index within the current page. If there are still items in the current page, it returns true immediately -- no new page fetch required. Only when the current page is exhausted does it call _pageFetcher to load the next batch. This is what gives us lazy evaluation: pages are only fetched when the consumer is ready for more data.
The end-of-data detection relies on a simple convention: an empty page signals that there are no more records. This is a common pattern in paginated APIs and database queries. The _finished flag prevents redundant fetches after the data source has been exhausted.
The Dispose() method clears the reference to the current page, allowing the garbage collector to reclaim that memory. In a more complex scenario -- say, if the page fetcher opened database connections or HTTP streams -- you could extend this to clean up those resources as well. Proper IDisposable implementation matters for the iterator pattern because foreach calls Dispose() automatically when the loop completes or when a break statement exits early.
The Reset() method rewinds the enumerator back to the beginning. While most modern .NET code doesn't call Reset() directly, implementing it correctly ensures the enumerator behaves predictably if re-enumeration is needed.
Adding Async Support with IAsyncEnumerable
The synchronous version works well for in-memory data sources and local databases, but real-world data access is almost always asynchronous. When your page fetcher calls an HTTP API or runs an async database query, you need an async iterator. C# makes this straightforward with IAsyncEnumerable<T> and await foreach.
Here's the async version:
using System.Runtime.CompilerServices;
public sealed class AsyncPaginatedEnumerable<T>
: IAsyncEnumerable<T>
{
private readonly Func<int, CancellationToken,
Task<IReadOnlyList<T>>> _pageFetcher;
public AsyncPaginatedEnumerable(
Func<int, CancellationToken,
Task<IReadOnlyList<T>>> pageFetcher)
{
_pageFetcher = pageFetcher
?? throw new ArgumentNullException(
nameof(pageFetcher));
}
public async IAsyncEnumerator<T> GetAsyncEnumerator(
[EnumeratorCancellation]
CancellationToken cancellationToken = default)
{
int pageNumber = 0;
while (!cancellationToken.IsCancellationRequested)
{
var page = await _pageFetcher(
pageNumber,
cancellationToken)
.ConfigureAwait(false);
if (page.Count == 0)
{
yield break;
}
foreach (var item in page)
{
yield return item;
}
pageNumber++;
}
cancellationToken.ThrowIfCancellationRequested();
}
}
The async version takes a different approach from the synchronous one. Instead of a separate enumerator class with manual state tracking, it uses C#'s yield return inside an async method. The compiler generates the state machine for us, which eliminates the boilerplate of tracking page numbers and indices manually.
The CancellationToken parameter flows through from await foreach to the page fetcher, allowing the consumer to cancel iteration at any point. The [EnumeratorCancellation] attribute tells the compiler to wire up the cancellation token from WithCancellation() calls automatically. This is critical for long-running data processing jobs where you need the ability to stop gracefully.
Notice the ConfigureAwait(false) call on the page fetch. This prevents the async continuation from capturing the synchronization context, which avoids potential deadlocks in UI applications and improves throughput in server-side code. If you're building patterns that coordinate multiple async operations, the observer pattern offers a complementary approach for broadcasting async events.
Consuming the async iterator is clean:
var customers = new AsyncPaginatedEnumerable<Customer>(
async (page, token) =>
await api.GetCustomersAsync(page, 100, token));
await foreach (var customer in customers)
{
await ProcessCustomerAsync(customer);
}
The caller writes a plain await foreach loop. No page tracking. No manual async state management. Just iterate.
Integration with the Repository Pattern
The iterator pattern becomes especially powerful when integrated into a data access layer. A repository can return IEnumerable<T> or IAsyncEnumerable<T>, and the service layer consumes the data without knowing that pagination is happening underneath.
Here's a repository that uses our paginated iterators:
public interface ICustomerRepository
{
IEnumerable<Customer> GetAll();
IAsyncEnumerable<Customer> GetAllAsync(
CancellationToken cancellationToken = default);
IAsyncEnumerable<Customer> GetByRegionAsync(
string region,
CancellationToken cancellationToken = default);
}
public sealed class CustomerRepository
: ICustomerRepository
{
private readonly ICustomerDataSource _dataSource;
private const int PageSize = 100;
public CustomerRepository(
ICustomerDataSource dataSource)
{
_dataSource = dataSource;
}
public IEnumerable<Customer> GetAll()
{
return new PaginatedEnumerable<Customer>(
pageNumber => _dataSource.FetchPage(
pageNumber, PageSize));
}
public IAsyncEnumerable<Customer> GetAllAsync(
CancellationToken cancellationToken = default)
{
return new AsyncPaginatedEnumerable<Customer>(
(pageNumber, token) =>
_dataSource.FetchPageAsync(
pageNumber, PageSize, token));
}
public IAsyncEnumerable<Customer> GetByRegionAsync(
string region,
CancellationToken cancellationToken = default)
{
return new AsyncPaginatedEnumerable<Customer>(
(pageNumber, token) =>
_dataSource.FetchByRegionAsync(
region, pageNumber, PageSize, token));
}
}
The repository methods are remarkably simple. Each one creates a paginated enumerable with the appropriate page-fetching delegate and returns it. The caller gets back an IEnumerable<Customer> or IAsyncEnumerable<Customer> -- standard .NET interfaces that work with foreach, LINQ, and every other enumeration mechanism in the framework.
Here's a service that consumes the repository:
public sealed class CustomerReportService
{
private readonly ICustomerRepository _repository;
public CustomerReportService(
ICustomerRepository repository)
{
_repository = repository;
}
public async Task<ReportSummary> GenerateRegionReportAsync(
string region,
CancellationToken cancellationToken = default)
{
int count = 0;
decimal totalRevenue = 0m;
await foreach (var customer in _repository
.GetByRegionAsync(region, cancellationToken))
{
count++;
totalRevenue += customer.TotalRevenue;
}
return new ReportSummary(region, count, totalRevenue);
}
public IEnumerable<Customer> GetHighValueCustomers()
{
return _repository.GetAll()
.Where(c => c.TotalRevenue > 10_000m)
.OrderByDescending(c => c.TotalRevenue);
}
}
public record ReportSummary(
string Region,
int CustomerCount,
decimal TotalRevenue);
public record Customer(
string Id,
string Name,
string Region,
decimal TotalRevenue);
The GetHighValueCustomers() method shows LINQ integration working transparently over paginated data. The .Where() and .OrderByDescending() calls operate on each item as it flows through the iterator -- no intermediate list of all customers is ever built in memory. This is the real payoff of the iterator pattern: the consumer code is clean, composable, and memory-efficient without any awareness of the underlying pagination.
This layered approach follows the same inversion of control principles that drive modern .NET architecture. The repository owns the "how" of data access. The service owns the "what" of business logic. The iterator pattern is the bridge between them.
Wiring Up Dependency Injection
Registering these components with IServiceCollection completes the integration:
using Microsoft.Extensions.DependencyInjection;
public static class CustomerServiceRegistration
{
public static IServiceCollection AddCustomerServices(
this IServiceCollection services)
{
services.AddSingleton<
ICustomerDataSource, DatabaseCustomerDataSource>();
services.AddScoped<
ICustomerRepository, CustomerRepository>();
services.AddTransient<CustomerReportService>();
return services;
}
}
In your Program.cs:
builder.Services.AddCustomerServices();
The DI container resolves the ICustomerDataSource dependency into the CustomerRepository, and the CustomerRepository into CustomerReportService. If you swap from a database to an API-backed data source, you change one registration line. The repository, service, and iterator code remain untouched.
Testing the Implementation
Testing paginated iterators is straightforward because the Func<int, IReadOnlyList<T>> delegate is easy to stub. Here are xUnit tests that cover the key scenarios:
using Xunit;
public sealed class PaginatedEnumerableTests
{
[Fact]
public void Enumerate_MultiplePages_ReturnsAllItems()
{
var pages = new Dictionary<int, IReadOnlyList<int>>
{
[0] = new[] { 1, 2, 3 },
[1] = new[] { 4, 5, 6 },
[2] = Array.Empty<int>()
};
var enumerable = new PaginatedEnumerable<int>(
page => pages.GetValueOrDefault(
page, Array.Empty<int>()));
var result = enumerable.ToList();
Assert.Equal(
new[] { 1, 2, 3, 4, 5, 6 },
result);
}
[Fact]
public void Enumerate_EmptyFirstPage_ReturnsEmpty()
{
var enumerable = new PaginatedEnumerable<int>(
_ => Array.Empty<int>());
var result = enumerable.ToList();
Assert.Empty(result);
}
[Fact]
public void Enumerate_SinglePage_ReturnsPageItems()
{
var pages = new Dictionary<int, IReadOnlyList<int>>
{
[0] = new[] { 10, 20, 30 },
[1] = Array.Empty<int>()
};
var enumerable = new PaginatedEnumerable<int>(
page => pages.GetValueOrDefault(
page, Array.Empty<int>()));
var result = enumerable.ToList();
Assert.Equal(new[] { 10, 20, 30 }, result);
}
[Fact]
public void Enumerate_EarlyBreak_StopsFetching()
{
int maxPageFetched = -1;
var enumerable = new PaginatedEnumerable<int>(
page =>
{
maxPageFetched = page;
return new[] { page * 10 };
});
var result = new List<int>();
foreach (var item in enumerable)
{
result.Add(item);
if (result.Count >= 2)
{
break;
}
}
Assert.Equal(new[] { 0, 10 }, result);
Assert.True(maxPageFetched <= 2);
}
[Fact]
public void Enumerate_WithLinq_FiltersCorrectly()
{
var pages = new Dictionary<int, IReadOnlyList<int>>
{
[0] = new[] { 1, 2, 3, 4 },
[1] = new[] { 5, 6, 7, 8 },
[2] = Array.Empty<int>()
};
var enumerable = new PaginatedEnumerable<int>(
page => pages.GetValueOrDefault(
page, Array.Empty<int>()));
var evens = enumerable
.Where(x => x % 2 == 0)
.ToList();
Assert.Equal(new[] { 2, 4, 6, 8 }, evens);
}
}
And the async tests:
using Xunit;
public sealed class AsyncPaginatedEnumerableTests
{
[Fact]
public async Task EnumerateAsync_MultiplePages_ReturnsAllItems()
{
var pages = new Dictionary<int, IReadOnlyList<int>>
{
[0] = new[] { 1, 2, 3 },
[1] = new[] { 4, 5 },
[2] = Array.Empty<int>()
};
var enumerable = new AsyncPaginatedEnumerable<int>(
(page, _) => Task.FromResult(
pages.GetValueOrDefault(
page,
(IReadOnlyList<int>)Array.Empty<int>())));
var result = new List<int>();
await foreach (var item in enumerable)
{
result.Add(item);
}
Assert.Equal(
new[] { 1, 2, 3, 4, 5 },
result);
}
[Fact]
public async Task EnumerateAsync_EmptySource_ReturnsEmpty()
{
var enumerable = new AsyncPaginatedEnumerable<int>(
(_, _) => Task.FromResult(
(IReadOnlyList<int>)Array.Empty<int>()));
var result = new List<int>();
await foreach (var item in enumerable)
{
result.Add(item);
}
Assert.Empty(result);
}
[Fact]
public async Task EnumerateAsync_Cancellation_StopsIteration()
{
var cts = new CancellationTokenSource();
int pagesRequested = 0;
var enumerable = new AsyncPaginatedEnumerable<int>(
(page, token) =>
{
pagesRequested++;
token.ThrowIfCancellationRequested();
return Task.FromResult(
(IReadOnlyList<int>)new[] { page });
});
var result = new List<int>();
await Assert.ThrowsAsync<OperationCanceledException>(
async () =>
{
await foreach (var item in enumerable
.WithCancellation(cts.Token))
{
result.Add(item);
if (result.Count >= 2)
{
cts.Cancel();
}
}
});
Assert.True(result.Count >= 2);
}
[Fact]
public async Task EnumerateAsync_SinglePage_ReturnsItems()
{
var pages = new Dictionary<int, IReadOnlyList<string>>
{
[0] = new[] { "alpha", "beta" },
[1] = Array.Empty<string>()
};
var enumerable = new AsyncPaginatedEnumerable<string>(
(page, _) => Task.FromResult(
pages.GetValueOrDefault(
page,
(IReadOnlyList<string>)
Array.Empty<string>())));
var result = new List<string>();
await foreach (var item in enumerable)
{
result.Add(item);
}
Assert.Equal(
new[] { "alpha", "beta" },
result);
}
}
These tests cover the scenarios that matter most for a paginated iterator. The multi-page test verifies that items from different pages are stitched together seamlessly. The empty-source test confirms the iterator handles zero records gracefully. The early-break test proves that lazy evaluation works -- pages beyond what the consumer needs are never fetched. The LINQ test confirms that standard query operators compose correctly over the paginated enumerable.
The cancellation test for the async version is especially important. It verifies that the CancellationToken flows through to the page fetcher and that iteration stops promptly when cancellation is requested. In production, this is what lets a batch job shut down cleanly when the application is stopping.
When to Use the Iterator Pattern for Data Access
The paginated iterator approach works best when your data set is large enough that loading it all at once is impractical, but the processing logic needs to treat items individually. Database record exports, API data synchronization, log file processing, and ETL pipelines are all strong candidates.
The pattern becomes less useful when you need random access to items by index, or when the entire collection must be in memory for operations like sorting or grouping. In those cases, you're better off loading the data into a list and working with it directly. The key question is whether your consumer processes items sequentially -- if it does, the iterator pattern gives you memory efficiency and clean abstractions for free.
One thing to watch for is multiple enumeration. Because PaginatedEnumerable<T> fetches data lazily, enumerating it twice means fetching all the pages twice. If you need to iterate multiple times, call .ToList() to materialize the results. This is the same trade-off that LINQ-to-Objects makes with deferred execution -- and the same solution applies.
The iterator pattern also composes well with other design patterns. You might use the adapter pattern to normalize different paginated APIs into a common page-fetching delegate. Or use the proxy pattern to add caching, logging, or retry logic around the page fetcher without modifying the iterator itself. The bridge pattern can separate the iteration abstraction from the data source implementation when you need to support multiple back-ends.
Frequently Asked Questions
How does the iterator pattern differ from just using yield return?
The yield return keyword in C# is the compiler's built-in support for the iterator pattern. When you write a method with yield return, the compiler generates a state machine that implements IEnumerator<T> for you. The custom PaginatedEnumerator<T> in this article gives you explicit control over the state -- page tracking, resource cleanup, and reset behavior -- that yield return handles implicitly. For the async version, we use yield return because the compiler-generated state machine handles the async complexity cleanly.
Can I use LINQ with paginated iterators?
Yes. Because PaginatedEnumerable<T> implements IEnumerable<T>, every LINQ extension method works out of the box. .Where(), .Select(), .Take(), .Skip(), and .Aggregate() all compose over the lazy iterator without materializing the full dataset. Be cautious with .OrderBy() and .GroupBy() since they need to consume the entire sequence before producing results, which defeats the lazy evaluation benefit.
What happens if the page fetcher throws an exception?
The exception propagates out of MoveNext() (or the await foreach loop for the async version) to the consumer's foreach block. The enumerator's Dispose() method is still called because foreach uses a try/finally internally. If you need retry logic, wrap the page-fetching delegate itself rather than adding retry logic to the enumerator -- this keeps the iterator pattern clean and the retry policy configurable.
How do I handle cursor-based pagination instead of page numbers?
Replace the Func<int, IReadOnlyList<T>> delegate with one that returns both the data and a continuation token. For example, use Func<string?, (IReadOnlyList<T> Items, string? NextCursor)> where a null cursor means no more pages. The enumerator tracks the cursor instead of a page number. The consumer-facing API stays identical -- foreach works the same way regardless of how the underlying pagination is implemented.
Is the iterator pattern thread-safe?
A single enumerator instance is not thread-safe and should not be shared across threads. Each foreach loop gets its own enumerator instance via GetEnumerator(), so concurrent foreach loops over the same PaginatedEnumerable<T> work fine -- they each maintain independent state. If the page-fetching delegate accesses shared resources, that delegate needs its own synchronization.
Should I use IEnumerable or IAsyncEnumerable for database access?
Use IAsyncEnumerable<T> for any I/O-bound data source. Database queries, HTTP API calls, and file reads should all use the async version to avoid blocking threads. Reserve the synchronous IEnumerable<T> version for in-memory data sources, CPU-bound transformations, or situations where the page data is already cached. In a web application, blocking a thread pool thread with synchronous I/O reduces your server's ability to handle concurrent requests.
How does this compare to EF Core's streaming queries?
Entity Framework Core can stream query results using AsAsyncEnumerable(), which internally uses a similar lazy-fetching strategy. The custom AsyncPaginatedEnumerable<T> in this article is useful when you're working with data sources that EF Core doesn't manage -- REST APIs, legacy databases with custom pagination, or third-party services. It's also helpful when you want explicit control over page boundaries for logging, metrics, or progress reporting.
Wrapping Up This Iterator Pattern Real-World Example
This implementation shows the iterator pattern solving a genuine production problem -- paginated data access that's transparent to consumers. We started with a pagination loop tangled into business logic and ended with a clean separation: PaginatedEnumerable<T> handles page fetching, CustomerRepository provides the data access API, and CustomerReportService consumes records with a simple foreach loop.
The synchronous and async versions cover the two most common data access scenarios. LINQ integration comes for free because we implement standard .NET interfaces. The xUnit tests verify that multi-page iteration, empty datasets, early termination, and cancellation all work correctly.
Take the PaginatedEnumerable<T> and AsyncPaginatedEnumerable<T> classes, replace the simulated data source with your actual database or API client, and you have a reusable, testable data access layer that scales to millions of records without scaling memory consumption. The iterator pattern keeps your pagination logic centralized, your consumer code clean, and your application responsive.