Skip to main content

HiLo Algorithm

How the HiLo algorithm works in RavenDB​

Generating unique IDs efficiently:​

The HiLo algorithm is efficient because the client can generate unique document IDs locally,
without checking with the server or the cluster every time a new document is stored.

  • The client gets a range of numbers from the server:
    When the client needs IDs for a collection, it asks the server for a range of numbers reserved for that collection.
    Each time a session stores a new document whose Id is null, the client uses the next number from that range.

    For example, the first range reserved for the Orders collection can be 1-32.
    The next range reserved for the same collection can be 33-64, and so on.

  • The default HiLo document ID includes the collection prefix and server node tag:
    When the client generates a full document ID, the number from the HiLo range is combined with the collection prefix and the server node tag, such as orders/54-A.

  • The reserved range is the main uniqueness mechanism:
    During normal operation, different clients receive different ranges for the same collection.
    The node tag provides an additional distinction across the cluster. For example, if two nodes ever allocate the same numeric value, the generated IDs can still differ: orders/54-B and orders/54-C.

Thus, with only occasional trips to the server, the client can automatically assign unique document IDs that include the collection prefix, the next number from the reserved range, and the server node tag.

Using HiLo documents:​

HiLo documents are used by the server to allocate the next range of numbers.
To let multiple clients generate IDs independently without producing duplicates,
the server tracks the allocated HiLo range for each collection.

This is handled by Raven/Hilo/<collection> documents, stored in the @hilo collection in the database.
These documents are created and modified by the server and have a simple structure:

{
    "Max": 32,
    "@metadata": {
        "@collection": "@hilo"
    }
}

The Max property stores the highest HiLo number currently recorded for the collection.
It represents the upper end of the last allocated range, unless unused values were returned to the server.
For example, if the HiLo document contains "Max": 32, the next range allocated for this collection can start at 33.

The process works as follows:

  1. The client asks the server for a range of numbers that it can use to generate document IDs for the collection.
    The initial range size is 32, but the range size can change dynamically based on how frequently the client requests new HiLo ranges.
  2. The server reads the Max value from the collection's HiLo document.
  3. The server allocates the next range and updates Max to the upper end of that range.
    For example, if the current Max is 32, the server can allocate the range 33-64 and update Max to 64.
  4. The client receives the range boundaries from the server and keeps them locally.
  5. The client uses the next number from the local range each time it needs to generate a new document ID.
  6. When the local range is exhausted, the client asks the server for another range.

Returning HiLo ranges​

When the document store is disposed, the client sends the server two values for each HiLo range it holds:

  • The last number it used from the range.
  • The end of the range that was allocated to it.

If the server-side Max value is equal to the upper end of the client's range, and the last number used by the client is smaller than or equal to that Max value, the server updates Max to the last number used by the client.

var store = new DocumentStore();

using (var session = store.OpenSession())
{
    // Storing the first entity causes the client to receive the initial HiLo range (1-32)
    session.Store(new Employee
    {
        FirstName = "John",
        LastName = "Doe"
    });

    session.SaveChanges();
    // The document ID will be: employees/1-A
}

// Dispose the store when the application shuts down.
// This also returns unused HiLo numbers from the current range.    
store.Dispose();

store.Dispose() is used in this example only to demonstrate that unused HiLo numbers can be returned to the server.
In normal use, the store should be disposed only when the application shuts down.

After the code above runs, the HiLo document for the Employees collection will have "Max": 1.
This is because the client received the range 1-32 but used only 1 before the store was disposed.

The next time a client asks the server for a HiLo range for the Employees collection,
the server will allocate the range 2-33 in this example.

var newStore = new DocumentStore();
    
using (var session = newStore.OpenSession())
{
    // Storing an entity after disposing the store in the previous example
    // causes the client to receive the next HiLo range (2-33)
    session.Store(new Employee
    {
        FirstName = "Dave",
        LastName = "Brown"
    });

    session.SaveChanges();
    // The document ID will be: employees/2-A
}

Identity parts separator​

  • By default, generated document IDs use / as the separator between the collection prefix and the numeric part of the ID. For example: employees/1-A.

  • You can customize this separator by setting the IdentityPartsSeparator convention.
    The separator can be set to any character except |.

Manual HiLo ID generation​

  • Automatic generation:
    When the session stores a new document whose Id property is null,
    RavenDB's default HiLo generator creates the document ID automatically.
    The generated ID includes the collection prefix, the next number from the HiLo range, and the server node tag,
    for example products/1-A.

  • Manual generation:
    You can ask the default HiLo generator for the next ID value without storing a document first.
    You can retrieve either the next number from the HiLo range or the next full document ID,
    and then use that value when storing a document, as explained below:

Get next ID - number only​

You can use the HiLo algorithm to get the next number from the HiLo range and include it in a custom document ID.

  • Manually getting the next HiLo number returns only the next number from the HiLo range.
    It does not include the collection prefix or the server node tag.

  • Therefore, when you use this number to create your own document IDs,
    you are responsible for ensuring that those IDs are unique within the database.

Syntax:​

Each overload returns the next available number from the HiLo range for the specified collection.
If the client does not already have an available HiLo number for that collection, it requests a new range from the server.
The returned number can then be used when creating a custom document ID.

Task<long> GenerateNextIdForAsync(string database, object entity);

Task<long> GenerateNextIdForAsync(string database, Type type);

Task<long> GenerateNextIdForAsync(string database, string collectionName);

ParameterTypeDescription
databasestringThe database for which to get the next HiLo number.
null uses the default database configured in the document store.
collectionNamestringThe collection for which to get the next HiLo number.
entityobjectAn entity instance from which the collection name is resolved.
typeTypeAn entity type from which the collection name is resolved.
Return valueTypeDescription
nextIdlongThe next available number from the collection's HiLo range.

Example:​

The following example shows how to get the next number from the HiLo range.
The returned value is only the numeric part of the ID. It does not include the collection prefix or the server node tag.

The returned number is then used to create a custom document ID before storing the document.

Calling GenerateNextIdForAsync uses the client-side HiLo generator,
so the client can reuse its local HiLo range and avoid a server request for every ID.

using (var session = store.OpenSession())
{
    // Use any overload to get the next HiLo number:
    // (Note how the number increases with each call)
    // ==============================================
    
    var nextId = await store.HiLoIdGenerator.GenerateNextIdForAsync(null, "Products");
    // nextId = 1
    
    nextId = await store.HiLoIdGenerator.GenerateNextIdForAsync(null, new Product());
    // nextId = 2
    
    nextId = await store.HiLoIdGenerator.GenerateNextIdForAsync(null, typeof(Product));
    // nextId = 3

    // Use the returned number in a custom document ID
    // ===============================================
    
    var product = new Product
    {
        Id = "MyCustomId/" + nextId
    };
    
    // Store the new document
    // The document ID will be: "MyCustomId/3"
    session.Store(product);
    session.SaveChanges();
}

Unique IDs across the cluster​

When you call GenerateNextIdForAsync, RavenDB returns only the next number from the HiLo range.
It does not include the collection prefix or the server node tag.

If you use that number to build your own document IDs,
make sure your custom ID format keeps IDs unique across the database.

To use RavenDB's default cluster-safe HiLo ID format, use the default HiLo generator,
which creates full IDs such as products/1-A.

You can also use the cluster-wide identities generator, which guarantees uniqueness across the cluster. This is more costly than the default HiLo generator because each ID requires a server request and a Raft consensus check.

Get next ID - full document ID​

You can request the next full document ID from the default HiLo generator without storing a document first.

Syntax:​

Each overload returns the next full document ID for the specified collection.
If the client does not already have an available HiLo number for that collection, it requests a new range from the server. The returned ID can then be used when storing a new document.

// This overload is deprecated and will be removed in RavenDB 8.0.
// Use the `collectionName` or `Type` overload instead.         
Task<string> GenerateDocumentIdAsync(string database, object entity);

Task<string> GenerateDocumentIdAsync(string database, Type type);

Task<string> GenerateDocumentIdAsync(string database, string collectionName);

ParameterTypeDescription
databasestringThe database for which to get the document ID.
null uses the default database configured in the document store.
collectionNamestringThe collection for which to get the document ID.
entityobjectAn entity instance from which the collection name is resolved.
(This overload is deprecated; see the note above.)
typeTypeAn entity type from which the collection name is resolved.
Return valueTypeDescription
nextFullIdstringThe next full document ID, including the collection prefix, number, and server node tag, (e.g. products/4-A).

Example:​

The latest HiLo number generated in the previous example was 3.
Therefore, when running the following example immediately after,
the consecutive numbers 4 and 5 are used to create the full document IDs products/4-A and products/5-A.

using (var session = store.OpenSession())
{
    // Use either overload to get the next full document ID:
    // (Note how the number increases with each call)
    // ====================================================

    var nextFullId = await store.HiLoIdGenerator.GenerateDocumentIdAsync(null, "Products");
    // nextFullId = "products/4-A"

    nextFullId = await store.HiLoIdGenerator.GenerateDocumentIdAsync(null, typeof(Product));
    // nextFullId = "products/5-A"

    // Use the returned full document ID when storing the document
    // ==========================================================
                        
    var product = new Product
    {
        Id = nextFullId
    };

    session.Store(product);
    session.SaveChanges();
    // The document ID will be: "products/5-A"
}

Overriding the HiLo algorithm​

  • RavenDB's default HiLo generator is exposed through the HiLoIdGenerator property on the DocumentStore.

  • To override the default ID generation behavior, set the AsyncDocumentIdGenerator convention to your own implementation.

  • Once you configure a custom AsyncDocumentIdGenerator:

    • Your custom ID generation logic is used whenever the session stores a document whose Id property is null.
    • The store's default HiLoIdGenerator is no longer available. Calling GenerateNextIdForAsync or GenerateDocumentIdAsync through store.HiLoIdGenerator will throw an exception.

In this article