Dynamic Index Fields

Name: RavenDB
Author: RavenDB

In RavenDB, different documents can have different shapes.
Documents are schemaless - new fields can be added or removed as needed.
For such dynamic data, you can define indexes with dynamic-index-fields.
Dynamic-index-fields are created at indexing time, with names and values derived from document content,
rather than being predefined as fixed index fields in the index definition.
This allows you to query the index by fields that are not known at index definition or deployment time,
which is very useful when working with highly dynamic systems.
Any value type can be indexed, string, number, date, etc.
An index definition can contain both dynamic-index-fields and regular-index-fields.
In this article:

Indexing field KEYS

Example - index every field under an object

The following example allows you to:

Index any field that is under the same object from the document.
After index is deployed, any new field added to this object will be indexed as well.

The document:

public class Product
{
    public string Id { get; set; }
    
    // The KEYS under the Attributes object will be dynamically indexed 
    // Fields added to this object after index creation time will also get indexed
    public Dictionary<string, object> Attributes { get; set; }
}

// Sample document content
{
    "Attributes": {
        "Color": "Red",
        "Size": 42
    }
}

The index:

The following index will index any field under the Attributes object from the document,
a dynamic-index-field will be created for each such field.
New fields added to the object after index creation time will be dynamically indexed as well.

The actual dynamic-index-field name on which you can query will be the attribute field key.
E.g., Keys Color & Size will become the actual dynamic-index-fields.

LINQ_index
JavaScript_index

public class Products_ByAttributeKey : AbstractIndexCreationTask<Product>
{
    public Products_ByAttributeKey()
    {
        Map = products => from p in products
            select new
            {
                // Call 'CreateField' to generate dynamic-index-fields 
                // from the Attributes object keys.
    
                // Using '_' is just a convention. 
                // Any other string can be used instead of '_'
                
                // The actual field name will be 'item.Key'
                // The actual field terms will be derived from 'item.Value'
                _ = p.Attributes.Select(item => CreateField(item.Key, item.Value))
            };
    }
}

public class Products_ByAttributeKey_JS : AbstractJavaScriptIndexCreationTask
{
    public Products_ByAttributeKey_JS()
    {
        Maps = new HashSet<string>
        {
            @"map('Products', function (p) {
                return {
                    // The field name will be the key
                    // The field terms will be derived from the corresponding value
                    _: Object.keys(p.Attributes)
                           .map(key => createField(key, p.Attributes[key]))
                };
            })"
        };
    }
}

The query:

You can now query the generated dynamic-index fields.
The _ property is Not queryable but used only in the index definition syntax.

To get all documents with a specific Size use:
- DocumentQuery
- RQL
IList<Product> matchingDocuments = session .Advanced .DocumentQuery<Product, Products_ByAttributeKey>() // 'Size' is a dynamic-index-field that was indexed from the Attributes object .WhereEquals("Size", 42) .ToList();
// 'Size' is a dynamic-index-field that was indexed from the Attributes object from index 'Products/ByAttributeKey' where Size = 42
Dynamic index fields are queried with DocumentQuery or RQL because the field names (e.g. Size, Color)
are generated at indexing time and are not properties on the Product class.
A strongly typed LINQ query such as Query<Product>().Where(p => p.Size == 42) would not compile.
Use the string-based WhereEquals("Size", 42) or RQL where Size = 42 instead.

Example - index every field

The following example allows you to:

Define an index on a collection without needing any common structure between the indexed documents.
After the index is deployed, any new field added to any document in the collection will be indexed as well.

Use this approach with care.
Indexing every field can increase indexing time, memory usage, and disk space,
especially for large or frequently updated documents.

The document:

public class Product
{
    public string Id { get; set; }

    // All KEYS in the document will be dynamically indexed 
    // Fields added to the document after index creation time will also get indexed
    public string FirstName { get; set; }
    public string LastName { get; set; }
    public string Title { get; set; }
    // ...
}

// Sample document content
    {
        "FirstName": "John",
        "LastName": "Doe",
        "Title": "Engineer",
        // ...
}

The index:

The following index will index any field from the document,
a dynamic-index-field will be created for each field.
New fields added to the document after index creation time will be dynamically indexed as well.

The actual dynamic-index-field name on which you can query will be the field key.
E.g., Keys FirstName & LastName will become the actual dynamic-index-fields.

The example uses a JavaScript index because it enumerates all top-level document fields at indexing time with Object.keys(...). LINQ indexes typically create dynamic fields from a known enumerable structure, such as a dictionary or collection.
- JavaScript_index
public class Products_ByAnyField_JS : AbstractJavaScriptIndexCreationTask { public Products_ByAnyField_JS() { // This will index EVERY FIELD under the top level of the document Maps = new HashSet<string> { @"map('Products', function (p) { return { _: Object.keys(p).map(key => createField(key, p[key])) }; })" }; } }

The query:

To get all documents with a specific LastName use:

DocumentQuery
RQL

IList<Product> matchingDocuments = session
    .Advanced
    .DocumentQuery<Product, Products_ByAnyField_JS>()
     // 'LastName' is a dynamic-index-field that was indexed from the document
    .WhereEquals("LastName", "Doe")
    .ToList();

// 'LastName' is a dynamic-index-field that was indexed from the document
from index 'Products/ByAnyField/JS' where LastName = "Doe"

Indexing field VALUES

Example - basic

The following example shows:

Only the basic concept of creating a dynamic-index-field from the value of a document field.
Documents can then be queried based on those indexed values.
For a more practical usage see the Example below.

The document:

public class Product
{
    public string Id { get; set; }
    
    // The VALUE of ProductType will be dynamically indexed
    public string ProductType { get; set; }
    public int PricePerUnit { get; set; }
}

// Sample document content
{
    "ProductType": "Electronics",
    "PricePerUnit": 23
}

The index:

The following index will index the value of document field 'ProductType'.

This value will be the dynamic-index-field name on which you can query.
E.g., Field value Electronics will be the dynamic-index-field.

LINQ_index
JavaScript_index

public class Products_ByProductType : AbstractIndexCreationTask<Product>
{
    public Products_ByProductType()
    {
        Map = products => from p in products
            select new
            {
                // Call 'CreateField' to generate the dynamic-index-fields
                // The field name will be the value of document field 'ProductType'
                // The field terms will be derived from document field 'PricePerUnit'
                _ = CreateField(p.ProductType, p.PricePerUnit)
            };
    }
}

public class Products_ByProductType_JS : AbstractJavaScriptIndexCreationTask
{
    public Products_ByProductType_JS()
    {
        Maps = new HashSet<string>
        {
            @"map('Products', function (p) {
                return {
                    // The field name will be the value of document field 'ProductType'
                    // The field terms will be derived from document field 'PricePerUnit'
                    _: createField(p.ProductType, p.PricePerUnit)
                };
            })"
        };
    }
}

The query:

To get all documents of some product type having a specific price per unit use:

DocumentQuery
RQL

IList<Product> matchingDocuments = session
    .Advanced
    .DocumentQuery<Product, Products_ByProductType>()
     // 'Electronics' is the dynamic-index-field that was indexed
     // from document field 'ProductType'
    .WhereEquals("Electronics", 23)
    .ToList();

// 'Electronics' is the dynamic-index-field that was indexed
// from document field 'ProductType'
from index 'Products/ByProductType' where Electronics = 23

Example - list

The following example allows you to:

Index values from items in a list
After index is deployed, any item added to this list in the document will be dynamically indexed as well.

The document:

public class Product
{
    public string Id { get; set; }
    public string Name { get; set; }
    
    // For each element in this list, 
    // the VALUE of property 'PropName' will be dynamically indexed.
    // e.g. Color, Width, Length (in ex. below) will become dynamic-index-fields.
    public List<Attribute> Attributes { get; set; }
}

public class Attribute
{
    public string PropName { get; set; }
    public string PropValue { get; set; }
}

// Sample document content
{
    "Name": "SomeName",
    "Attributes": [
       {  
           "PropName": "Color",
           "PropValue": "Blue"
       },
       {
           "PropName": "Width",
           "PropValue": "10"
       },
       {
           "PropName": "Length",
           "PropValue": "20"
       },
       ...
   ] 
}

The index:

The following index will create a dynamic-index-field per item in the document's Attributes list.
New items added to the Attributes list after index creation time will be dynamically indexed as well.

The actual dynamic-index-field name on which you can query will be the item's PropName value.
E.g., 'PropName' value Width will be a dynamic-index-field.

LINQ_index
JavaScript_index

public class Attributes_ByName : AbstractIndexCreationTask<Product>
{
    public Attributes_ByName()
    {
        Map = products => from a in products
            select new
            {
                // Define the dynamic-index-fields by calling 'CreateField'
                // A dynamic-index-field will be generated for each item in 'Attributes'
                
                // For each item, the field name will be the value of field 'PropName'
                // The field terms will be derived from field 'PropValue'
                _ = a.Attributes.Select(item => 
                        CreateField(item.PropName, item.PropValue)),
                
                // A regular index field can be defined as well:
                Name = a.Name
            };
    }
}

public class Attributes_ByName_JS : AbstractJavaScriptIndexCreationTask
{
    public Attributes_ByName_JS()
    {
        Maps = new HashSet<string>
        {
            @"map('Products', function (p) {
                return {
                    // For each item, 
                    // the field name will be the value of field 'PropName'
                    // The field terms will be derived from field 'PropValue'
                    _: p.Attributes.map(item => createField(
                           item.PropName, item.PropValue)),

                    // A regular index field can be defined as well:
                    Name: p.Name
                };
            })"
        };
    }
}

The query:

To get all documents matching a specific attribute property use:

DocumentQuery
RQL

IList<Product> matchingDocuments = session
    .Advanced
    .DocumentQuery<Product, Attributes_ByName>()
     // 'Width' is a dynamic-index-field that was indexed from the Attributes list
    .WhereEquals("Width", 10)
    .ToList();

// 'Width' is a dynamic-index-field that was indexed from the Attributes list
from index 'Attributes/ByName' where Width = 10

Indexing dynamic fields for full-text search

Dynamic fields can be indexed for Full-text search by setting FieldIndexing.Search in CreateFieldOptions.

At indexing time:
At indexing time, RavenDB analyzes the dynamic field value and creates the indexed terms.
The analyzer used at indexing time is resolved from the index definition.
You can configure an analyzer for a specific generated field name, or configure AllFields as a fallback.
See Configuring analyzers for dynamic fields below.
- If the generated dynamic field name has an explicit analyzer configuration, that analyzer is used.
- Otherwise, if AllFields has an analyzer configuration, RavenDB uses it as a fallback.
- If no analyzer is configured for the field, RavenDB uses the Default search analyzer.
At querying time:
For full-text search to return the expected results, the analyzer used at indexing time and the analyzer used for the query term should produce compatible terms.

When a search() query targets a dynamic field that does not have an explicit analyzer configuration for its generated field name, RavenDB uses the Default analyzer to analyze the query term by default.

Set the Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery
configuration key to true to make RavenDB use the Default search analyzer as the fallback analyzer for the query term instead.

This configuration can be set at different scopes: server-wide, per database, or per index.
The example below sets it at the index scope, in the index definition.

For a summary of how RavenDB resolves the analyzer used for the query term,
see Which analyzer is used for the query term below.

The following example indexes localized product descriptions as dynamic fields and queries one of those generated fields using full-text search:

The document:

public class Product
{
    public string Id { get; set; }

    public Dictionary<string, string> Descriptions { get; set; }
}

// Sample document content
{
    "Descriptions": {
        "English": "North wind jacket",
        "French": "Veste vent du nord"
    }
}

The index:

The following index creates one dynamic field for each entry in the Descriptions dictionary.
For example, the key English creates the dynamic field Description_English.

Each generated dynamic field is indexed for full-text search.

LINQ_index
JavaScript_index

public class Products_ByLocalizedDescription : AbstractIndexCreationTask<Product>
{
    private const string UseSearchAnalyzerForDynamicFields =
        "Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery";

    public Products_ByLocalizedDescription()
    {
        Map = products => from product in products
            select new
            {
                _ = product.Descriptions.Select(x =>
                    CreateField(
                        "Description_" + x.Key,
                        x.Value,
                        new CreateFieldOptions
                        {
                            // Index each generated dynamic field for FTS
                            Indexing = FieldIndexing.Search,
                            Storage = FieldStorage.No
                        }))
            };

        StoreAllFields(FieldStorage.No);

        // Set this configuration key to true so that search() queries on dynamic fields
        // will use the 'Default Search Analyzer' when the generated field name
        // has no explicit analyzer configuration.
        Configuration[UseSearchAnalyzerForDynamicFields] = "true";
    }
}

public class Products_ByLocalizedDescription_JS : AbstractJavaScriptIndexCreationTask
{
    private const string UseSearchAnalyzerForDynamicFields =
        "Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery";
    
    public Products_ByLocalizedDescription_JS()
    {
        Maps = new HashSet<string>
        {
            @"map('Products', function (product) {
                return {
                    _: Object.keys(product.Descriptions)
                        .map(function (key) {
                            return createField(
                              'Description_' + key,
                              product.Descriptions[key],
                              {
                                 // Index each generated dynamic field for FTS
                                 indexing: 'Search',
                                 storage: 'No'
                              });
                    })
                };
            })"
        }; 

        Fields[Constants.Documents.Indexing.Fields.AllFields] = new IndexFieldOptions
        {
            Storage = FieldStorage.No
        };

        // Set this configuration key to true so that search() queries on dynamic fields
        // will use the 'Default Search Analyzer' when the generated field name
        // has no explicit analyzer configuration.
        Configuration[UseSearchAnalyzerForDynamicFields] = "true";
    }
}        

Full-text search query:

Query the generated dynamic field by its field name.

In this example, the query targets the generated field Description_English.

Since Indexing.Querying.UseSearchAnalyzerForDynamicFieldsIfNotSetExplicitlyInSearchQuery is set to true,
the searched term "north wind" is analyzed with the Default search analyzer.
- DocumentQuery
- RQL
List<Product> results = session.Advanced .DocumentQuery<Product, Products_ByLocalizedDescription>() .Search("Description_English", "north wind") .ToList();
from index "Products/ByLocalizedDescription" where search(Description_English, "north wind")

Configuring analyzers for dynamic fields

CreateFieldOptions does not let you specify a custom analyzer directly.
However, you can still configure which analyzer RavenDB will use for dynamic fields in the index definition by either:

Configuring an analyzer for a specific generated dynamic field name.
Configuring AllFields as a fallback for fields that do not have their own explicit analyzer configuration.

Use AllFields with care, since it can affect any field that does not have its own explicit analyzer configuration.

Configure analyzer for a specific dynamic field

LINQ_index
JavaScript_index

public class Products_ByLocalizedDescription : AbstractIndexCreationTask<Product>
{
    public Products_ByLocalizedDescription()
    {
        Map = products => from product in products
            select new
            {
                _ = product.Descriptions.Select(x =>
                    CreateField(
                        "Description_" + x.Key,
                        x.Value,
                        new CreateFieldOptions
                        {
                            Indexing = FieldIndexing.Search,
                            Storage = FieldStorage.No
                        }))
            };

        StoreAllFields(FieldStorage.No);

        // Configure an analyzer for a known generated dynamic field name
        // ==============================================================   
        Analyze("Description_English", "StopAnalyzer");
    }
}

public class Products_ByLocalizedDescription_JS : AbstractJavaScriptIndexCreationTask
{
    public Products_ByLocalizedDescription_JS()
    {
        Maps = new HashSet<string>
        {
            @"map('Products', function (product) {
                return {
                    _: Object.keys(product.Descriptions)
                        .map(function (key) {
                            return createField(
                              'Description_' + key,
                              product.Descriptions[key],
                              {
                                 indexing: 'Search',
                                 storage: 'No'
                              });
                    })
                };
            })"
        };

        Fields[Constants.Documents.Indexing.Fields.AllFields] = new IndexFieldOptions
        {
            Storage = FieldStorage.No
        };

        // Configure an analyzer for a known generated dynamic field name
        // ==============================================================
        Fields["Description_English"] = new IndexFieldOptions
        {
            Analyzer = "StopAnalyzer"
        };
    }
}

Configure a fallback analyzer for all fields

LINQ_index
JavaScript_index

public class Products_ByLocalizedDescription : AbstractIndexCreationTask<Product>
{
    public Products_ByLocalizedDescription()
    {
        Map = products => from product in products
            select new
            {
                _ = product.Descriptions.Select(x =>
                    CreateField(
                        "Description_" + x.Key,
                        x.Value,
                        new CreateFieldOptions
                        {
                            Indexing = FieldIndexing.Search,
                            Storage = FieldStorage.No
                        }))
            };
    
        StoreAllFields(FieldStorage.No);
    
        // Use this as a fallback for fields that do not have
        // their own explicit analyzer configuration
        // ==================================================   
        Analyze(Constants.Documents.Indexing.Fields.AllFields, "StopAnalyzer");
    }
}

public class Products_ByLocalizedDescription_JS : AbstractJavaScriptIndexCreationTask
{
    public Products_ByLocalizedDescription_JS()
    {
        Maps = new HashSet<string>
        {
            @"map('Products', function (product) {
                return {
                    _: Object.keys(product.Descriptions)
                        .map(function (key) {
                            return createField(
                              'Description_' + key,
                              product.Descriptions[key],
                              {
                                 indexing: 'Search',
                                 storage: 'No'
                              });
                    })
                };
            })"
        };

        // Use this as a fallback for fields that do not have
        // their own explicit analyzer configuration
        // ==================================================
        Fields[Constants.Documents.Indexing.Fields.AllFields] = new IndexFieldOptions
        {
            Storage = FieldStorage.No,
            Analyzer = "StopAnalyzer"
        };
    }
}

Which analyzer is used for the query term?

The following applies when a search() query targets a dynamic field:

Index configuration	Configuration key	Analyzer used for the query term
No analyzer is explicitly configured for the generated field name	`false`	The Default analyzer
No analyzer is explicitly configured for the generated field name	`true`	The Default search analyzer
`AllFields` is configured with a specific analyzer, e.g. `WhitespaceAnalyzer`	`false`	The analyzer resolved by the normal field fallback path, e.g. `WhitespaceAnalyzer`
`AllFields` is configured with a specific analyzer, e.g. `WhitespaceAnalyzer`	`true`	The Default search analyzer
The generated field name is explicitly configured, e.g. `Analyze("Description_English", "StopAnalyzer")`	`false`	The explicitly configured analyzer, e.g. `StopAnalyzer`
The generated field name is explicitly configured, e.g. `Analyze("Description_English", "StopAnalyzer")`	`true`	The explicitly configured analyzer, e.g. `StopAnalyzer`

CreateField syntax

Syntax for LINQ-index:

object CreateField(string name, object value);

object CreateField(string name, object value, bool stored, bool analyzed);

object CreateField(string name, object value, CreateFieldOptions options);

Syntax for JavaScript-index:

createField(fieldName, fieldValue, options); // returns object

Parameters	Type	Description
fieldName	`string`	Name of the dynamic-index-field
fieldValue	`object`	Value of the dynamic-index-field The field terms are derived from this value.
stored	`bool`	Sets FieldStorage `false` - will set `FieldStorage.No` (default value) `true` - will set `FieldStorage.Yes`
analyzed	`bool`	Sets FieldIndexing `null` - `FieldIndexing.Default` (default value) `false` - `FieldIndexing.Exact` `true` - `FieldIndexing.Search`
options	`CreateFieldOptions`	Dynamic-index-field options

CreateFieldOptions
Storage	`FieldStorage?`	Learn about storing data in the index.
Indexing	`FieldIndexing?`	Sets the field indexing behavior. Learn about using analyzers in the index.
TermVector	`FieldTermVector?`	Sets the field term-vector behavior. Learn about term vectors in the index.

All above examples have used the character _ in the dynamic-index-field definition.
However, using _ is just a convention. Any other string can be used instead.
This property is Not queryable, it is only used in the index definition syntax.
The actual dynamic-index-fields that are generated are defined by the CreateField method.

Indexed fields & terms view

The generated dynamic index fields and their indexed terms can be viewed in the Terms View in Studio.
Below are sample index fields & their terms generated from this example.

Figure 1. Go to terms view

Figure 2. Indexed fields & terms

Indexing field KEYS​

Example - index every field under an object​

Example - index every field​

Indexing field VALUES​

Example - basic​

Example - list​

Indexing dynamic fields for full-text search​

Configuring analyzers for dynamic fields​

Configure analyzer for a specific dynamic field​

Configure a fallback analyzer for all fields​

Which analyzer is used for the query term?​

CreateField syntax​

Syntax for LINQ-index:​

Syntax for JavaScript-index:​

Indexed fields & terms view​

Indexing field KEYS

Example - index every field under an object

Example - index every field

Indexing field VALUES

Example - basic

Example - list

Indexing dynamic fields for full-text search

Configuring analyzers for dynamic fields

Configure analyzer for a specific dynamic field

Configure a fallback analyzer for all fields

Which analyzer is used for the query term?

CreateField syntax

Syntax for LINQ-index:

Syntax for JavaScript-index:

Indexed fields & terms view