Indexes: Dynamic Index Fields

In RavenDB different documents can have different shapes.
Documents are schemaless - new fields can be added or removed as needed.
For such dynamic data, you can define indexes with dynamic-index-fields.
This allows querying the index on fields that aren't yet known at index creation time,
which is very useful when working on highly dynamic systems.
Any value type can be indexed, string, number, date, etc.
An index definition can contain both dynamic-index-fields and regular-index-fields.
In this page:

Indexing documents fields KEYS

Example - index any field under object

The following allows you to:

Index any field that is under the some object from the document.
After index is deployed, any new field added to the this object will be indexed as well. The document:

public class Product
{
  public string Id { get; set; }
  
  // The KEYS under the Attributes object will be dynamically indexed 
  // Fields added to this object after index creation time will also get indexed
  public Dictionary<string, object> Attributes { get; set; }
}

// Sample document content
{
  "Attributes": {
      "Color": "Red",
      "Size": 42
  }
}

The index:

The following index will index any field under the Attributes object from the document,
a dynamic-index-field will be created for each such field.
New fields added to the object after index creation time will be dynamically indexed as well.
The actual dynamic-index-field name on which you can query will be the attribute field key.
e.g. Keys Color & Size will become the actual dynamic-index-fields.

LINQ-index
JavaScript-index

public class Products_ByAttributeKey : AbstractIndexCreationTask<Product>
{
  public Products_ByAttributeKey()
  {
      Map = products => from p in products
          select new
          {
              // Call 'CreateField' to generate dynamic-index-fields from the Attributes object keys
              // Using '_' is just a convention. Any other string can be used instead of '_'
              
              // The actual field name will be item.Key
              // The actual field terms will be derived from item.Value
              _ = p.Attributes.Select(item => CreateField(item.Key, item.Value))
          };
  }
}

public class Products_ByAttributeKey_JS : AbstractJavaScriptIndexCreationTask
{
  public Products_ByAttributeKey_JS()
  {
      Maps = new HashSet<string>
      {
          @"map('Products', function (p) {
              return {
                  _: Object.keys(p.Attributes).map(key => createField(key, p.Attributes[key],
                      { indexing: 'Search', storage: true, termVector: null }))
              };
          })"
      };
  }
}

The query:

You can now query the generated dynamic-index fields.
Property _ is Not queryable, it is only used in the index definition syntax.
To get all documents with some 'Size' use:

DocumentQuery
RQL

IList<Product> matchingDocuments = session
  .Advanced
  .DocumentQuery<Product, Products_ByAttributeKey>()
   // 'Size' is a dynamic-index-field that was indexed from the Attributes object
  .WhereEquals("Size", 42)
  .ToList();

// 'Size' is a dynamic-index-field that was indexed from the Attributes object
from index 'Products/ByAttributeKey' where Size = 42

Example - index any field

The following allows you to:

Define an index on a collection without needing any common structure between the indexed documents.
After index is deployed, any new field added to the document will be indexed as well.

Consider if that is a true necessity, as indexing every single field can end up costing time and disk space.

The document:

public class Product
{
  public string Id { get; set; }

  // All KEYS in the document will be dynamically indexed 
  // Fields added to the document after index creation time will also get indexed
  public string FirstName { get; set; }
  public string LastName { get; set; }
  public string Title { get; set; }
  // ...
}

// Sample document content
  {
      "FirstName": "John",
      "LastName": "Doe",
      "Title": "Engineer",
      // ...
}

The index:

The following index will index any field from the document,
a dynamic-index-field will be created for each field.
New fields added to the document after index creation time will be dynamically indexed as well.
The actual dynamic-index-field name on which you can query will be the field key.
e.g. Keys FirstName & LastName will become the actual dynamic-index-fields.

JavaScript-index

public class Products_ByAnyField_JS : AbstractJavaScriptIndexCreationTask
{
  public Products_ByAnyField_JS()
  {
      // This will index EVERY FIELD under the top level of the document
      Maps = new HashSet<string>
      {
          @"map('Products', function (p) {
                return {
                    _: Object.keys(p).map(key => createField(key, p[key],
                        { indexing: 'Search', storage: true, termVector: null }))
                }
           })"
      };
  }
}

The query:

To get all documents with some 'LastName' use:

DocumentQuery
RQL

IList<Product> matchingDocuments = session
  .Advanced
  .DocumentQuery<Product, Products_ByAnyField_JS>()
   // 'LastName' is a dynamic-index-field that was indexed from the document
  .WhereEquals("LastName", "Doe")
  .ToList();

// 'LastName' is a dynamic-index-field that was indexed from the document
from index 'Products/ByAnyField/JS' where LastName = "Doe"

Indexing documents fields VALUES

Example - basic

This example shows:

Only the basic concept of creating a dynamic-index-field from the value of a document field.
Documents can then be queried based on those indexed values.
For a more practical usage see the Example below. The document:

public class Product
{
  public string Id { get; set; }
  
  // The VALUE of ProductType will be dynamically indexed
  public string ProductType { get; set; }
  public int PricePerUnit { get; set; }
}

// Sample document content
{
  "ProductType": "Electronics",
  "PricePerUnit": 23
}

The index:

The following index will index the value of document field 'ProductType'.
This value will be the dynamic-index-field name on which you can query.
e.g. Field value Electronics will be the dynamic-index-field.

LINQ-index
JavaScript-index

public class Products_ByProductType : AbstractIndexCreationTask<Product>
{
  public Products_ByProductType()
  {
      Map = products => from p in products
          select new
          {
              // Call 'CreateField' to generate the dynamic-index-fields
              // The field name will be the value of document field 'ProductType'
              // The field terms will be derived from document field 'PricePerUnit'
              _ = CreateField(p.ProductType, p.PricePerUnit)
          };
  }
}

public class Products_ByProductType_JS : AbstractJavaScriptIndexCreationTask
{
  public Products_ByProductType_JS()
  {
      Maps = new HashSet<string>
      {
          @"map('Products', function (p) {
              return {
                  _: createField(p.ProductType, p.PricePerUnit,
                      { indexing: 'Search', storage: true, termVector: null })
              };
          })"
      };
  }
}

The query:

To get all documents of some product type having a specific price per unit use:

DocumentQuery
RQL

IList<Product> matchingDocuments = session
  .Advanced
  .DocumentQuery<Product, Products_ByProductType>()
   // 'Electronics' is the dynamic-index-field that was indexed from document field 'ProductType'
  .WhereEquals("Electronics", 23)
  .ToList();

// 'Electronics' is the dynamic-index-field that was indexed from document field 'ProductType'
from index 'Products/ByProductType' where Electronics = 23

Example - list

The following allows you to:

Index values from items in a list
After index is deployed, any item added this list in the document will be dynamically indexed as well. The document:

public class Product
{
  public string Id { get; set; }
  public string Name { get; set; }
  
  // For each element in this list, the VALUE of property 'PropName' will be dynamically indexed
  // e.g. Color, Width, Length (in ex. below) will become dynamic-index-fields
  public List<Attribute> Attributes { get; set; }
}

public class Attribute
{
  public string PropName { get; set; }
  public string PropValue { get; set; }
}

// Sample document content
{
  "Name": "SomeName",
  "Attributes": [
      {  
          "PropName": "Color",
          "PropValue": "Blue"
      },
      {
          "PropName": "Width",
          "PropValue": "10"
      },
      {
          "PropName": "Length",
          "PropValue": "20"
      },
      ...
  ]
}

The index:

The following index will create a dynamic-index-field per item in the document's Attributes list.
New items added to the Attributes list after index creation time will be dynamically indexed as well.
The actual dynamic-index-field name on which you can query will be the item's PropName value.
e.g. 'PropName' value Width will be a dynamic-index-field.

LINQ-index
JavaScript-index

public class Attributes_ByName : AbstractIndexCreationTask<Product>
{
  public Attributes_ByName()
  {
      Map = products => from a in products
          select new
          {
              // Define the dynamic-index-fields by calling CreateField
              // A dynamic-index-field will be generated for each item in the Attributes list
              
              // For each item, the field name will be the value of field 'PropName'
              // The field terms will be derived from field 'PropValue'
              _ = a.Attributes.Select(item => CreateField(item.PropName, item.PropValue)),
              
              // A regular index field can be defined as well:
              Name = a.Name
          };
  }
}

public class Attributes_ByName_JS : AbstractJavaScriptIndexCreationTask
{
  public Attributes_ByName_JS()
  {
      Maps = new HashSet<string>
      {
          @"map('Products', function (p) {
              return {
                  _: p.Attributes.map(item => createField(item.PropName, item.PropValue,
                      { indexing: 'Search', storage: true, termVector: null })),
                 Name: p.Name
              };
          })"
      };
  }
}

The query:

To get all documents matching a specific attribute property use:

DocumentQuery
RQL

IList<Product> matchingDocuments = session
  .Advanced
  .DocumentQuery<Product, Attributes_ByName>()
   // 'Width' is a dynamic-index-field that was indexed from the Attributes list
  .WhereEquals("Width", 10)
  .ToList();

// 'Width' is a dynamic-index-field that was indexed from the Attributes list
from index 'Attributes/ByName' where Width = 10

CreateField syntax

Syntax for LINQ-index:

object CreateField(string name, object value);

object CreateField(string name, object value, bool stored, bool analyzed);

object CreateField(string name, object value, CreateFieldOptions options);

Syntax for JavaScript-index:

createField(fieldName, fieldValue, options); // returns object

Parameters
fieldName	`string`	Name of the dynamic-index-field
fieldValue	`object`	Value of the dynamic-index-field
The field Terms are derived from this value.
stored	`bool`	Sets FieldStorage

false - will set FieldStorage.No (default value) true - will set FieldStorate.Yes | | analyzed | bool | Sets FieldIndexing

null - FieldIndexing.Default (default value) false - FieldIndexing.Exact true - FieldIndexing.Search | | options | CreateFieldOptions | Dynamic-index-field options |

CreateFieldOptions
Storage	`FieldStorage?`	Learn about storing data in the index.
Indexing	`FieldIndexing?`	Learn about using analyzers in the index.
TermVector	`FieldTermVector?`	Learn about term vectors in the index.

All above examples have used the character _ in the dynamic-index-field definition.
However, using _ is just a convention. Any other string can be used instead.
This property is Not queryable, it is only used in the index definition syntax.
The actual dynamic-index-fields that are generated are defined by the CreateField method.

Indexed fields & terms view

The generated dynamic-index-fields and their indexed terms can be viewed in the Terms View.
Below are sample index fields & their terms generated from the last example.

Figure 1. Go to terms view

Figure 2. Indexed fields & terms

Indexing documents fields KEYS​

Example - index any field under object​

Example - index any field​

Indexing documents fields VALUES​

Example - basic​

Example - list​

CreateField syntax​

Syntax for LINQ-index:​

Syntax for JavaScript-index:​

Indexed fields & terms view​

Indexing documents fields KEYS

Example - index any field under object

Example - index any field

Indexing documents fields VALUES

Example - basic

Example - list

CreateField syntax

Syntax for LINQ-index:

Syntax for JavaScript-index:

Indexed fields & terms view