Skip to main content

Debugging Index Errors

Index compilation errors​

An index definition like this will fail:

{
    "Name": "Posts_TitleLength",
    "Maps" : [
        "from doc in docs where doc.Type == 'posts' select new { doc.Title.Length }"
    ]
}

The error is caused by using single quotes to enclose a string.
In C#, strings must be enclosed in double quotes.

This results in the following compilation error:

IndexCompilationException: Failed to compile index Posts_TitleLength

using System;
using System.Collections;
using System.Collections.Generic;
using System.Globalization;
using System.Linq;
using System.Text.RegularExpressions;
using Lucene.Net.Documents;
using Raven.Server.Documents.Indexes.Static;
using Raven.Server.Documents.Indexes.Static.Linq;
using Raven.Server.Documents.Indexes.Static.Extensions;

namespace Raven.Server.Documents.Indexes.Static.Generated
{
    public class Index_Posts_TitleLength : StaticIndexBase
    {
        IEnumerable Map_0(IEnumerable<dynamic> docs)
        {
            foreach (var doc in docs)
            {
                if ((doc.Type == 'posts') == false)
                    continue;
                yield return new
                {
                    doc.Title.Length
                }

                ;
            }
        }

        public Index_Posts_TitleLength()
        {
            this.AddMap("@all_docs", this.Map_0);
            this.OutputFields = new string[] { "Length" };
        }
    }
}

(20,34): error CS1012: Too many characters in character literal

The compiler output indicates that the error is at line 20, column 34: if ((doc.Type == 'posts') == false).

This is enough information to identify the problem in the index definition.
Compilation errors are detected immediately, before the index starts processing documents.
To resolve the error, fix the index definition.

Generated Code​

The generated index code may differ from the index definition you submitted.
RavenDB transforms and optimizes the submitted index definition before compiling it to achieve best performance.

Index execution errors​

A common cause of execution errors is an index that assumes all documents have a specific structure.
For example, consider this index:

{
    "Name": "YearOfBirth",
    "Maps" : [
        "from doc in docs select new { YearOfBirth = DateTime.Parse(doc.DateOfBirth).Year }"
    ]
}

This index assumes that every document has a DateOfBirth property and that its value can be parsed as a DateTime.
If a document doesn't have this property, accessing it returns null, and DateTime.Parse will throw an ArgumentNullException while the index is running.

Because indexes are updated in the background, users may not notice these errors immediately.
Index execution errors can be viewed in the following ways:

  • Use the database REST API endpoints:
    /databases/<database-name>/indexes/stats and /databases/<database-name>/indexes/errors.
  • In Studio, view index activity and error indicators in the indexes list view.
  • For a dedicated index errors view, go to Indexes > Index Errors.
{
   "Results":[
      {
         "Name":"TitleLength",
         "MapAttempts":2,
         "MapSuccesses":2,
         "MapErrors":2,
         "ReduceAttempts":null,
         "ReduceSuccesses":null,
         "ReduceErrors":null,
         "MappedPerSecondRate":0.0,
         "ReducedPerSecondRate":0.0,
         "MaxNumberOfOutputsPerDocument":0,
         "Collections":{
            "@all_docs":{ (
               "LastProcessedDocumentEtag":791,
               "LastProcessedTombstoneEtag":0,
               "DocumentLag":0,
               "TombstoneLag":0
            }
         },
         "LastQueryingTime":"2018-02-26T14:17:08.7454587Z",
         "State":"Error",
         "Priority":"Normal",
         "CreatedTimestamp":"2018-02-26T14:17:08.7092294Z",
         "LastIndexingTime":"2018-02-26T14:17:08.7512648Z",
         "IsStale":true,
         "LockMode":"Unlock",
         "Type":"Map",
         "Status":"Paused",
         "EntriesCount":0,
         "ErrorsCount":2,
         "IsInvalidIndex":true
      }
   ]
}

RavenDB shows which index encountered errors, which document caused each error, and the exception that was thrown. The errors collection contains the last 500 errors stored per index.

In addition, the server logs may contain additional information about the error.

Index marked as faulty on startup​

Unlike compilation and execution errors, which occur while an index is running,
an index can also be marked as Faulty when the database starts up - before the index ever processes a document.

This startup fault happens when RavenDB detects that the index storage was created for a different database than the one currently loading it. The faulty index will not open, and any query against it will throw an exception:

IndexOpenException: Could not open index because stored database ID ('<source-database-id>') is
different than current database ID ('<current-database-id>').
A common reason for this is that the index was copied from another database.

What RavenDB checks​

Each index stores the Database ID of the database it was created for.
On startup, RavenDB compares that stored ID with the ID of the database currently loading the index.

If the IDs do not match, RavenDB treats the index files as copied from another database and marks the index as Faulty. This most commonly happens when raw database/index files are copied between databases as a manual deploy, backup, or restore shortcut.

This validation applies to both auto-indexes and static indexes.

The Indexing.SkipDatabaseIdValidationOnIndexOpening configuration key disables this check. It is intended for support scenarios only, not as a fix - a mismatched index that is allowed to open can still return incomplete results.

Why this validation matters​

An index records the last document etag it processed. RavenDB uses that value to skip documents whose etag is at or below it and continue indexing from the next document.

If index files are copied from another database, the copied index can carry a last-processed etag from the source database. That etag may be higher than the etags of documents in the target database, so RavenDB can treat those documents as already indexed and skip them.

The result is a silent query-correctness issue: the documents exist, but their index terms were never written, so queries that rely on the copied index can miss them, even though the index appears healthy (not stale, not paused, no errors).

Failing fast at startup turns that silent data-correctness problem into an explicit, visible error.

How to recover​

To avoid this situation,
use RavenDB's backup & restore or export & import features instead of copying data directories between databases.

Marking index as errored​

  • An index can end up in the Error state in more than one way:

    • Repeated execution errors:
      RavenDB can mark a running index as errored when it keeps failing while processing documents.
    • Startup/opening failures:
      RavenDB can also create a faulty, errored index instance when an existing index cannot be opened,
      such as when the index storage belongs to a different database.

  • To protect the database from indexes that keep failing during execution, RavenDB tracks indexing attempts and errors. The failure rate is calculated from map, referenced-document map, and reduce work.

    For a stale index, RavenDB waits until more than 100 indexing attempts have been made before applying the failure-rate threshold. Once the index is no longer stale, RavenDB can evaluate the failure rate sooner.

    In either case, once RavenDB evaluates the failure rate, it marks the index as errored if the rate exceeds 15%.
    As a special case, a non-stale index for which every indexing attempt has failed is marked as errored immediately, without waiting for the usual attempt threshold.

  • An errored index cannot be queried.
    Queries that use an errored index will throw an exception.

  • Recovery depends on the cause:

    • If the index definition is invalid, fix or replace it.
    • If the index data is invalid or must be rebuilt, reset the index or delete and recreate it.
    • For startup faults caused by copied index storage, use the recovery steps described in Index faulted on startup.

In this article