Querying: MoreLikeThis

MoreLikeThis returns a list of similar documents that are related to a given document. This feature can be used for situations like when a user views an article. Many news sites show a list of the related articles at the bottom of the page. To accomplish this, the RavenDB MoreLikeThis uses the MoreLikeThis from the Lucene contrib project. To find out more about the algorithm, please read Aaron Johnson excellent blog post that is available here.

Setup

In order to work, MoreLikeThis requires access to the text in the index. Therefore, the index being queried needs to store the fields or store the term vectors for those fields.

public class Article
{
  public string Id { get; set; }
  public string Name { get; set; }
  public string ArticleBody { get; set; }
}

public class Articles_ByArticleBody : AbstractIndexCreationTask<Article>
{
  public Articles_ByArticleBody()
  {
      Map = docs => from doc in docs
                    select new
                    {
                        doc.ArticleBody
                    };

      Stores.Add(x => x.ArticleBody, FieldStorage.Yes);
      Analyzers.Add(x => x.ArticleBody, "StandardAnalyzer");
  }
}

Basic Usage

MoreLikeThis has many defaults already set and the simplest mode will satisfy the majority of the usage scenarios.

• Query
• DocumentQuery
• RQL

List<Article> articles = session
  .Query<Article, Articles_ByArticleBody>()
  .MoreLikeThis(builder => builder
      .UsingDocument(x => x.Id == "articles/1"))
  .ToList();

List<Article> articles = session.Advanced
  .DocumentQuery<Article, Articles_ByArticleBody>()
  .MoreLikeThis(builder => builder
      .UsingDocument(x => x.WhereEquals(y => y.Id, "articles/1")))
  .ToList();

from index 'Articles/ByArticleBody' 
where morelikethis(id() = 'articles/1')

MoreLikeThis will use all the fields defined in an index. To use only a specific field or fields, pass them in MoreLikeThisOptions.Fields property.

• Query
• DocumentQuery
• RQL

List<Article> articles = session
  .Query<Article, Articles_ByArticleBody>()
  .MoreLikeThis(builder => builder
      .UsingDocument(x => x.Id == "articles/1")
      .WithOptions(new MoreLikeThisOptions
      {
          Fields = new[] { nameof(Article.ArticleBody) }
      }))
  .ToList();

List<Article> articles = session.Advanced
  .DocumentQuery<Article, Articles_ByArticleBody>()
  .MoreLikeThis(builder => builder
      .UsingDocument(x => x.WhereEquals(y => y.Id, "articles/1"))
      .WithOptions(new MoreLikeThisOptions
      {
          Fields = new[] { nameof(Article.ArticleBody) }
      }))
  .ToList();

from index 'Articles/ByArticleBody' 
where morelikethis(id() = 'articles/1', '{ "Fields" : [ "ArticleBody" ] }')

Options

Default parameters can be changed by manipulating MoreLikeThisOptions properties and passing them to the MoreLikeThis.

Options
MinimumTermFrequency	int?	Ignores terms with less than this frequency in the source doc
MaximumQueryTerms	int?	Returns a query with no more than this many terms
MaximumNumberOfTokensParsed	int?	The maximum number of tokens to parse in each example doc field that is not stored with TermVector support
MinimumWordLength	int?	Ignores words less than this length or, if 0, then this has no effect
MaximumWordLength	int?	Ignores words greater than this length or if 0 then this has no effect
MinimumDocumentFrequency	int?	Ignores words which do not occur in at least this many documents
MaximumDocumentFrequency	int?	Ignores words which occur in more than this many documents
MaximumDocumentFrequencyPercentage	int?	Ignores words which occur in more than this percentage of documents
Boost	bool?	Boost terms in query based on score
BoostFactor	float?	Boost factor when boosting based on score
StopWordsDocumentId	string	Document ID containing custom stop words
Fields	string[]	Fields to compare

Stop Words

Some of Lucene analyzers have a built-in list of common English words that are usually not useful for searching (like "a", "as", "the" etc.). Those words are called stop words and they are considered to be uninteresting and ignored. If a used analyzer does not support stop words, or you need to overload them, you can specify your own set of stop words. A document with a list of stop words can be stored in RavenDB by storing the MoreLikeThisStopWords document:

session.Store(new MoreLikeThisStopWords
{
  Id = "Config/Stopwords",
  StopWords = new List<string> { "I", "A", "Be" }
});

The document ID is then set in the MoreLikeThisOptions.

Remarks

Please note that default values for settings, like MinimumDocumentFrequency, MinimumTermFrequency, and MinimumWordLength, may result in filtering out related articles, especially when there is little data set (e.g. during development).