Sharding: Restore

• A sharded database's backup is a set of backup files that were created by the database's shards.

• To restore a sharded database, we need to pass the restore operation paths to the locations of the backup files so it can retrieve and restore them.

  Shards must be restored in their original order.
  The backup of shard 1, for example, must be restored as shard 1.

• Backup files can be restored from local shard node storage or from a remote location.

• A backed-up sharded database can be restored in part or in full, to a sharded or a non-sharded database.

• Only logical backups are supported. Snapshot backups cannot be created or restored for sharded databases.

• .ravendbdump files (exported from RavenDB databases) and backup files can also be imported into a database (sharded or non-sharded).

• A backup created for a non-sharded database cannot be restored as a sharded database.

• In this page:

  • Restore
    • Set Paths to Backup Files Locations
    • Define a Restore Configuration
      • RestoreBackupConfigurationBase
    • Run RestoreBackupOperation with the Restore Configuration
    • Examples
  • Restoring to a Selected Restore Point
  • Restore Options Summary

Restore

To restore a sharded database, we need to:

Set Paths to Backup Files Locations

When a shard stores a backup file, it may store it locally (on the shard node's storage) or remotely (supported platforms currently include S3 Buckets, Azure Blobs, and Google cloud).

To restore the backup files, we need to provide the restore process with each shard's backup folder location.
The shards' backup folder locations, and additional data regarding the backups, are provided in a dictionary of SingleShardRestoreSetting objects.

SingleShardRestoreSetting

public class SingleShardRestoreSetting
{
  // Shard number 
  public int ShardNumber { get; set; }
  // Node tag
  public string NodeTag { get; set; }
  // Folder name
  public string FolderName { get; set; }
  // Restore up to (including) this incremental backup file
  public string LastFileNameToRestore { get; set; }
}

• Parameters:

  Parameter	Value	Functionality
  ShardNumber	int	The shard number that will be given to the restored shard. should normally be similar to the original shard number.
  NodeTag	string	The node to restore the shard on.
  FolderName	string	The name of the folder that holds the backup file/s.
  LastFileNameToRestore	string	Last incremental backup file to restore. If omitted, restore all backup files in folder.

  When setting ShardNumber, please make sure that all shards are given the same numbers they had when they were backed-up.
  Giving a restored shard a number different than its original number will place buckets on the wrong shards and cause mapping errors.

  E.g., a backup of shard 1 must be restored as shard 1: ShardNumber = 1

  When restoring a local shard backup, make sure that the backup file resides on the node that the shard's NodeTag property is set to, so the restore process can find the file.

  E.g., if a backup file that's been produced by node A is now restored to node B (NodeTag = "B"), place the backup file in the backup folder of node B before initiating the restore operation.

Define a Restore Configuration

To restore the database, we pass the Restore Operation a configuration object.

• The configuration object inherits properties from the RestoreBackupConfigurationBase class (discussed below) and defines additional sharding-specific settings.

• We choose what configuration object to pass the restore operation, by the backup files' location.
  The backup files may be located locally (on each shard machine storage) or remotely (in a cloud location).

  Configuration Object	Backup Location	Additional Properties
  RestoreBackupConfiguration	Local shard storage	None (see example)
  RestoreFromS3Configuration	AWS S3 Bucket	S3Settings (see S3 example)
  RestoreFromAzureConfiguration	MS Azure Blob	AzureSettings (see Azure example)
  RestoreFromGoogleCloudConfiguration	Google Cloud Bucket	GoogleCloudSettings (see Google Cloud example)

RestoreBackupConfigurationBase

RestoreBackupConfigurationBase is a parent class to all the configuration types mentioned above, allowing you to set backups encryption settings among other options.

public abstract class RestoreBackupConfigurationBase
{
  public string DatabaseName { get; set; }

  public string LastFileNameToRestore { get; set; }

  public string DataDirectory { get; set; }

  public string EncryptionKey { get; set; }

  public bool DisableOngoingTasks { get; set; }

  public bool SkipIndexes { get; set; }

  public ShardedRestoreSettings ShardRestoreSettings { get; set; }

  public BackupEncryptionSettings BackupEncryptionSettings { get; set; }

}

• Parameters:

  Parameter	Value	Functionality
  DatabaseName	string	Name for the new database.
  LastFileNameToRestore (Optional - omit for default)	string	Last incremental backup file to restore. Ignored when restoring a sharded database. SingleShardRestoreSetting.LastFileNameToRestore used instead, per shard.
  DataDirectory (Optional - omit for default)	string	The new database data directory. Default folder: Under the "Databases" folder In a folder carrying the restored database name.
  EncryptionKey (Optional - omit for default)	string	A key for an encrypted database. Default behavior: Try restoring as if DB is unencrypted.
  DisableOngoingTasks (Optional - omit for default)	boolean	true - disable ongoing tasks when Restore is complete. false - enable ongoing tasks when Restore is complete. Default: false Ongoing tasks will run when Restore is complete.
  SkipIndexes (Optional - omit for default)	boolean	true - disable indexes import, false - enable indexes import. Default: false Restore all indexes.
  ShardRestoreSettings	ShardedRestoreSettings	a dictionary of SingleShardRestoreSetting instances defining paths to backup locations

{`public class ShardedRestoreSettings
\{
  public Dictionary<int, 
    SingleShardRestoreSetting> Shards 
                            \{ get; set; \}
\}
`}

| | BackupEncryptionSettings | BackupEncryptionSettings | Backup Encryption Settings |

Verify that RavenDB has full access to the backup locations and database files.
Make sure your server has write permission to DataDirectory.

Run RestoreBackupOperation with the Restore Configuration

Pass the configuration object you defined to the RestoreBackupOperation store operation to restore the database.

public RestoreBackupOperation(RestoreBackupConfigurationBase restoreConfiguration)

• Instead of RestoreBackupConfigurationBase, use the configuration object you prepared:
  • RestoreBackupConfiguration for locally-stored backups
  • RestoreFromS3Configuration to restore from S3
  • RestoreFromAzureConfiguration to restore from Azure
  • RestoreFromGoogleCloudConfiguration to restore from Google Cloud

Examples

Here are examples for restoring a sharded database using backup files stored locally and remotely.

• Local_Storage
• S3_Bucket
• Azure_Blob
• Google_Cloud

// Create a dictionary with paths to shard backups
var restoreSettings = new ShardedRestoreSettings
{
  Shards = new Dictionary<int, SingleShardRestoreSetting>(),
};

// First shard
restoreSettings.Shards.Add(0, new SingleShardRestoreSetting
{
  // Shard Number - which shard to restore to.
  // Please make sure that each shard is given 
  // the same number it had when it was backed up.
  ShardNumber = 0,
  // Node Tag - which node to restore to
  NodeTag = "A",
  // Backups Folder Name
  FolderName = "E:/RavenBackups/2023-02-12-09-52-27.ravendb-Books$0-A-backup"
});

// Second shard
restoreSettings.Shards.Add(1, new SingleShardRestoreSetting
{
  ShardNumber = 1,
  NodeTag = "B",
  FolderName = "E:/RavenBackups/2023-02-12-09-52-27.ravendb-Books$1-B-backup"
});

// Third shard
restoreSettings.Shards.Add(2, new SingleShardRestoreSetting
{
  ShardNumber = 2,
  NodeTag = "C",
  FolderName = "E:/RavenBackups/2023-02-12-09-52-27.ravendb-Books$2-C-backup",
});

var restoreBackupOperation = new RestoreBackupOperation(new RestoreBackupConfiguration
{
  // Database Name
  DatabaseName = "Books",
  // Paths to backup files
  ShardRestoreSettings = restoreSettings
});

var operation = await docStore.Maintenance.Server.SendAsync(restoreBackupOperation);

// Create a dictionary with paths to shard backups
var restoreSettings = new ShardedRestoreSettings
{
  Shards = new Dictionary<int, SingleShardRestoreSetting>(),
};

// First shard
restoreSettings.Shards.Add(0, new SingleShardRestoreSetting
{
  // Shard Number - which shard to restore to.
  // Please make sure that each shard is given 
  // the same number it had when it was backed up.
  ShardNumber = 0,
  // Node Tag - which node to restore to
  NodeTag = "A",
  // Backups Folder Name
  FolderName = "RavenBackups/2023-02-12-09-52-27.ravendb-Books$0-A-backup"
});

// Second shard
restoreSettings.Shards.Add(1, new SingleShardRestoreSetting
{
  ShardNumber = 1,
  NodeTag = "B",
  FolderName = "RavenBackups/2023-02-12-09-52-27.ravendb-Books$1-B-backup"
});

// Third shard
restoreSettings.Shards.Add(2, new SingleShardRestoreSetting
{
  ShardNumber = 2,
  NodeTag = "C",
  FolderName = "RavenBackups/2023-02-12-09-52-27.ravendb-Books$2-C-backup",
});

var restoreBackupOperation = new RestoreBackupOperation(new RestoreFromS3Configuration
{
  // Database Name
  DatabaseName = "Books",
  // Paths to backup files
  ShardRestoreSettings = restoreSettings,
  // S3 Bucket settings
  Settings = new S3Settings 
  {
      AwsRegionName = "us-east-1", // Optional
      BucketName = "your bucket name here",
      RemoteFolderName = "", // Replaced by restoreSettings.Shards.FolderName 
      AwsAccessKey = "your access key here",
      AwsSecretKey = "your secret key here",
  } 
});

var operation = await docStore.Maintenance.Server.SendAsync(restoreBackupOperation);

// Create a dictionary with paths to shard backups
var restoreSettings = new ShardedRestoreSettings
{
  Shards = new Dictionary<int, SingleShardRestoreSetting>(),
};

// First shard
restoreSettings.Shards.Add(0, new SingleShardRestoreSetting
{
  // Shard Number - which shard to restore to.
  // Please make sure that each shard is given 
  // the same number it had when it was backed up.
  ShardNumber = 0,
  // Node Tag - which node to restore to
  NodeTag = "A",
  // Backups Folder Name
  FolderName = "RavenBackups/2023-02-12-09-52-27.ravendb-Books$0-A-backup"
});

// Second shard
restoreSettings.Shards.Add(1, new SingleShardRestoreSetting
{
  ShardNumber = 1,
  NodeTag = "B",
  FolderName = "RavenBackups/2023-02-12-09-52-27.ravendb-Books$1-B-backup"
});

// Third shard
restoreSettings.Shards.Add(2, new SingleShardRestoreSetting
{
  ShardNumber = 2,
  NodeTag = "C",
  FolderName = "RavenBackups/2023-02-12-09-52-27.ravendb-Books$2-C-backup",
});

var restoreBackupOperation = new RestoreBackupOperation(new RestoreFromAzureConfiguration
{
  // Database Name
  DatabaseName = "Books",
  // Paths to backup files
  ShardRestoreSettings = restoreSettings,
  // Azure Blob settings
  Settings = new AzureSettings
  {
      StorageContainer = "storageContainer",
      RemoteFolderName = "", // Replaced by restoreSettings.Shards.FolderName 
      AccountName = "your account name here",
      AccountKey = "your account key here",
  }
});

var operation = await docStore.Maintenance.Server.SendAsync(restoreBackupOperation);

// Create a dictionary with paths to shard backups
var restoreSettings = new ShardedRestoreSettings
{
  Shards = new Dictionary<int, SingleShardRestoreSetting>(),
};

// First shard
restoreSettings.Shards.Add(0, new SingleShardRestoreSetting
{
  // Shard Number - which shard to restore to.
  // Please make sure that each shard is given 
  // the same number it had when it was backed up.
  ShardNumber = 0,
  // Node Tag - which node to restore to
  NodeTag = "A",
  // Backups Folder Name
  FolderName = "RavenBackups/2023-02-12-09-52-27.ravendb-Books$0-A-backup"
});

// Second shard
restoreSettings.Shards.Add(1, new SingleShardRestoreSetting
{
  ShardNumber = 1,
  NodeTag = "B",
  FolderName = "RavenBackups/2023-02-12-09-52-27.ravendb-Books$1-B-backup"
});

// Third shard
restoreSettings.Shards.Add(2, new SingleShardRestoreSetting
{
  ShardNumber = 2,
  NodeTag = "C",
  FolderName = "RavenBackups/2023-02-12-09-52-27.ravendb-Books$2-C-backup",
});

var restoreBackupOperation = new RestoreBackupOperation(new RestoreFromGoogleCloudConfiguration
{
  // Database Name
  DatabaseName = "Books",
  // Paths to backup files
  ShardRestoreSettings = restoreSettings,
  // Google Cloud settings
  Settings = new GoogleCloudSettings
  {
      BucketName = "your bucket name here",
      RemoteFolderName = "", // Replaced by restoreSettings.Shards.FolderName 
      GoogleCredentialsJson = "your credentials here"
  }
});

var operation = await docStore.Maintenance.Server.SendAsync(restoreBackupOperation);

Restoring to a Selected Restore Point

The default procedure

When a full backup is created, (for either a non sharded database or for any shard of a sharded database), a backup folder is created to contain it.
When incremental backups are created, they are stored in the folder of the last full backup.
To restore a backup, its folder name is provided. By default, the full backup stored in this folder will be restored, as well as all the incremental backups that were added to it over time.

Restoring a Non-sharded database to a selected restore point

• Read here about restoring a non-sharded database.
• In short, restoring a non-sharded database to a selected restore point is done by filling:
  • RestoreBackupConfiguration.BackupLocation with the backup location.
  • RestoreBackupConfiguration.LastFileNameToRestore with the name of the last incremental backup to restore.

Restoring a Sharded database to a selected restore point

• When a sharded database is restored, RestoreBackupConfiguration.BackupLocation and RestoreBackupConfiguration.LastFileNameToRestore mentioned above are overridden by per-shard settings.

• To restore the database of a particular shard up to a selected restore point simply add ShardRestoreSettings.SingleShardRestoreSetting.LastFileNameToRestore to the shard configuration and fill it with the name of the last incremental backup to restore.

• Example:

• Local_Storage
• S3_Bucket
• Azure_Blob
• Google_Cloud

// Create a dictionary with paths to shard backups
var restoreSettings = new ShardedRestoreSettings
{
  Shards = new Dictionary<int, SingleShardRestoreSetting>(),
};

// First shard
restoreSettings.Shards.Add(0, new SingleShardRestoreSetting
{
  ShardNumber = 0,
  NodeTag = "A",
  // Backups Folder Name
  FolderName = "E:/RavenBackups/2023-02-12-09-52-27.ravendb-Books$0-A-backup",
  // Last incremental backup to restore
  LastFileNameToRestore = "2023-02-12-10-30-00.ravendb-incremental-backup"
});

var restoreBackupOperation = new RestoreBackupOperation(new RestoreBackupConfiguration
{
  // Database Name
  DatabaseName = "Books",
  // Paths to backup files
  ShardRestoreSettings = restoreSettings
});

var operation = await docStore.Maintenance.Server.SendAsync(restoreBackupOperation);

// Create a dictionary with paths to shard backups
var restoreSettings = new ShardedRestoreSettings
{
  Shards = new Dictionary<int, SingleShardRestoreSetting>(),
};

// First shard
restoreSettings.Shards.Add(0, new SingleShardRestoreSetting
{
  ShardNumber = 0,
  NodeTag = "A",
  // Backups Folder Name
  FolderName = "RavenBackups/2023-02-12-09-52-27.ravendb-Books$0-A-backup",
  // Last incremental backup to restore
  LastFileNameToRestore = "2023-02-12-10-30-00.ravendb-incremental-backup"
});

var restoreBackupOperation = new RestoreBackupOperation(new RestoreFromS3Configuration
{
  // Database Name
  DatabaseName = "Books",
  // Paths to backup files
  ShardRestoreSettings = restoreSettings,
  // S3 Bucket settings
  Settings = new S3Settings
  {
      AwsRegionName = "us-east-1", // Optional
      BucketName = "your bucket name here",
      RemoteFolderName = "", // Replaced by restoreSettings.Shards.FolderName 
      AwsAccessKey = "your access key here",
      AwsSecretKey = "your secret key here",
  }
});

var operation = await docStore.Maintenance.Server.SendAsync(restoreBackupOperation);

// Create a dictionary with paths to shard backups
var restoreSettings = new ShardedRestoreSettings
{
  Shards = new Dictionary<int, SingleShardRestoreSetting>(),
};

// First shard
restoreSettings.Shards.Add(0, new SingleShardRestoreSetting
{
  ShardNumber = 0,
  NodeTag = "A",
  // Backups Folder Name
  FolderName = "RavenBackups/2023-02-12-09-52-27.ravendb-Books$0-A-backup",
  // Last incremental backup to restore
  LastFileNameToRestore = "2023-02-12-10-30-00.ravendb-incremental-backup"
});

var restoreBackupOperation = new RestoreBackupOperation(new RestoreFromAzureConfiguration
{
  // Database Name
  DatabaseName = "Books",
  // Paths to backup files
  ShardRestoreSettings = restoreSettings,
  // Azure Blob settings
  Settings = new AzureSettings
  {
      StorageContainer = "storageContainer",
      RemoteFolderName = "", // Replaced by restoreSettings.Shards.FolderName 
      AccountName = "your account name here",
      AccountKey = "your account key here",
  }
});

var operation = await docStore.Maintenance.Server.SendAsync(restoreBackupOperation);

// Create a dictionary with paths to shard backups
var restoreSettings = new ShardedRestoreSettings
{
  Shards = new Dictionary<int, SingleShardRestoreSetting>(),
};

// First shard
restoreSettings.Shards.Add(0, new SingleShardRestoreSetting
{
  ShardNumber = 0,
  NodeTag = "A",
  // Backups Folder Name
  FolderName = "RavenBackups/2023-02-12-09-52-27.ravendb-Books$0-A-backup",
  // Last incremental backup to restore
  LastFileNameToRestore = "2023-02-12-10-30-00.ravendb-incremental-backup"
});

var restoreBackupOperation = new RestoreBackupOperation(new RestoreFromGoogleCloudConfiguration
{
  // Database Name
  DatabaseName = "Books",
  // Paths to backup files
  ShardRestoreSettings = restoreSettings,
  // Google Cloud settings
  Settings = new GoogleCloudSettings
  {
      BucketName = "your bucket name here",
      RemoteFolderName = "", // Replaced by restoreSettings.Shards.FolderName 
      GoogleCredentialsJson = "your credentials here"
  }
});

var operation = await docStore.Maintenance.Server.SendAsync(restoreBackupOperation);

Restore Options Summary

Option	Supported on a Sharded Database	Comment
Restore from local shard storage	Yes
Restore from a remote location	Yes	Define a restore configuration with S3, Azure, or Google Cloud settings.
Restore a sharded database backup to a sharded database	Yes
Restore a sharded database backup to a non-sharded database	Yes
Restore a non-sharded database backup to a sharded database	No	A backup created for a non-sharded database CANNOT be restored to a sharded database.
Restore a Full database backup	Yes
Restore a Partial database backup	Yes
Restore a Logical database backup	Yes
Restore a Snapshot database backup	No	A snapshot backup CANNOT be restored by a sharded database.
Restore backed-up shards in a different order than the original	No	Always restore the shards in their original order.