Skip to main content

Store using Bulk Insert

What is Bulk Insert​

  • Bulk Insert is RavenDB's high-performance API for efficiently inserting large volumes of documents and related data (such as attachments) in a streamed manner from the client to the server.

  • Instead of storing documents using the session (as explained in Storing entities), which opens an HTTP request for each SaveChanges call, Bulk Insert opens a single long-lived request and sends multiple documents in batches.
    This reduces the overhead of repeatedly opening and closing sessions or transactions for each individual request, and lowers network and processing costs.

  • The client organizes the data into batches before streaming it to the server.
    The server processes the stream as it arrives, each batch is processed and committed as a separate transaction.

  • There is no global transaction across the entire Bulk Insert, which means the operation as a whole is not atomic.
    If a failure occurs, previously committed batches remain, and only the current batch is rolled back.

  • Learn more about Bulk Insert in How to work with Bulk Insert.

Inserting attachments with Bulk Insert​

  • RavenDB’s Bulk Insert API lets you stream documents and add their attachments in a single bulk insert operation.

  • Use the AttachmentsFor(documentId).Store(...) method on the bulk insert instance to add one or more attachments to each document. This avoids the overhead of sending a separate request for every attachment, just like Bulk Insert does for documents.

  • Using the bulk insert operation, you can store a new document and then add attachments to it,
    or load or query existing documents beforehand and then add attachments to them during the bulk insert.
    Attachments can be stored locally on the RavenDB server or in remote storage.

  • A single TCP/HTTP streaming connection transports both documents and attachments.
    The client organizes the data into batches before streaming it to the server.
    The server processes the stream as it arrives; each batch is processed and committed as a separate transaction.

  • If storing a document or attachment fails, the server rolls back the current batch.
    Previously committed batches remain unaffected.

  • Note:

    • If an attachment with the same name is already associated with the document, its content will be overwritten.
    • The target document must already exist before its attachments are added. If you try to store an attachment for a document that hasn’t been stored yet, the operation will fail with a DocumentDoesNotExistException.

  • When to use Bulk Insert for attachments:

    • You need to insert a large number of attachments.
    • You don’t require full atomicity across the entire operation, partial success is acceptable.
    • You prioritize performance and throughput over session-level convenience or per-document consistency.

Bulk Insert attachments - Local storage​

Bulk insert documents with multiple attachments​

var basePath = @"C:\temp"; // folder containing files to attach
    
// Create a BulkInsert instance
using (var bulkInsertObj = store.BulkInsert())
{
    for (int i = 0; i < 10; i++)
    {
        var docId = $"users/{i}";
        var metadata = new MetadataAsDictionary
        {
            // Specify the target collection for the new documents
            // If not specified, documents will be created under the '@empty' collection
            [Constants.Documents.Metadata.Collection] = "users"
        };
        // Store a new user document in the BulkInsert instance
        bulkInsertObj.Store(new { Name = $"User {i}" }, docId, metadata);
    
        // Prepare attachment names
        var attachmentName1 = $"user_{i}.png";  // name for first attachment 
        var attachmentName2 = $"notes_{i}.txt"; // name for second attachment
        
        var filePath1 = Path.Combine(basePath, attachmentName1);
        var filePath2 = Path.Combine(basePath, attachmentName2);
        
        // Call 'AttachmentsFor' to get the BulkInsert attachment interface 
        // Pass the document ID for which to attach the files
        var attachmentsBulkInsert = bulkInsertObj.AttachmentsFor(docId);
        
        // Attach the first file:
        if (File.Exists(filePath1))
        {
            using (var fileStream1 = File.OpenRead(filePath1))
            {
                // Call 'Store' to add the file to the BulkInsert instance
                // The data stored in BulkInsert will be streamed to the server in batches 
                attachmentsBulkInsert.Store(attachmentName1, fileStream1, "image/png");
            }
        }
        // Attach the second file:
        if (File.Exists(filePath2))
        {
            using (var fileStream2 = File.OpenRead(filePath2))
            {
                attachmentsBulkInsert.Store(attachmentName2, fileStream2, "text/plain");
            }
        }
    }
}

Add attachments to existing documents using Bulk Insert​

// Query for existing user documents:
List<User> users;
    
using (var session = store.OpenSession())
{
    users = session.Query<User>()
        .Where(u => u.Age < 30)
        .ToList();
}

var basePath = @"C:\temp"; // folder containing files to attach

// Create a BulkInsert instance
using (var bulkInsertObj = store.BulkInsert())
{
    foreach (var user in users)
    {
        // Use the existing document id
        var docId = user.Id;

        // Prepare attachment name
        var attachmentName = $"user_{user.Id}.png";

        var filePath = Path.Combine(basePath, attachmentName);

        // Call 'AttachmentsFor' to get the BulkInsert attachment interface 
        // Pass the document ID for which to attach the files
        var attachmentsBulkInsert = bulkInsertObj.AttachmentsFor(docId);

        // Attach the file:
        if (File.Exists(filePath))
        {
            using (var fileStream = File.OpenRead(filePath))
            {
                // Call 'Store' to add the file to the BulkInsert instance
                // The data stored in BulkInsert will be streamed to the server in batches
                attachmentsBulkInsert.Store(attachmentName, fileStream, "image/png");
            }
        }
    }
}

Bulk Insert attachments - Remote storage​

  • Using the Bulk Insert operation, you can add attachments and mark them to be stored in remote storage.

  • Ensure that the remote attachment feature is enabled for the database and that remote destinations are configured.
    See Configure remote attachments.

  • Inserting attachments via Bulk Insert for remote storage is similar to inserting attachments for local storage,
    except that you need to specify the remote destination identifier and the time to upload.

  • Once the attachment is added to the document on the server, its content is first stored locally.
    The background task will then upload it to the specified remote destination when the scheduled time arrives,
    as described in Remote attachment storage flow.

var basePath = @"C:\temp"; // folder containing files to attach
    
// Create a BulkInsert instance
using (var bulkInsertObj = store.BulkInsert())
{
    for (int i = 0; i < 10; i++)
    {
        var docId = $"users/{i}";
        var metadata = new MetadataAsDictionary
        {
            // Specify the target collection for the new documents
            // If not specified, documents will be created under the '@empty' collection
            [Constants.Documents.Metadata.Collection] = "users"
        };
    
        // Store a new user document in the BulkInsert instance
        bulkInsertObj.Store(new { Name = $"User {i}" }, docId, metadata);
    
        // Prepare attachment name and path
        var attachmentName = $"user_{i}.png";
        var filePath = Path.Combine(basePath, attachmentName);
        
        // Call 'AttachmentsFor' to get the BulkInsert attachment interface 
        // Pass the document ID for which to attach the files
        var attachmentsBulkInsert = bulkInsertObj.AttachmentsFor(docId);
        
        // Attach the file:
        if (File.Exists(filePath))
        {
            using (var fileStream = File.OpenRead(filePath))
            {
                // Use the overload that accepts StoreAttachmentParameters
                // so we can specify the remote parameters
                var parameters = new StoreAttachmentParameters(attachmentName, fileStream)
                {
                    ContentType = "image/png",
                    RemoteParameters = new RemoteAttachmentParameters(
                        "my-amazon-storage",        // Remote destination ID
                        DateTime.UtcNow.AddDays(1)) // When to upload to remote storage
                };

                // Call 'Store' to add the file to the BulkInsert instance
                // Documents and attachments will be streamed to the server in batches
                attachmentsBulkInsert.Store(parameters);
            }
        }
    }
}

Syntax​

public AttachmentsBulkInsert AttachmentsFor(string id)
ParameterTypeDescription
idstringThe ID of the document to which the attachment will be added.
// Available overloads:    
    
public void Store(string name, Stream stream, string contentType = null)
public Task StoreAsync(
    string name, Stream stream, string contentType = null, CancellationToken token = default)
    
public void Store(StoreAttachmentParameters parameters)
public Task StoreAsync(
    StoreAttachmentParameters parameters, CancellationToken token = default)
ParameterTypeDescription
namestringName of attachment.
streamStreamThe stream containing the binary content of the attachment.
contentTypestringThe MIME type of the attachment (optional).
parametersStoreAttachmentParametersAn object that encapsulates all parameters required to store an attachment.
public class StoreAttachmentParameters
{
    public string Name { get; set; }   // The name of the attachment to store.
    public Stream Stream { get; set; } // The stream containing the attachment data.
    public string ChangeVector { get; set; } 
    public string ContentType { get; set; }
    
    // Set to null or omit to store the attachment locally
    public RemoteAttachmentParameters RemoteParameters { get; set; }
}
public class RemoteAttachmentParameters
{
    // The scheduled date and time when the attachment should be uploaded to the remote destination.
    // Preferably in UTC.
    public DateTime At { get; set; }
    
    // The identifier of the remote storage destination to which the attachment should be uploaded.
    public string Identifier { get; set; }
}
In this article