---
title: "Sharding: ETL"
description: "* From a user's point of view, ETL usage on a sharded RavenDB database"
url: "https://docs.ravendb.net/6.2/sharding/etl"
version: "6.2"
---

# Sharding: ETL

> RavenDB publishes an official agent skill for coding assistants at https://docs.ravendb.net/skills/ravendb/SKILL.md (also installable with `npx skills add ravendb/skills`). The skill describes itself as: "Read this before writing any RavenDB query, client call or /databases/ request. Covers server setup, document modelling, the Node.js, .NET and Python clients, RQL, indexes, vector search and AI agents. Code written from recall tends to run and return the wrong rows, because RQL resembles SQL and the clients take field names as plain strings."

## Sharding: ETL

> **Note**
>
> - From a user's point of view, ETL usage on a sharded RavenDB database
>   is similar to its usage on a non-sharded database, including unchanged
>   syntax and support for the same list of destinations.
>
> - Most changes are meant to remain behind the scenes and allow the
>   transition to a sharded database to be seamless.
>
> - A user defines an ETL task once.  
>   As the shards detect that a task was defined, each shard defines
>   its own local ETL task, based on the user's settings.
>
> - Resharding may result in the transfer of some documents more than
>   once to the ETL destination. It is the user's responsibility to
>   [detect and handle](#etl-and-resharding) such duplicates.
>
> - ETL tasks information is available [per database and per shard](#retrieving-shard-specific-etl-task-info).
>
> - In this page:
>   - [ETL](#etl)
>     - [Sharded and Non-Sharded ETL Tasks](#sharded-and-non-sharded-etl-tasks)
>     - [Non-Sharded Database ETL Tasks](#non-sharded-database-etl-tasks)
>     - [Sharded Database ETL Tasks](#sharded-database-etl-tasks)
>   - [ETL and Resharding](#etl-and-resharding)
>   - [ETL Queries](#etl-queries)
>   - [Retrieving Shard-Specific ETL Task Info](#retrieving-shard-specific-etl-task-info)

## ETL

## Sharded and Non-Sharded ETL Tasks

From a user's point of view, creating an ongoing ETL process is
done by defining and running **a single ETL task**, just like it
is done under a non-sharded database.

Behind the scenes, though, each shard defines and uses its own
ETL task to send data from its database to the ETL destination
independently from other shards.

Distributing the ETL responsibility between the shards allows
RavenDB to keep its ETL destination updated with data additions
and modifications no matter how large the overall database gets.

### Non-Sharded Database ETL Tasks

- A complete replica of the database is kept by each cluster node.
- Any node can therefore be made
  [responsible](https://docs.ravendb.net/6.2/server/clustering/distribution/highly-available-tasks.md#responsible-node)
  for ETL by the cluster.
- The responsible node runs the ETL task periodically to update
  the ETL destination with any data changes.

### Sharded Database ETL Tasks

- Each shard hosts a unique dataset, so no single node can
  monitor the entire database.
- When a user defines an ETL task, either via Studio or
  using API commands like `PutConnectionStringOperation`
  and `AddEtlOperation`, the change made in the database
  record triggers each shard to create an ETL task of
  its own, based on the user-defined task.  
  This creation of multiple ETL tasks, one per shard, is
  automatic and requires no additional actions from the user.
- Each shard appoints [one of its nodes](https://docs.ravendb.net/6.2/sharding/overview.md#shard-replication)
  responsible for the execution of the shard's ETL task.
- The shards' ETL tasks behave just like an ETL task of
  a non-sharded database would, **E**xtractng relevant
  data from the shard's database, **T**ransforming it using
  a user-defined script, and **L**oading it to the destination.
- If the responsible node fails a failover scenario will start,
  another shard node will be made responsible for the task,
  and the transfer will continue from the point of failure.

## ETL Queries

Queries used by an ETL task's transform script on a sharded
database are basically no different than queries executed over
a non-sharded database.  
However, as some querying features are
[not yet implemented](https://docs.ravendb.net/6.2/sharding/querying.md#unsupported-querying-features)
on a sharded database (e.g. loading a document that resides
on a different shard will fail) and others behave a little
differently than their non-sharded equivalents (e.g.
[filer](https://docs.ravendb.net/6.2/sharding/querying.md#filtering-results-in-a-sharded-database)),
it is recommended to read the pages dedicated to [indexing](https://docs.ravendb.net/6.2/sharding/indexing.md)
and [querying](https://docs.ravendb.net/6.2/sharding/querying.md) on a sharded database.

## ETL and Resharding

It may happen that an ETL task would send the same data more than once.  
One scenario that would make this happen is resharding: a document can
be sent from one shard by the shard's ETL task, resharded, and then
sent again to the ETL destination by its new shard's ETL task.

Some ETL destinations will store duplicate incoming documents instead
of their former copies. Others, like OLAP and [Queue ETL](https://docs.ravendb.net/6.2/server/ongoing-tasks/etl/queue-etl/overview.md)
destinations, will **Not** automatically recognize such events.  
It is the user's responsibility to verify that the loaded documents
are handled as expected when they arrive.

> **Note**
>
> OLAP helps users detect duplications using `lastModified`, see a more
> thorough discussion of this [here](https://docs.ravendb.net/6.2/studio/database/tasks/ongoing-tasks/olap-etl-task.md#transform-scripts)
> and relevant code samples [here](https://docs.ravendb.net/6.2/server/ongoing-tasks/etl/olap.md#athena-examples).

## Retrieving Shard-Specific ETL Task Info

- The [GetOngoingTaskInfoOperation](https://docs.ravendb.net/6.2/client-api/operations/maintenance/ongoing-tasks/ongoing-task-operations.md)
  store operation can be used on a non-sharded database to retrieve a task's information.

- `GetOngoingTaskInfoOperation` can also be used on a sharded database.
  - **Get Task Info Per Database**  
    Run `GetOngoingTaskInfoOperation` using `store.Maintenance.Send`
    to retrieve information regarding the basic task defined by the user.  
    The information includes the task's name and ID.
  - **Get Task Info Per Shard**  
    Run `GetOngoingTaskInfoOperation` using `store.Maintenance.ForShard(x).Send`,
    where x is the shard number, to retrieve information about the selected
    shard's task.  
    Much more information is available here, including details of the
    responsible and mentor nodes.

```csharp
// Get basic info regarding the user-defined task
var ongoingTask = store.Maintenance.Send(
              new GetOngoingTaskInfoOperation(name, OngoingTaskType.RavenEtl));

if (ongoingTask != null)
{
  // go through the shards and retrieve their info
  for (int i = 0; i < 3; i++)
  {
      var singleShardInfo = store.Maintenance.ForShard(i).Send(
              new GetOngoingTaskInfoOperation(name, OngoingTaskType.RavenEtl));

      shardName[i] = singleShardInfo.TaskName;
      NodeTag[i] = singleShardInfo.ResponsibleNode.NodeTag;
      ongoingTaskConnectionStatus[i] = singleShardInfo.TaskConnectionStatus;
      mentorNode[i] = singleShardInfo.MentorNode;
      ongoingTaskConnectionStatus[i] = singleShardInfo.TaskConnectionStatus;

  }
}
```
