Add ETL Operation
-
Use the
AddEtlOperationmethod to add a new ongoing ETL task to your database. -
To learn about ETL (Extract, Transfer, Load) ongoing tasks, see the ETL Basics article.
To learn how to manage ETL tasks from Studio, see Ongoing tasks - overview. -
In this page:
Add RavenDB ETL task
- Learn about the RavenDB ETL task in the RavenDB ETL task article.
- Learn how to define a connection string for the RavenDB ETL task here: Add a RavenDB connection string
- To manage the RavenDB ETL task from Studio, see Studio: RavenDB ETL task.
The following example adds a RavenDB ETL task:
// Define the RavenDB ETL task configuration object
// ================================================
var ravenEtlConfig = new RavenEtlConfiguration
{
Name = "task-name",
ConnectionStringName = "raven-connection-string-name",
Transforms =
{
new Transformation
{
// The script name
Name = "script-name",
// RavenDB collections the script uses
Collections = { "Employees" },
// The transformation script
Script = @"loadToEmployees ({
Name: this.FirstName + ' ' + this.LastName,
Title: this.Title
});"
}
},
// Do not prevent task failover to another node (optional)
PinToMentorNode = false
};
// Define the AddEtlOperation
// ==========================
var operation = new AddEtlOperation<RavenConnectionString>(ravenEtlConfig);
// Execute the operation by passing it to Maintenance.Send
// =======================================================
AddEtlOperationResult result = store.Maintenance.Send(operation);
Add SQL ETL task
- Learn about the SQL ETL task in the SQL ETL task article.
- Learn how to define a connection string for the SQL ETL task here: Add an SQL connection string
The following example adds an SQL ETL task:
// Define the SQL ETL task configuration object
// ============================================
var sqlEtlConfig = new SqlEtlConfiguration
{
Name = "task-name",
ConnectionStringName = "sql-connection-string-name",
SqlTables =
{
new SqlEtlTable {TableName = "Orders", DocumentIdColumn = "Id", InsertOnlyMode = false},
new SqlEtlTable {TableName = "OrderLines", DocumentIdColumn = "OrderId", InsertOnlyMode = false},
},
Transforms =
{
new Transformation
{
Name = "script-name",
Collections = { "Orders" },
Script = @"var orderData = {
Id: id(this),
OrderLinesCount: this.Lines.length,
TotalCost: 0
};
for (var i = 0; i < this.Lines.length; i++) {
var line = this.Lines[i];
orderData.TotalCost += line.PricePerUnit;
// Load to SQL table 'OrderLines'
loadToOrderLines({
OrderId: id(this),
Qty: line.Quantity,
Product: line.Product,
Cost: line.PricePerUnit
});
}
orderData.TotalCost = Math.round(orderData.TotalCost * 100) / 100;
// Load to SQL table 'Orders'
loadToOrders(orderData)"
}
},
// Do not prevent task failover to another node (optional)
PinToMentorNode = false
};
// Define the AddEtlOperation
// ===========================
var operation = new AddEtlOperation<SqlConnectionString>(sqlEtlConfig);
// Execute the operation by passing it to Maintenance.Send
// =======================================================
AddEtlOperationResult result = store.Maintenance.Send(operation);
Add Snowflake ETL task
- Learn about the Snowflake ETL task in the Snowflake ETL task article.
- Learn how to define a connection string for the Snowflake ETL task here: Add a Snowflake connection string
The following example adds a Snowflake ETL task:
// Define the Snowflake ETL task configuration object
// ============================================
var snowflakeEtlConfig = new SnowflakeEtlConfiguration
{
Name = "task-name",
ConnectionStringName = "snowflake-connection-string-name",
SnowflakeTables =
{
new SnowflakeEtlTable {TableName = "Orders", DocumentIdColumn = "Id", InsertOnlyMode = false},
new SnowflakeEtlTable {TableName = "OrderLines", DocumentIdColumn = "OrderId", InsertOnlyMode = false},
},
Transforms =
{
new Transformation
{
Name = "script-name",
Collections = { "Orders" },
Script = @"var orderData = {
Id: id(this),
OrderLinesCount: this.Lines.length,
TotalCost: 0
};
for (var i = 0; i < this.Lines.length; i++) {
var line = this.Lines[i];
orderData.TotalCost += line.PricePerUnit;
// Load to SQL table 'OrderLines'
loadToOrderLines({
OrderId: id(this),
Qty: line.Quantity,
Product: line.Product,
Cost: line.PricePerUnit
});
}
orderData.TotalCost = Math.round(orderData.TotalCost * 100) / 100;
// Load to SQL table 'Orders'
loadToOrders(orderData)"
}
},
// Do not prevent task failover to another node (optional)
PinToMentorNode = false
};
// Define the AddEtlOperation
// ===========================
var operation = new AddEtlOperation<SnowflakeConnectionString>(snowflakeEtlConfig);
// Execute the operation by passing it to Maintenance.Send
// =======================================================
AddEtlOperationResult result = store.Maintenance.Send(operation);
Add OLAP ETL task
- Learn about the OLAP ETL task in the OLAP ETL task article.
- Learn how to define a connection string for the OLAP ETL task here: Add an OLAP connection string
- To manage the OLAP ETL task from Studio, see Studio: OLAP ETL task.
The following example adds an OLAP ETL task:
// Define the OLAP ETL task configuration object
// =============================================
var olapEtlConfig = new OlapEtlConfiguration
{
Name = "task-name",
ConnectionStringName = "olap-connection-string-name",
Transforms =
{
new Transformation
{
Name = "script-name",
Collections = {"Orders"},
Script = @"var orderDate = new Date(this.OrderedAt);
var year = orderDate.getFullYear();
var month = orderDate.getMonth();
var key = new Date(year, month);
loadToOrders(key, {
Company : this.Company,
ShipVia : this.ShipVia
})"
}
}
};
// Define the AddEtlOperation
// ==========================
var operation = new AddEtlOperation<OlapConnectionString>(olapEtlConfig);
// Execute the operation by passing it to Maintenance.Send
// =======================================================
AddEtlOperationResult result = store.Maintenance.Send(operation);
Add Elasticsearch ETL task
- Learn about the Elasticsearch ETL task in the Elasticsearch ETL task article.
- Learn how to define a connection string for the Elasticsearch ETL task here: Add an Elasticsearch connection string
- To manage the Elasticsearch ETL task from Studio, see Studio: Elasticsearch ETL task.
The following example adds an Elasticsearch ETL task:
// Define the Elasticsearch ETL task configuration object
// ======================================================
var elasticsearchEtlConfig = new ElasticSearchEtlConfiguration
{
Name = "task-name",
ConnectionStringName = "elasticsearch-connection-string-name",
ElasticIndexes =
{
// Define Elasticsearch Indexes
new ElasticSearchIndex
{
// Elasticsearch Index name
IndexName = "orders",
// The Elasticsearch document property that will contain the source RavenDB document id.
// Make sure this property is also defined inside the transform script.
DocumentIdProperty = "DocId",
InsertOnlyMode = false
},
new ElasticSearchIndex
{
IndexName = "lines",
DocumentIdProperty = "OrderLinesCount",
// If true, don't send _delete_by_query before appending docs
InsertOnlyMode = true
}
},
Transforms =
{
new Transformation()
{
Collections = { "Orders" },
Script = @"var orderData = {
DocId: id(this),
OrderLinesCount: this.Lines.length,
TotalCost: 0
};
// Write the `orderData` as a document to the Elasticsearch 'orders' index
loadToOrders(orderData);",
Name = "script-name"
}
}
};
// Define the AddEtlOperation
// ==========================
var operation = new AddEtlOperation<ElasticSearchConnectionString>(elasticsearchEtlConfig);
// Execute the operation by passing it to Maintenance.Send
// =======================================================
store.Maintenance.Send(operation);
Add Kafka ETL task
- Learn about the Kafka ETL task in the Kafka ETL task article.
- Learn how to define a connection string for the Kafka ETL task here: Add a Kafka connection string
- To manage the Kafka ETL task from Studio, see Studio: Kafka ETL task.
- Examples showing how to add a Kafka ETL task are available in the Add a Kafka ETL task section.
Add RabbitMQ ETL task
- Learn about the RabbitMQ ETL task in the RabbitMQ ETL task article.
- Learn how to define a connection string for the RabbitMQ ETL task here: Add a RabbitMQ connection string
- To manage the RabbitMQ ETL task from Studio, see Studio: RabbitMQ ETL task.
- Examples showing how to add a RabbitMQ ETL task are available in the Add a RabbitMQ ETL task section.
Add Azure Queue Storage ETL task
- Learn about the Azure Queue Storage ETL task in the Azure Queue Storage ETL task article.
- Learn how to define a connection string for the Azure Queue Storage ETL task here: Add an Azure Queue Storage connection string
- To manage the Azure Queue Storage ETL task from Studio, see Studio: Azure Queue Storage ETL task.
- Examples showing how to add an Azure Queue Storage ETL task are available in the Add a Azure Queue Storage ETL task section.
Add Amazon SQS ETL task
- Learn about the AWS SQS ETL task in the Amazon SQS ETL task article.
- This section shows how to define a connection string to the SQS destination.
- This section shows how to run an ETL task that uses the defined connection string.
- To learn how to manage the task from Studio, see Studio: Amazon SQS ETL Task.
Syntax
public AddEtlOperation(EtlConfiguration<T> configuration)
| Parameter | Type | Description |
|---|---|---|
| configuration | EtlConfiguration<T> | The ETL configuration object where T is the connection string type |