MongoDB Atlas v3.18.0, Sep 12 24

MongoDB Atlas v3.18.0 published on Thursday, Sep 12, 2024 by Pulumi

mongodbatlas.DataLakePipeline

Explore with Pulumi AI

MongoDB Atlas v3.18.0 published on Thursday, Sep 12, 2024 by Pulumi

pulumi/pulumi-mongodbatlas

# Resource: mongodbatlas.DataLakePipeline

mongodbatlas.DataLakePipeline provides a Data Lake Pipeline resource.

NOTE: Groups and projects are synonymous terms. You may find group_id in the official documentation.

Example Usage

S

Coming soon!

Coming soon!

Coming soon!

Coming soon!

package generated_program;

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.mongodbatlas.Project;
import com.pulumi.mongodbatlas.ProjectArgs;
import com.pulumi.mongodbatlas.AdvancedCluster;
import com.pulumi.mongodbatlas.AdvancedClusterArgs;
import com.pulumi.mongodbatlas.inputs.AdvancedClusterReplicationSpecArgs;
import com.pulumi.mongodbatlas.DataLakePipeline;
import com.pulumi.mongodbatlas.DataLakePipelineArgs;
import com.pulumi.mongodbatlas.inputs.DataLakePipelineSinkArgs;
import com.pulumi.mongodbatlas.inputs.DataLakePipelineSourceArgs;
import com.pulumi.mongodbatlas.inputs.DataLakePipelineTransformationArgs;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;

public class App {
    public static void main(String[] args) {
        Pulumi.run(App::stack);
    }

    public static void stack(Context ctx) {
        var projectTest = new Project("projectTest", ProjectArgs.builder()
            .name("NAME OF THE PROJECT")
            .orgId("ORGANIZATION ID")
            .build());

        var automatedBackupTest = new AdvancedCluster("automatedBackupTest", AdvancedClusterArgs.builder()
            .projectId(projectId)
            .name("automated-backup-test")
            .clusterType("REPLICASET")
            .backupEnabled(true)
            .replicationSpecs(AdvancedClusterReplicationSpecArgs.builder()
                .regionConfigs(AdvancedClusterReplicationSpecRegionConfigArgs.builder()
                    .priority(7)
                    .providerName("GCP")
                    .regionName("US_EAST_4")
                    .electableSpecs(AdvancedClusterReplicationSpecRegionConfigElectableSpecsArgs.builder()
                        .instanceSize("M10")
                        .nodeCount(3)
                        .build())
                    .build())
                .build())
            .build());

        var pipeline = new DataLakePipeline("pipeline", DataLakePipelineArgs.builder()
            .projectId(projectTest.projectId())
            .name("DataLakePipelineName")
            .sink(DataLakePipelineSinkArgs.builder()
                .type("DLS")
                .partitionFields(DataLakePipelineSinkPartitionFieldArgs.builder()
                    .name("access")
                    .order(0)
                    .build())
                .build())
            .source(DataLakePipelineSourceArgs.builder()
                .type("ON_DEMAND_CPS")
                .clusterName(automatedBackupTest.name())
                .databaseName("sample_airbnb")
                .collectionName("listingsAndReviews")
                .build())
            .transformations(            
                DataLakePipelineTransformationArgs.builder()
                    .field("test")
                    .type("EXCLUDE")
                    .build(),
                DataLakePipelineTransformationArgs.builder()
                    .field("test22")
                    .type("EXCLUDE")
                    .build())
            .build());

    }
}

resources:
  projectTest:
    type: mongodbatlas:Project
    properties:
      name: NAME OF THE PROJECT
      orgId: ORGANIZATION ID
  automatedBackupTest:
    type: mongodbatlas:AdvancedCluster
    name: automated_backup_test
    properties:
      projectId: ${projectId}
      name: automated-backup-test
      clusterType: REPLICASET
      backupEnabled: true # enable cloud backup snapshots
      replicationSpecs:
        - regionConfigs:
            - priority: 7
              providerName: GCP
              regionName: US_EAST_4
              electableSpecs:
                instanceSize: M10
                nodeCount: 3
  pipeline:
    type: mongodbatlas:DataLakePipeline
    properties:
      projectId: ${projectTest.projectId}
      name: DataLakePipelineName
      sink:
        type: DLS
        partitionFields:
          - name: access
            order: 0
      source:
        type: ON_DEMAND_CPS
        clusterName: ${automatedBackupTest.name}
        databaseName: sample_airbnb
        collectionName: listingsAndReviews
      transformations:
        - field: test
          type: EXCLUDE
        - field: test22
          type: EXCLUDE

Create DataLakePipeline Resource

Resources are created with functions called constructors. To learn more about declaring and configuring resources, see Resources.

Constructor syntax

new DataLakePipeline(name: string, args: DataLakePipelineArgs, opts?: CustomResourceOptions);

@overload
def DataLakePipeline(resource_name: str,
                     args: DataLakePipelineArgs,
                     opts: Optional[ResourceOptions] = None)

@overload
def DataLakePipeline(resource_name: str,
                     opts: Optional[ResourceOptions] = None,
                     project_id: Optional[str] = None,
                     name: Optional[str] = None,
                     sink: Optional[DataLakePipelineSinkArgs] = None,
                     source: Optional[DataLakePipelineSourceArgs] = None,
                     transformations: Optional[Sequence[DataLakePipelineTransformationArgs]] = None)

func NewDataLakePipeline(ctx *Context, name string, args DataLakePipelineArgs, opts ...ResourceOption) (*DataLakePipeline, error)

public DataLakePipeline(string name, DataLakePipelineArgs args, CustomResourceOptions? opts = null)

public DataLakePipeline(String name, DataLakePipelineArgs args)
public DataLakePipeline(String name, DataLakePipelineArgs args, CustomResourceOptions options)

type: mongodbatlas:DataLakePipeline
properties: # The arguments to resource properties.
options: # Bag of options to control resource's behavior.

Parameters

name string: The unique name of the resource.
args DataLakePipelineArgs: The arguments to resource properties.
opts CustomResourceOptions: Bag of options to control resource's behavior.

resource_name str: The unique name of the resource.
args DataLakePipelineArgs: The arguments to resource properties.
opts ResourceOptions: Bag of options to control resource's behavior.

ctx Context: Context object for the current deployment.
name string: The unique name of the resource.
args DataLakePipelineArgs: The arguments to resource properties.
opts ResourceOption: Bag of options to control resource's behavior.

name string: The unique name of the resource.
args DataLakePipelineArgs: The arguments to resource properties.
opts CustomResourceOptions: Bag of options to control resource's behavior.

name String: The unique name of the resource.
args DataLakePipelineArgs: The arguments to resource properties.
options CustomResourceOptions: Bag of options to control resource's behavior.

Constructor example

The following reference example uses placeholder values for all input properties.

var dataLakePipelineResource = new Mongodbatlas.DataLakePipeline("dataLakePipelineResource", new()
{
    ProjectId = "string",
    Name = "string",
    Sink = new Mongodbatlas.Inputs.DataLakePipelineSinkArgs
    {
        PartitionFields = new[]
        {
            new Mongodbatlas.Inputs.DataLakePipelineSinkPartitionFieldArgs
            {
                FieldName = "string",
                Order = 0,
            },
        },
        Provider = "string",
        Region = "string",
        Type = "string",
    },
    Source = new Mongodbatlas.Inputs.DataLakePipelineSourceArgs
    {
        ClusterName = "string",
        CollectionName = "string",
        DatabaseName = "string",
        PolicyItemId = "string",
        ProjectId = "string",
        Type = "string",
    },
    Transformations = new[]
    {
        new Mongodbatlas.Inputs.DataLakePipelineTransformationArgs
        {
            Field = "string",
            Type = "string",
        },
    },
});

example, err := mongodbatlas.NewDataLakePipeline(ctx, "dataLakePipelineResource", &mongodbatlas.DataLakePipelineArgs{
	ProjectId: pulumi.String("string"),
	Name:      pulumi.String("string"),
	Sink: &mongodbatlas.DataLakePipelineSinkArgs{
		PartitionFields: mongodbatlas.DataLakePipelineSinkPartitionFieldArray{
			&mongodbatlas.DataLakePipelineSinkPartitionFieldArgs{
				FieldName: pulumi.String("string"),
				Order:     pulumi.Int(0),
			},
		},
		Provider: pulumi.String("string"),
		Region:   pulumi.String("string"),
		Type:     pulumi.String("string"),
	},
	Source: &mongodbatlas.DataLakePipelineSourceArgs{
		ClusterName:    pulumi.String("string"),
		CollectionName: pulumi.String("string"),
		DatabaseName:   pulumi.String("string"),
		PolicyItemId:   pulumi.String("string"),
		ProjectId:      pulumi.String("string"),
		Type:           pulumi.String("string"),
	},
	Transformations: mongodbatlas.DataLakePipelineTransformationArray{
		&mongodbatlas.DataLakePipelineTransformationArgs{
			Field: pulumi.String("string"),
			Type:  pulumi.String("string"),
		},
	},
})

var dataLakePipelineResource = new DataLakePipeline("dataLakePipelineResource", DataLakePipelineArgs.builder()
    .projectId("string")
    .name("string")
    .sink(DataLakePipelineSinkArgs.builder()
        .partitionFields(DataLakePipelineSinkPartitionFieldArgs.builder()
            .fieldName("string")
            .order(0)
            .build())
        .provider("string")
        .region("string")
        .type("string")
        .build())
    .source(DataLakePipelineSourceArgs.builder()
        .clusterName("string")
        .collectionName("string")
        .databaseName("string")
        .policyItemId("string")
        .projectId("string")
        .type("string")
        .build())
    .transformations(DataLakePipelineTransformationArgs.builder()
        .field("string")
        .type("string")
        .build())
    .build());

data_lake_pipeline_resource = mongodbatlas.DataLakePipeline("dataLakePipelineResource",
    project_id="string",
    name="string",
    sink=mongodbatlas.DataLakePipelineSinkArgs(
        partition_fields=[mongodbatlas.DataLakePipelineSinkPartitionFieldArgs(
            field_name="string",
            order=0,
        )],
        provider="string",
        region="string",
        type="string",
    ),
    source=mongodbatlas.DataLakePipelineSourceArgs(
        cluster_name="string",
        collection_name="string",
        database_name="string",
        policy_item_id="string",
        project_id="string",
        type="string",
    ),
    transformations=[mongodbatlas.DataLakePipelineTransformationArgs(
        field="string",
        type="string",
    )])

const dataLakePipelineResource = new mongodbatlas.DataLakePipeline("dataLakePipelineResource", {
    projectId: "string",
    name: "string",
    sink: {
        partitionFields: [{
            fieldName: "string",
            order: 0,
        }],
        provider: "string",
        region: "string",
        type: "string",
    },
    source: {
        clusterName: "string",
        collectionName: "string",
        databaseName: "string",
        policyItemId: "string",
        projectId: "string",
        type: "string",
    },
    transformations: [{
        field: "string",
        type: "string",
    }],
});

type: mongodbatlas:DataLakePipeline
properties:
    name: string
    projectId: string
    sink:
        partitionFields:
            - fieldName: string
              order: 0
        provider: string
        region: string
        type: string
    source:
        clusterName: string
        collectionName: string
        databaseName: string
        policyItemId: string
        projectId: string
        type: string
    transformations:
        - field: string
          type: string

DataLakePipeline Resource Properties

To learn more about resource properties and how to use them, see Inputs and Outputs in the Architecture and Concepts docs.

Inputs

The DataLakePipeline resource accepts the following input properties:

ProjectId string

The unique ID for the project to create a data lake pipeline.

Name string

Name of the Atlas Data Lake Pipeline.

Sink DataLakePipelineSink

Source DataLakePipelineSource

Transformations List<DataLakePipelineTransformation>

Fields to be excluded for this Data Lake Pipeline.

transformations.#.field - Key in the document.
transformations.#.type - Type of transformation applied during the export of the namespace in a Data Lake Pipeline.

ProjectId string

The unique ID for the project to create a data lake pipeline.

Name string

Name of the Atlas Data Lake Pipeline.

Sink DataLakePipelineSinkArgs

Source DataLakePipelineSourceArgs

Transformations []DataLakePipelineTransformationArgs

Fields to be excluded for this Data Lake Pipeline.

transformations.#.field - Key in the document.
transformations.#.type - Type of transformation applied during the export of the namespace in a Data Lake Pipeline.

projectId String

The unique ID for the project to create a data lake pipeline.

name String

Name of the Atlas Data Lake Pipeline.

sink DataLakePipelineSink

source DataLakePipelineSource

transformations List<DataLakePipelineTransformation>

Fields to be excluded for this Data Lake Pipeline.

transformations.#.field - Key in the document.
transformations.#.type - Type of transformation applied during the export of the namespace in a Data Lake Pipeline.

projectId string

The unique ID for the project to create a data lake pipeline.

name string

Name of the Atlas Data Lake Pipeline.

sink DataLakePipelineSink

source DataLakePipelineSource

transformations DataLakePipelineTransformation[]

Fields to be excluded for this Data Lake Pipeline.

transformations.#.field - Key in the document.
transformations.#.type - Type of transformation applied during the export of the namespace in a Data Lake Pipeline.

project_id str

The unique ID for the project to create a data lake pipeline.

name str

Name of the Atlas Data Lake Pipeline.

sink DataLakePipelineSinkArgs

source DataLakePipelineSourceArgs

transformations Sequence[DataLakePipelineTransformationArgs]

Fields to be excluded for this Data Lake Pipeline.

transformations.#.field - Key in the document.
transformations.#.type - Type of transformation applied during the export of the namespace in a Data Lake Pipeline.

projectId String

The unique ID for the project to create a data lake pipeline.

name String

Name of the Atlas Data Lake Pipeline.

sink Property Map

source Property Map

transformations List<Property Map>

Fields to be excluded for this Data Lake Pipeline.

transformations.#.field - Key in the document.
transformations.#.type - Type of transformation applied during the export of the namespace in a Data Lake Pipeline.

Outputs

All input properties are implicitly available as output properties. Additionally, the DataLakePipeline resource produces the following output properties:

CreatedDate string

Timestamp that indicates when the Data Lake Pipeline was created.

Id string

The provider-assigned unique ID for this managed resource.

IngestionSchedules List<DataLakePipelineIngestionSchedule>

List of backup schedule policy items that you can use as a Data Lake Pipeline source.

ingestion_schedules.#.id - Unique 24-hexadecimal digit string that identifies this backup policy item.
ingestion_schedules.#.frequency_type - Human-readable label that identifies the frequency type associated with the backup policy.
ingestion_schedules.#.frequency_interval - Number that indicates the frequency interval for a set of snapshots.
ingestion_schedules.#.retention_unit - Unit of time in which MongoDB Atlas measures snapshot retention.
ingestion_schedules.#.retention_value - Duration in days, weeks, or months that MongoDB Atlas retains the snapshot.

LastUpdatedDate string

Timestamp that indicates the last time that the Data Lake Pipeline was updated.

Snapshots List<DataLakePipelineSnapshot>

List of backup snapshots that you can use to trigger an on demand pipeline run.

snapshots.#.id - Unique 24-hexadecimal digit string that identifies the snapshot.
snapshots.#.provider - Human-readable label that identifies the cloud provider that stores this snapshot.
snapshots.#.created_at - Date and time when MongoDB Atlas took the snapshot.
snapshots.#.expires_at - Date and time when MongoDB Atlas deletes the snapshot.
snapshots.#.frequency_type - Human-readable label that identifies how often this snapshot triggers.
snapshots.#.master_key - Unique string that identifies the Amazon Web Services (AWS) Key Management Service (KMS) Customer Master Key (CMK) used to encrypt the snapshot.
snapshots.#.mongod_version - Version of the MongoDB host that this snapshot backs up.
snapshots.#.replica_set_name - Human-readable label that identifies the replica set from which MongoDB Atlas took this snapshot.
snapshots.#.type - Human-readable label that categorizes the cluster as a replica set or sharded cluster.
snapshots.#.snapshot_type - Human-readable label that identifies when this snapshot triggers.
snapshots.#.status - Human-readable label that indicates the stage of the backup process for this snapshot.
snapshots.#.size - List of backup snapshots that you can use to trigger an on demand pipeline run.
snapshots.#.copy_region - List that identifies the regions to which MongoDB Atlas copies the snapshot.
snapshots.#.policies - List that contains unique identifiers for the policy items.

State string

State of this Data Lake Pipeline.

CreatedDate string

Timestamp that indicates when the Data Lake Pipeline was created.

Id string

The provider-assigned unique ID for this managed resource.

IngestionSchedules []DataLakePipelineIngestionSchedule

List of backup schedule policy items that you can use as a Data Lake Pipeline source.

ingestion_schedules.#.id - Unique 24-hexadecimal digit string that identifies this backup policy item.
ingestion_schedules.#.frequency_type - Human-readable label that identifies the frequency type associated with the backup policy.
ingestion_schedules.#.frequency_interval - Number that indicates the frequency interval for a set of snapshots.
ingestion_schedules.#.retention_unit - Unit of time in which MongoDB Atlas measures snapshot retention.
ingestion_schedules.#.retention_value - Duration in days, weeks, or months that MongoDB Atlas retains the snapshot.

LastUpdatedDate string

Timestamp that indicates the last time that the Data Lake Pipeline was updated.

Snapshots []DataLakePipelineSnapshot

List of backup snapshots that you can use to trigger an on demand pipeline run.

snapshots.#.id - Unique 24-hexadecimal digit string that identifies the snapshot.
snapshots.#.provider - Human-readable label that identifies the cloud provider that stores this snapshot.
snapshots.#.created_at - Date and time when MongoDB Atlas took the snapshot.
snapshots.#.expires_at - Date and time when MongoDB Atlas deletes the snapshot.
snapshots.#.frequency_type - Human-readable label that identifies how often this snapshot triggers.
snapshots.#.master_key - Unique string that identifies the Amazon Web Services (AWS) Key Management Service (KMS) Customer Master Key (CMK) used to encrypt the snapshot.
snapshots.#.mongod_version - Version of the MongoDB host that this snapshot backs up.
snapshots.#.replica_set_name - Human-readable label that identifies the replica set from which MongoDB Atlas took this snapshot.
snapshots.#.type - Human-readable label that categorizes the cluster as a replica set or sharded cluster.
snapshots.#.snapshot_type - Human-readable label that identifies when this snapshot triggers.
snapshots.#.status - Human-readable label that indicates the stage of the backup process for this snapshot.
snapshots.#.size - List of backup snapshots that you can use to trigger an on demand pipeline run.
snapshots.#.copy_region - List that identifies the regions to which MongoDB Atlas copies the snapshot.
snapshots.#.policies - List that contains unique identifiers for the policy items.

State string

State of this Data Lake Pipeline.

createdDate String

Timestamp that indicates when the Data Lake Pipeline was created.

id String

The provider-assigned unique ID for this managed resource.

ingestionSchedules List<DataLakePipelineIngestionSchedule>

List of backup schedule policy items that you can use as a Data Lake Pipeline source.

ingestion_schedules.#.id - Unique 24-hexadecimal digit string that identifies this backup policy item.
ingestion_schedules.#.frequency_type - Human-readable label that identifies the frequency type associated with the backup policy.
ingestion_schedules.#.frequency_interval - Number that indicates the frequency interval for a set of snapshots.
ingestion_schedules.#.retention_unit - Unit of time in which MongoDB Atlas measures snapshot retention.
ingestion_schedules.#.retention_value - Duration in days, weeks, or months that MongoDB Atlas retains the snapshot.

lastUpdatedDate String

Timestamp that indicates the last time that the Data Lake Pipeline was updated.

snapshots List<DataLakePipelineSnapshot>

List of backup snapshots that you can use to trigger an on demand pipeline run.

snapshots.#.id - Unique 24-hexadecimal digit string that identifies the snapshot.
snapshots.#.provider - Human-readable label that identifies the cloud provider that stores this snapshot.
snapshots.#.created_at - Date and time when MongoDB Atlas took the snapshot.
snapshots.#.expires_at - Date and time when MongoDB Atlas deletes the snapshot.
snapshots.#.frequency_type - Human-readable label that identifies how often this snapshot triggers.
snapshots.#.master_key - Unique string that identifies the Amazon Web Services (AWS) Key Management Service (KMS) Customer Master Key (CMK) used to encrypt the snapshot.
snapshots.#.mongod_version - Version of the MongoDB host that this snapshot backs up.
snapshots.#.replica_set_name - Human-readable label that identifies the replica set from which MongoDB Atlas took this snapshot.
snapshots.#.type - Human-readable label that categorizes the cluster as a replica set or sharded cluster.
snapshots.#.snapshot_type - Human-readable label that identifies when this snapshot triggers.
snapshots.#.status - Human-readable label that indicates the stage of the backup process for this snapshot.
snapshots.#.size - List of backup snapshots that you can use to trigger an on demand pipeline run.
snapshots.#.copy_region - List that identifies the regions to which MongoDB Atlas copies the snapshot.
snapshots.#.policies - List that contains unique identifiers for the policy items.

state String

State of this Data Lake Pipeline.

createdDate string

Timestamp that indicates when the Data Lake Pipeline was created.

id string

The provider-assigned unique ID for this managed resource.

ingestionSchedules DataLakePipelineIngestionSchedule[]

List of backup schedule policy items that you can use as a Data Lake Pipeline source.

ingestion_schedules.#.id - Unique 24-hexadecimal digit string that identifies this backup policy item.
ingestion_schedules.#.frequency_type - Human-readable label that identifies the frequency type associated with the backup policy.
ingestion_schedules.#.frequency_interval - Number that indicates the frequency interval for a set of snapshots.
ingestion_schedules.#.retention_unit - Unit of time in which MongoDB Atlas measures snapshot retention.
ingestion_schedules.#.retention_value - Duration in days, weeks, or months that MongoDB Atlas retains the snapshot.

lastUpdatedDate string

Timestamp that indicates the last time that the Data Lake Pipeline was updated.

snapshots DataLakePipelineSnapshot[]

List of backup snapshots that you can use to trigger an on demand pipeline run.

snapshots.#.id - Unique 24-hexadecimal digit string that identifies the snapshot.
snapshots.#.provider - Human-readable label that identifies the cloud provider that stores this snapshot.
snapshots.#.created_at - Date and time when MongoDB Atlas took the snapshot.
snapshots.#.expires_at - Date and time when MongoDB Atlas deletes the snapshot.
snapshots.#.frequency_type - Human-readable label that identifies how often this snapshot triggers.
snapshots.#.master_key - Unique string that identifies the Amazon Web Services (AWS) Key Management Service (KMS) Customer Master Key (CMK) used to encrypt the snapshot.
snapshots.#.mongod_version - Version of the MongoDB host that this snapshot backs up.
snapshots.#.replica_set_name - Human-readable label that identifies the replica set from which MongoDB Atlas took this snapshot.
snapshots.#.type - Human-readable label that categorizes the cluster as a replica set or sharded cluster.
snapshots.#.snapshot_type - Human-readable label that identifies when this snapshot triggers.
snapshots.#.status - Human-readable label that indicates the stage of the backup process for this snapshot.
snapshots.#.size - List of backup snapshots that you can use to trigger an on demand pipeline run.
snapshots.#.copy_region - List that identifies the regions to which MongoDB Atlas copies the snapshot.
snapshots.#.policies - List that contains unique identifiers for the policy items.

state string

State of this Data Lake Pipeline.

created_date str

Timestamp that indicates when the Data Lake Pipeline was created.

id str

The provider-assigned unique ID for this managed resource.

ingestion_schedules Sequence[DataLakePipelineIngestionSchedule]

List of backup schedule policy items that you can use as a Data Lake Pipeline source.

ingestion_schedules.#.id - Unique 24-hexadecimal digit string that identifies this backup policy item.
ingestion_schedules.#.frequency_type - Human-readable label that identifies the frequency type associated with the backup policy.
ingestion_schedules.#.frequency_interval - Number that indicates the frequency interval for a set of snapshots.
ingestion_schedules.#.retention_unit - Unit of time in which MongoDB Atlas measures snapshot retention.
ingestion_schedules.#.retention_value - Duration in days, weeks, or months that MongoDB Atlas retains the snapshot.

last_updated_date str

Timestamp that indicates the last time that the Data Lake Pipeline was updated.

snapshots Sequence[DataLakePipelineSnapshot]

List of backup snapshots that you can use to trigger an on demand pipeline run.

snapshots.#.id - Unique 24-hexadecimal digit string that identifies the snapshot.
snapshots.#.provider - Human-readable label that identifies the cloud provider that stores this snapshot.
snapshots.#.created_at - Date and time when MongoDB Atlas took the snapshot.
snapshots.#.expires_at - Date and time when MongoDB Atlas deletes the snapshot.
snapshots.#.frequency_type - Human-readable label that identifies how often this snapshot triggers.
snapshots.#.master_key - Unique string that identifies the Amazon Web Services (AWS) Key Management Service (KMS) Customer Master Key (CMK) used to encrypt the snapshot.
snapshots.#.mongod_version - Version of the MongoDB host that this snapshot backs up.
snapshots.#.replica_set_name - Human-readable label that identifies the replica set from which MongoDB Atlas took this snapshot.
snapshots.#.type - Human-readable label that categorizes the cluster as a replica set or sharded cluster.
snapshots.#.snapshot_type - Human-readable label that identifies when this snapshot triggers.
snapshots.#.status - Human-readable label that indicates the stage of the backup process for this snapshot.
snapshots.#.size - List of backup snapshots that you can use to trigger an on demand pipeline run.
snapshots.#.copy_region - List that identifies the regions to which MongoDB Atlas copies the snapshot.
snapshots.#.policies - List that contains unique identifiers for the policy items.

state str

State of this Data Lake Pipeline.

createdDate String

Timestamp that indicates when the Data Lake Pipeline was created.

id String

The provider-assigned unique ID for this managed resource.

ingestionSchedules List<Property Map>

List of backup schedule policy items that you can use as a Data Lake Pipeline source.

ingestion_schedules.#.id - Unique 24-hexadecimal digit string that identifies this backup policy item.
ingestion_schedules.#.frequency_type - Human-readable label that identifies the frequency type associated with the backup policy.
ingestion_schedules.#.frequency_interval - Number that indicates the frequency interval for a set of snapshots.
ingestion_schedules.#.retention_unit - Unit of time in which MongoDB Atlas measures snapshot retention.
ingestion_schedules.#.retention_value - Duration in days, weeks, or months that MongoDB Atlas retains the snapshot.

lastUpdatedDate String

Timestamp that indicates the last time that the Data Lake Pipeline was updated.

snapshots List<Property Map>

List of backup snapshots that you can use to trigger an on demand pipeline run.

snapshots.#.id - Unique 24-hexadecimal digit string that identifies the snapshot.
snapshots.#.provider - Human-readable label that identifies the cloud provider that stores this snapshot.
snapshots.#.created_at - Date and time when MongoDB Atlas took the snapshot.
snapshots.#.expires_at - Date and time when MongoDB Atlas deletes the snapshot.
snapshots.#.frequency_type - Human-readable label that identifies how often this snapshot triggers.
snapshots.#.master_key - Unique string that identifies the Amazon Web Services (AWS) Key Management Service (KMS) Customer Master Key (CMK) used to encrypt the snapshot.
snapshots.#.mongod_version - Version of the MongoDB host that this snapshot backs up.
snapshots.#.replica_set_name - Human-readable label that identifies the replica set from which MongoDB Atlas took this snapshot.
snapshots.#.type - Human-readable label that categorizes the cluster as a replica set or sharded cluster.
snapshots.#.snapshot_type - Human-readable label that identifies when this snapshot triggers.
snapshots.#.status - Human-readable label that indicates the stage of the backup process for this snapshot.
snapshots.#.size - List of backup snapshots that you can use to trigger an on demand pipeline run.
snapshots.#.copy_region - List that identifies the regions to which MongoDB Atlas copies the snapshot.
snapshots.#.policies - List that contains unique identifiers for the policy items.

state String

State of this Data Lake Pipeline.

Look up Existing DataLakePipeline Resource

Get an existing DataLakePipeline resource’s state with the given name, ID, and optional extra properties used to qualify the lookup.

public static get(name: string, id: Input<ID>, state?: DataLakePipelineState, opts?: CustomResourceOptions): DataLakePipeline

@staticmethod
def get(resource_name: str,
        id: str,
        opts: Optional[ResourceOptions] = None,
        created_date: Optional[str] = None,
        ingestion_schedules: Optional[Sequence[DataLakePipelineIngestionScheduleArgs]] = None,
        last_updated_date: Optional[str] = None,
        name: Optional[str] = None,
        project_id: Optional[str] = None,
        sink: Optional[DataLakePipelineSinkArgs] = None,
        snapshots: Optional[Sequence[DataLakePipelineSnapshotArgs]] = None,
        source: Optional[DataLakePipelineSourceArgs] = None,
        state: Optional[str] = None,
        transformations: Optional[Sequence[DataLakePipelineTransformationArgs]] = None) -> DataLakePipeline

func GetDataLakePipeline(ctx *Context, name string, id IDInput, state *DataLakePipelineState, opts ...ResourceOption) (*DataLakePipeline, error)

public static DataLakePipeline Get(string name, Input<string> id, DataLakePipelineState? state, CustomResourceOptions? opts = null)

public static DataLakePipeline get(String name, Output<String> id, DataLakePipelineState state, CustomResourceOptions options)

Resource lookup is not supported in YAML

name: The unique name of the resulting resource.
id: The unique provider ID of the resource to lookup.
state: Any extra arguments used during the lookup.
opts: A bag of options that control this resource's behavior.

resource_name: The unique name of the resulting resource.
id: The unique provider ID of the resource to lookup.

name: The unique name of the resulting resource.
id: The unique provider ID of the resource to lookup.
state: Any extra arguments used during the lookup.
opts: A bag of options that control this resource's behavior.

name: The unique name of the resulting resource.
id: The unique provider ID of the resource to lookup.
state: Any extra arguments used during the lookup.
opts: A bag of options that control this resource's behavior.

name: The unique name of the resulting resource.
id: The unique provider ID of the resource to lookup.
state: Any extra arguments used during the lookup.
opts: A bag of options that control this resource's behavior.

The following state arguments are supported:

CreatedDate string

Timestamp that indicates when the Data Lake Pipeline was created.

IngestionSchedules List<DataLakePipelineIngestionSchedule>

List of backup schedule policy items that you can use as a Data Lake Pipeline source.

ingestion_schedules.#.id - Unique 24-hexadecimal digit string that identifies this backup policy item.
ingestion_schedules.#.frequency_type - Human-readable label that identifies the frequency type associated with the backup policy.
ingestion_schedules.#.frequency_interval - Number that indicates the frequency interval for a set of snapshots.
ingestion_schedules.#.retention_unit - Unit of time in which MongoDB Atlas measures snapshot retention.
ingestion_schedules.#.retention_value - Duration in days, weeks, or months that MongoDB Atlas retains the snapshot.

LastUpdatedDate string

Timestamp that indicates the last time that the Data Lake Pipeline was updated.

Name string

Name of the Atlas Data Lake Pipeline.

ProjectId string

The unique ID for the project to create a data lake pipeline.

Sink DataLakePipelineSink

Snapshots List<DataLakePipelineSnapshot>

List of backup snapshots that you can use to trigger an on demand pipeline run.

snapshots.#.id - Unique 24-hexadecimal digit string that identifies the snapshot.
snapshots.#.provider - Human-readable label that identifies the cloud provider that stores this snapshot.
snapshots.#.created_at - Date and time when MongoDB Atlas took the snapshot.
snapshots.#.expires_at - Date and time when MongoDB Atlas deletes the snapshot.
snapshots.#.frequency_type - Human-readable label that identifies how often this snapshot triggers.
snapshots.#.master_key - Unique string that identifies the Amazon Web Services (AWS) Key Management Service (KMS) Customer Master Key (CMK) used to encrypt the snapshot.
snapshots.#.mongod_version - Version of the MongoDB host that this snapshot backs up.
snapshots.#.replica_set_name - Human-readable label that identifies the replica set from which MongoDB Atlas took this snapshot.
snapshots.#.type - Human-readable label that categorizes the cluster as a replica set or sharded cluster.
snapshots.#.snapshot_type - Human-readable label that identifies when this snapshot triggers.
snapshots.#.status - Human-readable label that indicates the stage of the backup process for this snapshot.
snapshots.#.size - List of backup snapshots that you can use to trigger an on demand pipeline run.
snapshots.#.copy_region - List that identifies the regions to which MongoDB Atlas copies the snapshot.
snapshots.#.policies - List that contains unique identifiers for the policy items.

Source DataLakePipelineSource

State string

State of this Data Lake Pipeline.

Transformations List<DataLakePipelineTransformation>

Fields to be excluded for this Data Lake Pipeline.

transformations.#.field - Key in the document.
transformations.#.type - Type of transformation applied during the export of the namespace in a Data Lake Pipeline.

CreatedDate string

Timestamp that indicates when the Data Lake Pipeline was created.

IngestionSchedules []DataLakePipelineIngestionScheduleArgs

List of backup schedule policy items that you can use as a Data Lake Pipeline source.

ingestion_schedules.#.id - Unique 24-hexadecimal digit string that identifies this backup policy item.
ingestion_schedules.#.frequency_type - Human-readable label that identifies the frequency type associated with the backup policy.
ingestion_schedules.#.frequency_interval - Number that indicates the frequency interval for a set of snapshots.
ingestion_schedules.#.retention_unit - Unit of time in which MongoDB Atlas measures snapshot retention.
ingestion_schedules.#.retention_value - Duration in days, weeks, or months that MongoDB Atlas retains the snapshot.

LastUpdatedDate string

Timestamp that indicates the last time that the Data Lake Pipeline was updated.

Name string

Name of the Atlas Data Lake Pipeline.

ProjectId string

The unique ID for the project to create a data lake pipeline.

Sink DataLakePipelineSinkArgs

Snapshots []DataLakePipelineSnapshotArgs

List of backup snapshots that you can use to trigger an on demand pipeline run.

snapshots.#.id - Unique 24-hexadecimal digit string that identifies the snapshot.
snapshots.#.provider - Human-readable label that identifies the cloud provider that stores this snapshot.
snapshots.#.created_at - Date and time when MongoDB Atlas took the snapshot.
snapshots.#.expires_at - Date and time when MongoDB Atlas deletes the snapshot.
snapshots.#.frequency_type - Human-readable label that identifies how often this snapshot triggers.
snapshots.#.master_key - Unique string that identifies the Amazon Web Services (AWS) Key Management Service (KMS) Customer Master Key (CMK) used to encrypt the snapshot.
snapshots.#.mongod_version - Version of the MongoDB host that this snapshot backs up.
snapshots.#.replica_set_name - Human-readable label that identifies the replica set from which MongoDB Atlas took this snapshot.
snapshots.#.type - Human-readable label that categorizes the cluster as a replica set or sharded cluster.
snapshots.#.snapshot_type - Human-readable label that identifies when this snapshot triggers.
snapshots.#.status - Human-readable label that indicates the stage of the backup process for this snapshot.
snapshots.#.size - List of backup snapshots that you can use to trigger an on demand pipeline run.
snapshots.#.copy_region - List that identifies the regions to which MongoDB Atlas copies the snapshot.
snapshots.#.policies - List that contains unique identifiers for the policy items.

Source DataLakePipelineSourceArgs

State string

State of this Data Lake Pipeline.

Transformations []DataLakePipelineTransformationArgs

Fields to be excluded for this Data Lake Pipeline.

transformations.#.field - Key in the document.
transformations.#.type - Type of transformation applied during the export of the namespace in a Data Lake Pipeline.

createdDate String

Timestamp that indicates when the Data Lake Pipeline was created.

ingestionSchedules List<DataLakePipelineIngestionSchedule>

List of backup schedule policy items that you can use as a Data Lake Pipeline source.

ingestion_schedules.#.id - Unique 24-hexadecimal digit string that identifies this backup policy item.
ingestion_schedules.#.frequency_type - Human-readable label that identifies the frequency type associated with the backup policy.
ingestion_schedules.#.frequency_interval - Number that indicates the frequency interval for a set of snapshots.
ingestion_schedules.#.retention_unit - Unit of time in which MongoDB Atlas measures snapshot retention.
ingestion_schedules.#.retention_value - Duration in days, weeks, or months that MongoDB Atlas retains the snapshot.

lastUpdatedDate String

Timestamp that indicates the last time that the Data Lake Pipeline was updated.

name String

Name of the Atlas Data Lake Pipeline.

projectId String

The unique ID for the project to create a data lake pipeline.

sink DataLakePipelineSink

snapshots List<DataLakePipelineSnapshot>

List of backup snapshots that you can use to trigger an on demand pipeline run.

snapshots.#.id - Unique 24-hexadecimal digit string that identifies the snapshot.
snapshots.#.provider - Human-readable label that identifies the cloud provider that stores this snapshot.
snapshots.#.created_at - Date and time when MongoDB Atlas took the snapshot.
snapshots.#.expires_at - Date and time when MongoDB Atlas deletes the snapshot.
snapshots.#.frequency_type - Human-readable label that identifies how often this snapshot triggers.
snapshots.#.master_key - Unique string that identifies the Amazon Web Services (AWS) Key Management Service (KMS) Customer Master Key (CMK) used to encrypt the snapshot.
snapshots.#.mongod_version - Version of the MongoDB host that this snapshot backs up.
snapshots.#.replica_set_name - Human-readable label that identifies the replica set from which MongoDB Atlas took this snapshot.
snapshots.#.type - Human-readable label that categorizes the cluster as a replica set or sharded cluster.
snapshots.#.snapshot_type - Human-readable label that identifies when this snapshot triggers.
snapshots.#.status - Human-readable label that indicates the stage of the backup process for this snapshot.
snapshots.#.size - List of backup snapshots that you can use to trigger an on demand pipeline run.
snapshots.#.copy_region - List that identifies the regions to which MongoDB Atlas copies the snapshot.
snapshots.#.policies - List that contains unique identifiers for the policy items.

source DataLakePipelineSource

state String

State of this Data Lake Pipeline.

transformations List<DataLakePipelineTransformation>

Fields to be excluded for this Data Lake Pipeline.

transformations.#.field - Key in the document.
transformations.#.type - Type of transformation applied during the export of the namespace in a Data Lake Pipeline.

createdDate string

Timestamp that indicates when the Data Lake Pipeline was created.

ingestionSchedules DataLakePipelineIngestionSchedule[]

List of backup schedule policy items that you can use as a Data Lake Pipeline source.

ingestion_schedules.#.id - Unique 24-hexadecimal digit string that identifies this backup policy item.
ingestion_schedules.#.frequency_type - Human-readable label that identifies the frequency type associated with the backup policy.
ingestion_schedules.#.frequency_interval - Number that indicates the frequency interval for a set of snapshots.
ingestion_schedules.#.retention_unit - Unit of time in which MongoDB Atlas measures snapshot retention.
ingestion_schedules.#.retention_value - Duration in days, weeks, or months that MongoDB Atlas retains the snapshot.

lastUpdatedDate string

Timestamp that indicates the last time that the Data Lake Pipeline was updated.

name string

Name of the Atlas Data Lake Pipeline.

projectId string

The unique ID for the project to create a data lake pipeline.

sink DataLakePipelineSink

snapshots DataLakePipelineSnapshot[]

List of backup snapshots that you can use to trigger an on demand pipeline run.

snapshots.#.id - Unique 24-hexadecimal digit string that identifies the snapshot.
snapshots.#.provider - Human-readable label that identifies the cloud provider that stores this snapshot.
snapshots.#.created_at - Date and time when MongoDB Atlas took the snapshot.
snapshots.#.expires_at - Date and time when MongoDB Atlas deletes the snapshot.
snapshots.#.frequency_type - Human-readable label that identifies how often this snapshot triggers.
snapshots.#.master_key - Unique string that identifies the Amazon Web Services (AWS) Key Management Service (KMS) Customer Master Key (CMK) used to encrypt the snapshot.
snapshots.#.mongod_version - Version of the MongoDB host that this snapshot backs up.
snapshots.#.replica_set_name - Human-readable label that identifies the replica set from which MongoDB Atlas took this snapshot.
snapshots.#.type - Human-readable label that categorizes the cluster as a replica set or sharded cluster.
snapshots.#.snapshot_type - Human-readable label that identifies when this snapshot triggers.
snapshots.#.status - Human-readable label that indicates the stage of the backup process for this snapshot.
snapshots.#.size - List of backup snapshots that you can use to trigger an on demand pipeline run.
snapshots.#.copy_region - List that identifies the regions to which MongoDB Atlas copies the snapshot.
snapshots.#.policies - List that contains unique identifiers for the policy items.

source DataLakePipelineSource

state string

State of this Data Lake Pipeline.

transformations DataLakePipelineTransformation[]

Fields to be excluded for this Data Lake Pipeline.

transformations.#.field - Key in the document.
transformations.#.type - Type of transformation applied during the export of the namespace in a Data Lake Pipeline.

created_date str

Timestamp that indicates when the Data Lake Pipeline was created.

ingestion_schedules Sequence[DataLakePipelineIngestionScheduleArgs]

List of backup schedule policy items that you can use as a Data Lake Pipeline source.

ingestion_schedules.#.id - Unique 24-hexadecimal digit string that identifies this backup policy item.
ingestion_schedules.#.frequency_type - Human-readable label that identifies the frequency type associated with the backup policy.
ingestion_schedules.#.frequency_interval - Number that indicates the frequency interval for a set of snapshots.
ingestion_schedules.#.retention_unit - Unit of time in which MongoDB Atlas measures snapshot retention.
ingestion_schedules.#.retention_value - Duration in days, weeks, or months that MongoDB Atlas retains the snapshot.

last_updated_date str

Timestamp that indicates the last time that the Data Lake Pipeline was updated.

name str

Name of the Atlas Data Lake Pipeline.

project_id str

The unique ID for the project to create a data lake pipeline.

sink DataLakePipelineSinkArgs

snapshots Sequence[DataLakePipelineSnapshotArgs]

List of backup snapshots that you can use to trigger an on demand pipeline run.

snapshots.#.id - Unique 24-hexadecimal digit string that identifies the snapshot.
snapshots.#.provider - Human-readable label that identifies the cloud provider that stores this snapshot.
snapshots.#.created_at - Date and time when MongoDB Atlas took the snapshot.
snapshots.#.expires_at - Date and time when MongoDB Atlas deletes the snapshot.
snapshots.#.frequency_type - Human-readable label that identifies how often this snapshot triggers.
snapshots.#.master_key - Unique string that identifies the Amazon Web Services (AWS) Key Management Service (KMS) Customer Master Key (CMK) used to encrypt the snapshot.
snapshots.#.mongod_version - Version of the MongoDB host that this snapshot backs up.
snapshots.#.replica_set_name - Human-readable label that identifies the replica set from which MongoDB Atlas took this snapshot.
snapshots.#.type - Human-readable label that categorizes the cluster as a replica set or sharded cluster.
snapshots.#.snapshot_type - Human-readable label that identifies when this snapshot triggers.
snapshots.#.status - Human-readable label that indicates the stage of the backup process for this snapshot.
snapshots.#.size - List of backup snapshots that you can use to trigger an on demand pipeline run.
snapshots.#.copy_region - List that identifies the regions to which MongoDB Atlas copies the snapshot.
snapshots.#.policies - List that contains unique identifiers for the policy items.

source DataLakePipelineSourceArgs

state str

State of this Data Lake Pipeline.

transformations Sequence[DataLakePipelineTransformationArgs]

Fields to be excluded for this Data Lake Pipeline.

transformations.#.field - Key in the document.
transformations.#.type - Type of transformation applied during the export of the namespace in a Data Lake Pipeline.

createdDate String

Timestamp that indicates when the Data Lake Pipeline was created.

ingestionSchedules List<Property Map>

List of backup schedule policy items that you can use as a Data Lake Pipeline source.

ingestion_schedules.#.id - Unique 24-hexadecimal digit string that identifies this backup policy item.
ingestion_schedules.#.frequency_type - Human-readable label that identifies the frequency type associated with the backup policy.
ingestion_schedules.#.frequency_interval - Number that indicates the frequency interval for a set of snapshots.
ingestion_schedules.#.retention_unit - Unit of time in which MongoDB Atlas measures snapshot retention.
ingestion_schedules.#.retention_value - Duration in days, weeks, or months that MongoDB Atlas retains the snapshot.

lastUpdatedDate String

Timestamp that indicates the last time that the Data Lake Pipeline was updated.

name String

Name of the Atlas Data Lake Pipeline.

projectId String

The unique ID for the project to create a data lake pipeline.

sink Property Map

snapshots List<Property Map>

List of backup snapshots that you can use to trigger an on demand pipeline run.

snapshots.#.id - Unique 24-hexadecimal digit string that identifies the snapshot.
snapshots.#.provider - Human-readable label that identifies the cloud provider that stores this snapshot.
snapshots.#.created_at - Date and time when MongoDB Atlas took the snapshot.
snapshots.#.expires_at - Date and time when MongoDB Atlas deletes the snapshot.
snapshots.#.frequency_type - Human-readable label that identifies how often this snapshot triggers.
snapshots.#.master_key - Unique string that identifies the Amazon Web Services (AWS) Key Management Service (KMS) Customer Master Key (CMK) used to encrypt the snapshot.
snapshots.#.mongod_version - Version of the MongoDB host that this snapshot backs up.
snapshots.#.replica_set_name - Human-readable label that identifies the replica set from which MongoDB Atlas took this snapshot.
snapshots.#.type - Human-readable label that categorizes the cluster as a replica set or sharded cluster.
snapshots.#.snapshot_type - Human-readable label that identifies when this snapshot triggers.
snapshots.#.status - Human-readable label that indicates the stage of the backup process for this snapshot.
snapshots.#.size - List of backup snapshots that you can use to trigger an on demand pipeline run.
snapshots.#.copy_region - List that identifies the regions to which MongoDB Atlas copies the snapshot.
snapshots.#.policies - List that contains unique identifiers for the policy items.

source Property Map

state String

State of this Data Lake Pipeline.

transformations List<Property Map>

Fields to be excluded for this Data Lake Pipeline.

transformations.#.field - Key in the document.
transformations.#.type - Type of transformation applied during the export of the namespace in a Data Lake Pipeline.

Supporting Types

DataLakePipelineIngestionSchedule, DataLakePipelineIngestionScheduleArgs

FrequencyInterval int
FrequencyType string
Id string: Unique 24-hexadecimal digit string that identifies the Data Lake Pipeline.
RetentionUnit string
RetentionValue int

FrequencyInterval int
FrequencyType string
Id string: Unique 24-hexadecimal digit string that identifies the Data Lake Pipeline.
RetentionUnit string
RetentionValue int

frequencyInterval Integer
frequencyType String
id String: Unique 24-hexadecimal digit string that identifies the Data Lake Pipeline.
retentionUnit String
retentionValue Integer

frequencyInterval number
frequencyType string
id string: Unique 24-hexadecimal digit string that identifies the Data Lake Pipeline.
retentionUnit string
retentionValue number

frequency_interval int
frequency_type str
id str: Unique 24-hexadecimal digit string that identifies the Data Lake Pipeline.
retention_unit str
retention_value int

frequencyInterval Number
frequencyType String
id String: Unique 24-hexadecimal digit string that identifies the Data Lake Pipeline.
retentionUnit String
retentionValue Number

DataLakePipelineSink, DataLakePipelineSinkArgs

PartitionFields List<DataLakePipelineSinkPartitionField>

Ordered fields used to physically organize data in the destination.

partition_fields.#.field_name - Human-readable label that identifies the field name used to partition data.
partition_fields.#.order - Sequence in which MongoDB Atlas slices the collection data to create partitions. The resource expresses this sequence starting with zero.

Provider string

Target cloud provider for this Data Lake Pipeline.

Region string

Target cloud provider region for this Data Lake Pipeline. Supported cloud provider regions.

Type string

Type of ingestion source of this Data Lake Pipeline.

PartitionFields []DataLakePipelineSinkPartitionField

Ordered fields used to physically organize data in the destination.

partition_fields.#.field_name - Human-readable label that identifies the field name used to partition data.
partition_fields.#.order - Sequence in which MongoDB Atlas slices the collection data to create partitions. The resource expresses this sequence starting with zero.

Provider string

Target cloud provider for this Data Lake Pipeline.

Region string

Target cloud provider region for this Data Lake Pipeline. Supported cloud provider regions.

Type string

Type of ingestion source of this Data Lake Pipeline.

partitionFields List<DataLakePipelineSinkPartitionField>

Ordered fields used to physically organize data in the destination.

partition_fields.#.field_name - Human-readable label that identifies the field name used to partition data.
partition_fields.#.order - Sequence in which MongoDB Atlas slices the collection data to create partitions. The resource expresses this sequence starting with zero.

provider String

Target cloud provider for this Data Lake Pipeline.

region String

Target cloud provider region for this Data Lake Pipeline. Supported cloud provider regions.

type String

Type of ingestion source of this Data Lake Pipeline.

partitionFields DataLakePipelineSinkPartitionField[]

Ordered fields used to physically organize data in the destination.

partition_fields.#.field_name - Human-readable label that identifies the field name used to partition data.
partition_fields.#.order - Sequence in which MongoDB Atlas slices the collection data to create partitions. The resource expresses this sequence starting with zero.

provider string

Target cloud provider for this Data Lake Pipeline.

region string

Target cloud provider region for this Data Lake Pipeline. Supported cloud provider regions.

type string

Type of ingestion source of this Data Lake Pipeline.

partition_fields Sequence[DataLakePipelineSinkPartitionField]

Ordered fields used to physically organize data in the destination.

partition_fields.#.field_name - Human-readable label that identifies the field name used to partition data.
partition_fields.#.order - Sequence in which MongoDB Atlas slices the collection data to create partitions. The resource expresses this sequence starting with zero.

provider str

Target cloud provider for this Data Lake Pipeline.

region str

Target cloud provider region for this Data Lake Pipeline. Supported cloud provider regions.

type str

Type of ingestion source of this Data Lake Pipeline.

partitionFields List<Property Map>

Ordered fields used to physically organize data in the destination.

partition_fields.#.field_name - Human-readable label that identifies the field name used to partition data.
partition_fields.#.order - Sequence in which MongoDB Atlas slices the collection data to create partitions. The resource expresses this sequence starting with zero.

provider String

Target cloud provider for this Data Lake Pipeline.

region String

Target cloud provider region for this Data Lake Pipeline. Supported cloud provider regions.

type String

Type of ingestion source of this Data Lake Pipeline.

DataLakePipelineSinkPartitionField, DataLakePipelineSinkPartitionFieldArgs

FieldName string
Order int

FieldName string
Order int

fieldName String
order Integer

fieldName string
order number

field_name str
order int

fieldName String
order Number

DataLakePipelineSnapshot, DataLakePipelineSnapshotArgs

CopyRegion string
CreatedAt string
ExpiresAt string
FrequencyYype string
Id string: Unique 24-hexadecimal digit string that identifies the Data Lake Pipeline.
MasterKey string
MongodVersion string
Policies List<string>
Provider string: Target cloud provider for this Data Lake Pipeline.
ReplicaSetName string
Size int
SnapshotType string
Status string
Type string: Type of ingestion source of this Data Lake Pipeline.

CopyRegion string
CreatedAt string
ExpiresAt string
FrequencyYype string
Id string: Unique 24-hexadecimal digit string that identifies the Data Lake Pipeline.
MasterKey string
MongodVersion string
Policies []string
Provider string: Target cloud provider for this Data Lake Pipeline.
ReplicaSetName string
Size int
SnapshotType string
Status string
Type string: Type of ingestion source of this Data Lake Pipeline.

copyRegion String
createdAt String
expiresAt String
frequencyYype String
id String: Unique 24-hexadecimal digit string that identifies the Data Lake Pipeline.
masterKey String
mongodVersion String
policies List<String>
provider String: Target cloud provider for this Data Lake Pipeline.
replicaSetName String
size Integer
snapshotType String
status String
type String: Type of ingestion source of this Data Lake Pipeline.

copyRegion string
createdAt string
expiresAt string
frequencyYype string
id string: Unique 24-hexadecimal digit string that identifies the Data Lake Pipeline.
masterKey string
mongodVersion string
policies string[]
provider string: Target cloud provider for this Data Lake Pipeline.
replicaSetName string
size number
snapshotType string
status string
type string: Type of ingestion source of this Data Lake Pipeline.

copy_region str
created_at str
expires_at str
frequency_yype str
id str: Unique 24-hexadecimal digit string that identifies the Data Lake Pipeline.
master_key str
mongod_version str
policies Sequence[str]
provider str: Target cloud provider for this Data Lake Pipeline.
replica_set_name str
size int
snapshot_type str
status str
type str: Type of ingestion source of this Data Lake Pipeline.

copyRegion String
createdAt String
expiresAt String
frequencyYype String
id String: Unique 24-hexadecimal digit string that identifies the Data Lake Pipeline.
masterKey String
mongodVersion String
policies List<String>
provider String: Target cloud provider for this Data Lake Pipeline.
replicaSetName String
size Number
snapshotType String
status String
type String: Type of ingestion source of this Data Lake Pipeline.

DataLakePipelineSource, DataLakePipelineSourceArgs

ClusterName string: Human-readable name that identifies the cluster.
CollectionName string: Human-readable name that identifies the collection.
DatabaseName string: Human-readable name that identifies the database.
PolicyItemId string
ProjectId string: The unique ID for the project to create a data lake pipeline.
Type string: Type of ingestion source of this Data Lake Pipeline.

ClusterName string: Human-readable name that identifies the cluster.
CollectionName string: Human-readable name that identifies the collection.
DatabaseName string: Human-readable name that identifies the database.
PolicyItemId string
ProjectId string: The unique ID for the project to create a data lake pipeline.
Type string: Type of ingestion source of this Data Lake Pipeline.

clusterName String: Human-readable name that identifies the cluster.
collectionName String: Human-readable name that identifies the collection.
databaseName String: Human-readable name that identifies the database.
policyItemId String
projectId String: The unique ID for the project to create a data lake pipeline.
type String: Type of ingestion source of this Data Lake Pipeline.

clusterName string: Human-readable name that identifies the cluster.
collectionName string: Human-readable name that identifies the collection.
databaseName string: Human-readable name that identifies the database.
policyItemId string
projectId string: The unique ID for the project to create a data lake pipeline.
type string: Type of ingestion source of this Data Lake Pipeline.

cluster_name str: Human-readable name that identifies the cluster.
collection_name str: Human-readable name that identifies the collection.
database_name str: Human-readable name that identifies the database.
policy_item_id str
project_id str: The unique ID for the project to create a data lake pipeline.
type str: Type of ingestion source of this Data Lake Pipeline.

clusterName String: Human-readable name that identifies the cluster.
collectionName String: Human-readable name that identifies the collection.
databaseName String: Human-readable name that identifies the database.
policyItemId String
projectId String: The unique ID for the project to create a data lake pipeline.
type String: Type of ingestion source of this Data Lake Pipeline.

DataLakePipelineTransformation, DataLakePipelineTransformationArgs

Field string
Type string: Type of ingestion source of this Data Lake Pipeline.

Field string
Type string: Type of ingestion source of this Data Lake Pipeline.

field String
type String: Type of ingestion source of this Data Lake Pipeline.

field string
type string: Type of ingestion source of this Data Lake Pipeline.

field str
type str: Type of ingestion source of this Data Lake Pipeline.

field String
type String: Type of ingestion source of this Data Lake Pipeline.

Import

Data Lake Pipeline can be imported using project ID, name of the data lake and name of the AWS s3 bucket, in the format project_id–name, e.g.

$ pulumi import mongodbatlas:index/dataLakePipeline:DataLakePipeline example 1112222b3bf99403840e8934--test-data-lake-pipeline-test

See MongoDB Atlas API Documentation for more information.

To learn more about importing existing cloud resources, see Importing resources.

Package Details

Repository: MongoDB Atlas pulumi/pulumi-mongodbatlas
License: Apache-2.0
Notes: This Pulumi package is based on the mongodbatlas Terraform Provider.

MongoDB Atlas v3.18.0 published on Thursday, Sep 12, 2024 by Pulumi

pulumi/pulumi-mongodbatlas

mongodbatlas.DataLakePipeline

On this page

On this page

# Resource: mongodbatlas.DataLakePipeline

Example Usage

S

Create DataLakePipeline Resource

Constructor syntax

Parameters

Constructor example

DataLakePipeline Resource Properties

Inputs

Outputs

Look up Existing DataLakePipeline Resource

Supporting Types

DataLakePipelineIngestionSchedule, DataLakePipelineIngestionScheduleArgs

DataLakePipelineSink, DataLakePipelineSinkArgs

DataLakePipelineSinkPartitionField, DataLakePipelineSinkPartitionFieldArgs

DataLakePipelineSnapshot, DataLakePipelineSnapshotArgs

DataLakePipelineSource, DataLakePipelineSourceArgs

DataLakePipelineTransformation, DataLakePipelineTransformationArgs

Import

Package Details

On this page

On this page