Set Up S3 Bucket Integration

How to set up S3 Bucket integration for Open Loyalty data exports

Open Loyalty allows you to export data to an S3 bucket for seamless integration with your data lake. This guide will walk you through two approaches to setting up S3 Bucket integration:

1. Access to the S3 Bucket

2. Replicating Data Between S3 Buckets

Each method offers flexibility depending on your organization's existing infrastructure and preferences.

Open Loyalty generates daily data exports, providing delta changes for the previous day in a CSV format. For detailed information about each export type, refer to the following article:

https://help.openloyalty.io/technical-guide/s3-bucket-data-exports/data-structure-and-types

Access to the S3 Bucket

In this method, the client is granted access to the Open Loyalty S3 bucket. Here’s how to set it up:

1

Create an AWS User

Create a new user specifically for this integration, enabling relevant access.

2

Share IAM Role Details

• After creating the user, attach an IAM role with permissions to access the Open Loyalty S3 bucket.

• Share the following details with the Open Loyalty team:

  • Role ARN: The Amazon Resource Name of your role.

  • Account ID: Your AWS account ID.

3

Access the Open Loyalty S3 Bucket

• The Open Loyalty team will grant access to their S3 bucket for this IAM role.

• Once access is granted, you can start setting up a pipeline to integrate the data directly from the Open Loyalty bucket to your data lake.

Advantages:

• Reduced overhead as you don’t need to maintain your own intermediate bucket.

---

Replicating Data Between S3 Buckets

In this method, Open Loyalty replicates data to a client’s S3 bucket. This method is beneficial for organizations that prefer to maintain full control over their data pipeline.

1

Create an S3 Bucket

Create a new S3 bucket in your account, ensuring proper permissions are configured.

2

Share Bucket Details

Provide the following details to the Open Loyalty team:

• Bucket Name: The name of the S3 bucket where the data will be replicated.

• Bucket ARN: The Amazon Resource Name of the S3 bucket.

• Account ID: Your AWS account ID.

• Bucket Link: The URL link to your bucket.

Remember to enable bucket versioning.

3

Apply Bucket Policy

Open Loyalty will share with you the Bucket Policy to be applied to the bucket on your end.

4

Enable Data Replication

The Open Loyalty team will configure replication from their bucket to your designated S3 bucket.

Delta files are exported daily, after midnight UTC time.

5

Set Up a Pipeline

Once the data is replicated, you can create a pipeline to transfer data from your S3 bucket to your data lake using tools like AWS Glue, DataSync, or other tools.

Advantages:

• Full control over your S3 bucket and data pipeline.

• Flexibility to apply custom processing on your replicated data before integration.

---

Key Considerations

• Data Security: Ensure that IAM roles and bucket policies are configured with the principle of least privilege to safeguard your data.

• Costs: Replication and data transfers may incur AWS costs. Monitor your usage to optimize expenses.

• Performance: Choose the method that aligns with your performance and integration needs.

Summary

By following the steps outlined above, you can easily set up integration with Open Loyalty data using the method that best fits your organization’s requirements. Whether you prefer direct access to the Open Loyalty S3 bucket or replication to your own bucket, these approaches enable efficient data integration to power your analytics and insights.