Set Up S3 Bucket Integration
How to set up S3 Bucket integration for Open Loyalty data exports
Last updated
Was this helpful?
Technical Guide
How to set up S3 Bucket integration for Open Loyalty data exports
Last updated
Was this helpful?
Open Loyalty allows you to export data to an S3 bucket for seamless integration with your data lake. This guide will walk you through two approaches to setting up S3 Bucket integration:
Access to the S3 Bucket
Replicating Data Between S3 Buckets
Each method offers flexibility depending on your organization's existing infrastructure and preferences.
The export frequency is set to daily to ensure consistent and timely access to the data for all our customers.
There is one report set for the environment, regardless of the number of tenants created. There is a unique identifier (tenant_id) for all the reports. Then you can create pipelines to further split the data per tenant.
In this method, the client is granted access to the Open Loyalty S3 bucket. Here’s how to set it up:
Create an AWS User
Create a new user specifically for this integration, enabling relevant access.
Share IAM Role Details
After creating the user, attach an with permissions to access the Open Loyalty S3 bucket.
Share the following details with the Open Loyalty team:
Role ARN: The Amazon Resource Name of your role.
Account ID: Your AWS account ID.
Access the Open Loyalty S3 Bucket
The Open Loyalty team will grant access to their S3 bucket for this IAM role.
Once access is granted, you can start setting up a pipeline to integrate the data directly from the Open Loyalty bucket to your data lake.
Advantages:
Reduced overhead as you don’t need to maintain your own intermediate bucket.
In this method, Open Loyalty replicates data to a client’s S3 bucket. This method is beneficial for organizations that prefer to maintain full control over their data pipeline.
Create an S3 Bucket
Create a new S3 bucket in your account, ensuring proper permissions are configured.
Share Bucket Details
Provide the following details to the Open Loyalty team:
Bucket Name: The name of the S3 bucket where the data will be replicated.
Bucket ARN: The Amazon Resource Name of the S3 bucket.
Account ID: Your AWS account ID.
Bucket Link: The URL link to your bucket.
Remember to enable bucket versioning.
Apply Bucket Policy
Open Loyalty will share with you the Bucket Policy to be applied to the bucket on your end.
Enable Data Replication
The Open Loyalty team will configure replication from their bucket to your designated S3 bucket.
Set Up a Pipeline
Once the data is replicated, you can create a pipeline to transfer data from your S3 bucket to your data lake using tools like AWS Glue, DataSync, or other tools.
Advantages:
Full control over your S3 bucket and data pipeline.
Flexibility to apply custom processing on your replicated data before integration.
Data Security: Ensure that IAM roles and bucket policies are configured with the principle of least privilege to safeguard your data.
Costs: Replication and data transfers may incur AWS costs. Monitor your usage to optimize expenses.
Performance: Choose the method that aligns with your performance and integration needs.
By following the steps outlined above, you can easily set up integration with Open Loyalty data using the method that best fits your organization’s requirements. Whether you prefer direct access to the Open Loyalty S3 bucket or replication to your own bucket, these approaches enable efficient data integration to power your analytics and insights.
