How to Set Up Jenkins Continuous Integration with Amazon S3: Avoiding Root Uploads

How to Set Up Jenkins Continuous Integration with Amazon S3: Avoiding Root Uploads

If you’re a data scientist or software engineer, you’re likely familiar with Jenkins Continuous Integration (CI) and Amazon S3. But are you struggling with everything uploading to the root of your S3 bucket? This article will guide you through the correct setup to avoid this common issue.

What are Jenkins CI and Amazon S3?

First, let’s briefly cover what these two tools are:

• Jenkins CI is an open-source automation server that enables developers to reliably build, test, and deploy their software.
• Amazon S3 (Simple Storage Service) is an object storage service offering scalability, data availability, security, and performance.

Why do files upload to the root?

The most common reason for all files uploading to the root in S3 is a misconfiguration in the Jenkins job or the AWS CLI setup.

How to configure Jenkins CI with Amazon S3 correctly

To ensure your files upload to the correct location, follow these steps:

1. Install the necessary plugins. In your Jenkins dashboard, navigate to ‘Manage Jenkins’ > ‘Manage Plugins’ > ‘Available’. Search for and install the ‘Amazon S3 plugin’ and the ‘AWS CLI plugin’.

2. Set up AWS credentials in Jenkins. Go to ‘Manage Jenkins’ > ‘Manage Credentials’. Click on ‘(global)’, then ‘Add Credentials’. Choose ‘AWS Credentials’ from the ‘Kind’ dropdown, and fill in your ‘Access Key ID’ and ‘Secret Access Key’.

3. Configure the S3 profile in the AWS CLI. Connect to your Jenkins server and open a terminal. Run aws configure --profile your-profile-name and enter the necessary information when prompted.

4. Create a Jenkins job. Back in your Jenkins dashboard, click on ‘New Item’, name it, and choose ‘Freestyle project’. Under ‘Source Code Management’, set your repository and branch.

5. Configure the build step. In the project configuration, navigate to ‘Build’ > ‘Add build step’ > ‘Execute shell’. Here, define the commands to build your project.

6. Set up the post-build action. Scroll down to ‘Post-build Actions’ > ‘Add post-build action’ > ‘Publish artifacts to S3 Bucket’. Set your profile, bucket name, and files path (e.g., target/*.jar). Importantly, in ‘Storage Class’, select ‘Standard’. Also, under ‘Destination bucket’ > ‘Manage Files’, set the ‘Destination bucket base folder’ to your desired directory, not root (e.g., my-folder/). Click ‘Save’.

Now, when you run the job, the built artifacts will upload to your specified folder in the S3 bucket, rather than the root.

Conclusion

Correctly setting up Jenkins CI with Amazon S3 can streamline your development pipeline, but misconfiguration can lead to issues like root uploads. By following these steps, you can avoid this common pitfall and maximize the benefits of Jenkins CI and Amazon S3.

Remember, the key to successful integration is accurate configuration. With it, you can automate your workflows, enhance collaboration, and expedite your software development process.

About the author: The author is a seasoned data scientist and software engineer, specializing in DevOps practices and tools. They have extensive experience with Jenkins, Amazon S3, and continuous integration processes.

About Saturn Cloud

Saturn Cloud is your all-in-one solution for data science & ML development, deployment, and data pipelines in the cloud. Spin up a notebook with 4TB of RAM, add a GPU, connect to a distributed cluster of workers, and more. Join today and get 150 hours of free compute per month.