How to get AWS s3 cli to work with AWS Lightsail instances

This guide outlines how to configure AWS Lightsail to be able to transfer files to AWS S3 via the CLI for the purpose of backing up data from the Lightsail instance to S3. The guide also shows you how to fix for common errors such as:

• An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
• An error occurred (AccessDenied) when calling the PutObject operation: Access Denied

It is possible to connect from a Lightsail instance to S3 using the CLI to transfer files from the Lightsail instance to the S3 bucket but to do so there are a few quirks with the setup.

The following prerequisites are required:

• working Lightsail instance
• S3 bucket
• IAM User (linked to the policy below)
• IAM Policy
• the reader should have basic knowledge of AWS and CLI operations

I’m not going to detail how to setup a IAM User or IAM policy as that’s done to death already, just google it or see AWS IAM Users Managing Access Keys Creating IAM Policies in the AWS Console

• Create your bucket and get the ARN from the properties tab
• Create your IAM policy using the JSON below and change the relevant details so it matches your buckets ARN

IAM Policy (remember to replace “REPLACE-WITH-YOUR-BUCKET” with your actual bucket name)

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketLocation",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:PutObject",
                "s3:ListBucketMultipartUploads",
                "s3:ListMultipartUploadParts",
                "s3:ListJobs",
                "s3:AbortMultipartUpload",
                "s3:UpdateJobStatus",
                "s3:CreateJob",
                "s3:UpdateJobPriority"
            ],
            "Resource": [
                "arn:aws:s3:::REPLACE-WITH-YOUR-BUCKET",
                "arn:aws:s3:::REPLACE-WITH-YOUR-BUCKET/*"
            ]
        }
    ]
}

• Create the IAM user and attach the policy you created so the IAM user can access the S3 actions specified against the specified bucket. It’s good practice to limit the actions to the minimum set you need. Avoid delete actions if possible.
• Now in the IAM user area you will need to create access keys for the user
  • In the AWS Console go to IAM, then Choose Users.
  • In the Users list, choose the name of the IAM user.
  • Choose the Security Credentials tab. Under Access keys, choose Create access key.
  • Make sure you keep the access key details somewhere safe, the secret wont be shown again if you lose it you will have to create another one

On your Lightsail instance you will need to connect via SSH or the Lightsail console and test your setup. First you need to use ‘aws configure‘ to setup the access keys you created in the earlier step. It will guide you through a wizard, enter your access key and secret and once you are done you should be able to utilise the CLI to access S3.

You can test it out by using the following command

aws s3 ls s3://your-bucket/

which should list your bucket’s contents (if it’s empty put something in there using the AWS Management Console to test). If instead you get the following error:

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

Then you’ve got one of two issues:

1. You’ve not saved the AWS credentials under the user you are running the CLI command with i.e. you are running ‘aws s3 ls’ which runs as the user bitnami, or you are running ‘sudo aws s3 ls’ which is running as root. It’s easy to get them confused and setup the credentials under the bitnami user but then incorrectly try to run the cli as root which will result in Access Denied as the root account wont have the AWS access key you setup earlier.
2. Your policy isn’t setup properly or isn’t assigned to the user you setup. Open the user and make sure they have the policy listed.

To backup a folder from your Lightsail instance to S3 you can use the following command

aws s3 cp /backups s3://your-bucket/ --recursive

This will copy everything under the Lightsail directory /backups to your s3 bucket. You need the –recursive flag to copy folders.