╭─
╰─○ aws s3api head-object --bucket bhetal --key beach.jpg
{
"AcceptRanges": "bytes",
"LastModified": "2022-02-01T09:57:03+00:00",
"ContentLength": 1081193,
"ETag": "\"8c0f8376a8e35ca666a78866b1c2597f\"",
"ContentType": "binary/octet-stream",
"Metadata": {}
}
╭─
╰─○ aws s3 ls
2020-11-15 16:59:31 bhetal
2020-11-21 12:57:13 rk-test-nov2020
╭─
╰─○ aws s3api list-buckets
{
"Buckets": [
{
"Name": "bhetal",
"CreationDate": "2020-11-15T05:59:31+00:00"
},
{
"Name": "rk-test-nov2020",
"CreationDate": "2020-11-21T01:57:13+00:00"
}
],
"Owner": {
"ID": "XXXXXX"
}
}
There is a directory temp with the following contents.
╭─
╰─○ ls -l ~/Desktop/temp
total 12
-rw-rw-r-- 1 ryandam ryandam 12 Nov 21 12:31 first-file.txt
drwxrwxr-x 3 ryandam ryandam 4096 Nov 21 12:40 photos
drwxrwxr-x 2 ryandam ryandam 4096 Nov 21 13:27 videos
When syncing to S3, any directory can be excluded using --exclude flag.
╭─
╰─○ aws s3 sync ~/Desktop/temp s3://rk-test-nov2020 --exclude "videos/*"
Local directory videos is not uploaded to S3.
upload: Desktop/temp/first-file.txt to s3://rk-test-nov2020/first-file.txt
upload: Desktop/temp/photos/nature/photo1.txt to s3://rk-test-nov2020/photos/nature/photo1.txt
Once a directory is synced to S3, when next sync occurs only the new files and any updated files will be synced.
╭─
╰─○ aws s3 sync ~/Desktop/temp s3://rk-test-nov2020 --exclude "videos/*"
file2.txt and updated filefirst-file.txt are synced.upload: ./file2.txt to s3://rk-test-nov2020/file2.txt
upload: ./first-file.txt to s3://rk-test-nov2020/first-file.txt
Delete a file locally and sync to S3
╭─
╰─○ rm ~/Desktop/temp/file2.txt
deleting a file locally will not delete the file in S3 even after syncing.
╭─
╰─○ aws s3api list-objects --bucket rk-test-nov2020 --query 'Contents[].[{Key:Key,Size:Size}]' --output table
-------------------------------------
| ListObjects |
+----------------------------+-------+
| Key | Size |
+----------------------------+-------+
| file2.txt | 0 |
| first-file.txt | 19 |
| photos/nature/photo1.txt | 12 |
+----------------------------+-------+
╭─
╰─○ aws s3 sync ~/Desktop/temp s3://rk-test-nov2020 --exclude "videos/*"
╭─
╰─○ aws s3api list-objects --bucket rk-test-nov2020 --query 'Contents[].[{Key:Key,Size:Size}]' --output table
--------------------------------------
| ListObjects |
+----------------------------+-------+
| Key | Size |
+----------------------------+-------+
| file2.txt | 0 |
| first-file.txt | 19 |
| photos/nature/photo1.txt | 12 |
+----------------------------+-------+
To sync locally deleted files with S3, use --delete flag.
╭─
╰─○ aws s3 sync ~/Desktop/temp s3://rk-test-nov2020 --exclude "videos/*" --delete
delete: s3://rk-test-nov2020/file2.txt
╭─
╰─○ aws s3api list-objects --bucket rk-test-nov2020 --query 'Contents[].[{Key:Key,Size:Size}]' --output table
--------------------------------------
| ListObjects |
+----------------------------+-------+
| Key | Size |
+----------------------------+-------+
| first-file.txt | 19 |
| photos/nature/photo1.txt | 12 |
+----------------------------+-------+
Any objects that don't exist locally, but exist in S3 will be downloaded.
╭─
╰─○ aws s3 sync s3://rk-test-nov2020 ~/Desktop/temp
download: s3://rk-test-nov2020/first-file.txt to Desktop/temp/first-file.txt
aws s3 sync ryandam.net s3://ryandam.net \
--acl public-read \
--exclude ".git/*" \
--exclude ".idea/*" \
--exclude ".gitignore" \
--exclude "flutter*"
╭─
╰─○ aws s3api list-objects --bucket ryandam.net --query 'sort_by(Contents, &LastModified)[*].{Key: Key, LastModified: LastModified}' --output table
------------------------------------------------------------------------------------------------------------------------------
| ListObjects |
+------------------------------------------------------------------------------------------------+---------------------------+
| Key | LastModified |
+------------------------------------------------------------------------------------------------+---------------------------+
| flutter/1-async-await-in-dart/img/33b03ea05fb14a2a.jpg | 2020-07-09T01:49:51.000Z |
| flutter/1-async-await-in-dart/codelab.json | 2020-07-09T01:49:52.000Z |
| flutter/1-async-await-in-dart/img/779496d5ec612b0c.jpg | 2020-07-09T01:49:52.000Z |
| aws/2-hosting-a-static-website-on-s3/img/abf39eb479199bf4.jpeg | 2020-07-09T01:50:32.000Z |
| aws/3-iam-lab-to-grant-access-to-a-s3-bucket/img/20714d6aa45b1f8a.png | 2020-07-09T01:50:32.000Z |
| aws/3-iam-lab-to-grant-access-to-a-s3-bucket/img/34eef357645af6d3.png | 2020-07-09T01:50:32.000Z |
| aws/3-iam-lab-to-grant-access-to-a-s3-bucket/img/f04e66345b0e9be0.png | 2020-07-09T01:50:32.000Z |
| aws/4-spin-up-an-ec2-instance/img/2a2037465559f9c7.png | 2020-07-09T01:50:32.000Z |
| aws/4-spin-up-an-ec2-instance/img/461231c83f174082.png | 2020-07-09T01:50:32.000Z |
| aws/5-default-vpc/img/8168305fcb169609.png | 2020-07-09T01:50:33.000Z |
| aws/5-default-vpc/img/8fc8ccf1af8bd27b.png | 2020-07-09T01:50:33.000Z |
| aws/5-default-vpc/img/abc4fa5079ebb45.png | 2020-07-09T01:50:33.000Z |
Note If the bucket has more than 1000 objects, only 1000 are shown.
# Empyting
aws s3 ls | awk -F" " '{ print "aws s3 rm s3://" $3 " recursive"}'
# Deleting
aws s3 ls | awk -F" " '{ print "aws s3 rb s3://" $3 " --force"}'
cpaws s3 cp s3://firefund-assets/file.jpg s3://firefund-assets/file.jpg --content-type image/png
aws s3 cp \
s3://my.bucket/sub-directory/ s3://my.bucket/sub-directory/ \
--exclude '*.jpg' --exclude '*.png' \
--cache-control 'max-age=31104000' \
--recursive
# Copy a folder "flutter" from bucket "ryandam.net" to current directory.
aws s3 cp s3://ryandam.net/ . --recursive --exclude '*' --include 'flutter/*'
╭─
╰─○ aws s3 ls s3://ryandam.net
PRE assets/
PRE config/
PRE flutter/
PRE hr_tool/
PRE icons/
PRE images/
PRE travel/
PRE youtube/
2022-01-03 15:19:36 917 favicon.png
2022-01-03 15:19:35 7672 flutter_service_worker.js
2022-01-03 15:19:34 3856 index.html
2022-01-03 15:19:32 1888688 main.dart.js
2022-01-03 15:19:37 926 manifest.json
2022-01-06 22:09:26 259 sitemap.txt
2022-01-03 15:19:33 100 version.json
To show all objects in all prefixes, include option --recursive
╭─
╰─○ aws s3 ls --recursive s3://ryandam.net
2022-01-03 15:19:06 847 assets/AssetManifest.json
2022-01-03 15:19:11 208 assets/FontManifest.json
2022-01-03 15:19:10 810947 assets/NOTICES
2022-01-03 15:19:22 6148 assets/assets/icons/.DS_Store
2022-01-03 15:19:28 5548 assets/assets/icons/camera.svg
2022-01-03 15:19:25 5102 assets/assets/icons/document.svg
2022-01-03 15:19:21 101214 assets/assets/icons/hammock.svg
2022-01-03 15:19:26 2589 assets/assets/icons/info.svg
2022-01-03 15:19:29 3947 assets/assets/icons/jigsaw.svg
2022-01-03 15:19:23 9149 assets/assets/icons/thumbs-down.svg
2022-01-03 15:19:27 702 assets/assets/icons/thumbs-up.svg
2022-01-03 15:19:24 6059 assets/assets/icons/travel.svg
2022-01-03 15:19:14 6148 assets/assets/images/.DS_Store
2022-01-03 15:19:20 239814 assets/assets/images/hr_tool.png
2022-01-03 15:19:17 1240021 assets/assets/images/youtube_1.png
2022-01-03 15:19:16 835349 assets/assets/images/youtube_2.png
2022-01-03 15:19:13 1299300 assets/fonts/MaterialIcons-Regular.otf
2022-01-03 15:19:30 283452 assets/packages/cupertino_icons/assets/CupertinoIcons.ttf
2021-12-26 21:56:30 0 config/
2022-01-01 11:56:47 693 config/articles.json
...
s3api╭─
╰─○ aws s3api list-objects --bucket ryandam.net --query 'Contents[].[{Key:Key,Size:Size}]' --output table
---------------------------------------------------------------------------------------
| ListObjects |
+-------------------------------------------------------------------------+-----------+
| Key | Size |
+-------------------------------------------------------------------------+-----------+
| assets/AssetManifest.json | 847 |
| assets/FontManifest.json | 208 |
| assets/NOTICES | 810947 |
| assets/assets/icons/.DS_Store | 6148 |
| assets/assets/icons/camera.svg | 5548 |
...
--prefix╭─
╰─○ aws s3api list-objects --bucket ryandam.net --prefix flutter --query 'Contents[].[{Key:Key,Size:Size}]' --output table
--------------------------------------------------------------------
| ListObjects |
+------------------------------------------------------+-----------+
| Key | Size |
+------------------------------------------------------+-----------+
| flutter/ | 0 |
| flutter/amplify-authentication.html | 133855 |
| flutter/amplify-cli.html | 163864 |
| flutter/custom-themes-flutter.html | 125136 |
| flutter/images/aws-cognito-flutter.svg | 43097 |
| flutter/images/custom_theme.gif | 855896 |
+------------------------------------------------------+-----------+
list-objects-v2aws s3api list-objects-v2 --bucket ryandam.net --delimiter '/' --prefix 'flutter/' --query CommonPrefixes --output text
aws s3 ls --summarize --human-readable --recursive s3://ryandam.net
2021-12-25 19:14:21 1.6 MiB youtube/web/main.dart.js
2021-12-25 19:14:24 932 Bytes youtube/web/manifest.json
2021-12-25 19:14:16 106 Bytes youtube/web/version.json
Total Objects: 138
Total Size: 125.0 MiB
aws s3api list-objects-v2 --bucket ryandam.net --query "sort_by(Contents, &Size)[-5:].[Key, Size]"
# To include non-current versions
aws s3api list-object-versions --bucket ryandam.net --query "sort_by(Versions[*], &Size)[-5:]"
➜ aws s3api get-public-access-block --bucket bhetal
{
"PublicAccessBlockConfiguration": {
"BlockPublicAcls": true,
"IgnorePublicAcls": true,
"BlockPublicPolicy": true,
"RestrictPublicBuckets": true
}
}
When the Create Folder button is used in the Amazon S3 console, it creates a 'folder'. However, Amazon S3 does not use folders. Instead, it creates a zero-length object with the name of the folder.