Transfer Service & gsutil

Moving data into, out of, and between Cloud Storage buckets is a common task. Google Cloud provides multiple tools for data transfer — from the gsutil command-line tool for ad-hoc operations to the Storage Transfer Service for large-scale, scheduled, and cross-cloud migrations.

---

gsutil

gsutil is the original command-line tool for Cloud Storage. It provides a rich set of commands for working with buckets and objects.

Core gsutil Commands

Command	Description
gsutil mb	Create a bucket
gsutil cp	Copy objects (upload, download, or between buckets)
gsutil mv	Move or rename objects
gsutil rm	Delete objects
gsutil ls	List buckets or objects
gsutil rsync	Synchronise directories and buckets
gsutil stat	Display object metadata
gsutil du	Display disk usage (size)
gsutil cat	Output object contents to stdout
gsutil acl	Manage access control lists
gsutil iam	Manage IAM bindings
gsutil lifecycle	Manage lifecycle rules
gsutil versioning	Manage versioning
gsutil retention	Manage retention policies

Parallel Operations

The -m flag enables multi-threaded/multi-processing operations for significantly faster bulk transfers:

# Upload thousands of files in parallel
gsutil -m cp -r ./data/ gs://my-bucket/

# Download in parallel
gsutil -m cp -r gs://my-bucket/data/ ./local/

# Delete in parallel
gsutil -m rm gs://my-bucket/old-data/**

gsutil rsync

rsync synchronises a source and destination, transferring only new or changed files:

# Sync local directory to bucket
gsutil -m rsync -r ./local-dir gs://my-bucket/remote-dir

# Sync with deletion (remove files in destination that are not in source)
gsutil -m rsync -r -d ./local-dir gs://my-bucket/remote-dir

# Dry run (show what would change without making changes)
gsutil -m rsync -r -n ./local-dir gs://my-bucket/remote-dir

Flag	Description
-r	Recursive (process subdirectories)
-d	Delete destination files not in source
-n	Dry run
-m	Multi-threaded for speed

---

gcloud storage (Next-Gen CLI)

gcloud storage is the recommended CLI for new projects. It provides the same functionality as gsutil with better performance and unified syntax:

# Copy files
gcloud storage cp report.pdf gs://my-bucket/

# Sync directories
gcloud storage rsync ./local gs://my-bucket/remote --recursive

# List objects
gcloud storage ls gs://my-bucket/

# Create a bucket
gcloud storage buckets create gs://my-bucket --location=europe-west2

Key advantage: gcloud storage uses optimised transfer strategies and does not require a separate installation.

---

Storage Transfer Service