All Docs

Migrate from AWS S3

Migrate your data and applications from AWS S3 to NFYio. Assessment, sync steps, verification, and rollback planning.

This guide walks you through migrating your data and applications from AWS S3 to NFYio’s S3-compatible storage. You’ll assess your current setup, perform the migration, verify integrity, update application configs, and prepare a rollback plan.

Assessment

Before migrating, assess your current S3 usage to plan the migration effectively.

Bucket Inventory

List all buckets and their approximate sizes:

# List all buckets
aws s3 ls

# Get bucket sizes (requires aws s3api)
for bucket in $(aws s3 ls | awk '{print $3}'); do
  size=$(aws s3 ls s3://$bucket --recursive --summarize 2>/dev/null | grep "Total Size" | awk '{print $3}')
  count=$(aws s3 ls s3://$bucket --recursive --summarize 2>/dev/null | grep "Total Objects" | awk '{print $3}')
  echo "$bucket: $count objects, $size bytes"
done

Access Patterns

Document how your applications access S3:

PatternExampleNFYio Support
Standard S3 APIGetObject, PutObject, ListObjects✓ Full
Presigned URLsTemporary download/upload URLs✓ Full
Multipart uploadLarge file uploads✓ Full
S3 SelectSQL over objectsCheck compatibility
S3 Event NotificationsLambda triggersUse webhooks/callbacks
Cross-region replicationCRRManual or custom

Size and Bandwidth Estimates

# Total data size
aws s3 ls s3://my-bucket --recursive --summarize

# Estimate transfer time (example: 100GB at 50 Mbps = ~4.5 hours)
# Formula: (size_bytes * 8) / (bandwidth_bps) = seconds

Application Configuration Audit

Identify all places that reference S3:

  • Environment variables (AWS_ENDPOINT, S3_BUCKET, etc.)
  • Config files (YAML, JSON, .env)
  • IAM roles and access keys
  • CORS settings
  • Bucket policies and ACLs

Migration Steps

Step 1: Configure NFYio Endpoints

Ensure NFYio storage is running and note the endpoint:

# NFYio storage endpoint (adjust host/port for your deployment)
export NFYIO_ENDPOINT="https://storage.yourdomain.com"
# Or for local: http://localhost:7007

Create access keys in the NFYio dashboard or via API. You’ll need:

  • Access Key ID
  • Secret Access Key

Configure AWS CLI to use NFYio:

# Option A: Use --endpoint-url for each command
aws s3 ls --endpoint-url $NFYIO_ENDPOINT

# Option B: Create a named profile for NFYio
aws configure --profile nfyio
# AWS Access Key ID: your-nfyio-access-key
# AWS Secret Access Key: your-nfyio-secret-key
# Default region: us-east-1 (or any; NFYio ignores region)
# Default output format: json

# Add to ~/.aws/config:
# [profile nfyio]
# endpoint_url = https://storage.yourdomain.com

Step 2: Create Buckets in NFYio

Create buckets that mirror your S3 buckets:

# Create buckets (one per S3 bucket)
aws s3 mb s3://my-app-assets --endpoint-url $NFYIO_ENDPOINT --profile nfyio
aws s3 mb s3://my-app-uploads --endpoint-url $NFYIO_ENDPOINT --profile nfyio
aws s3 mb s3://backups --endpoint-url $NFYIO_ENDPOINT --profile nfyio

Step 3: Sync Data

The AWS CLI uses a single endpoint per command, so for S3 → NFYio migration use a two-step approach (via local) or rclone.

Option A: Two-Step Sync (AWS CLI)

# Step 1: Sync from AWS S3 to local
mkdir -p ./migration-temp
aws s3 sync s3://my-app-assets ./migration-temp/my-app-assets

# Step 2: Sync from local to NFYio
aws s3 sync ./migration-temp/my-app-assets s3://my-app-assets \
  --endpoint-url $NFYIO_ENDPOINT \
  --profile nfyio

# Cleanup
rm -rf ./migration-temp

Note: Requires sufficient local disk space for the bucket contents.

Option B: Stream with Pipe (No Local Copy)

For smaller buckets or when streaming is acceptable:

# List objects and copy one-by-one (scriptable)
aws s3 ls s3://my-app-assets --recursive | while read -r line; do
  key=$(echo "$line" | awk '{print $4}')
  aws s3 cp "s3://my-app-assets/$key" "s3://my-app-assets/$key" \
    --endpoint-url $NFYIO_ENDPOINT \
    --profile nfyio \
    --copy-props none
done

For production, prefer rclone (Option C) for checksums and efficiency.

Step 4: Using rclone (Alternative)

rclone supports S3-to-S3-compatible sync with checksums:

# Install rclone: https://rclone.org/install/

# Configure remotes
rclone config
# n) New remote
# name> aws
# type> s3
# provider> AWS
# env_auth> false
# access_key_id> YOUR_AWS_KEY
# secret_access_key> YOUR_AWS_SECRET
# region> us-east-1

# name> nfyio
# type> s3
# provider> Other
# endpoint> https://storage.yourdomain.com
# access_key_id> YOUR_NFYIO_KEY
# secret_access_key> YOUR_NFYIO_SECRET

# Sync with verification
rclone sync aws:my-app-assets nfyio:my-app-assets \
  --checksum \
  --transfers 8 \
  --progress

Step 5: Verify Migration

Object Count

# AWS source
aws s3 ls s3://my-app-assets --recursive --summarize

# NFYio destination
aws s3 ls s3://my-app-assets --recursive --summarize \
  --endpoint-url $NFYIO_ENDPOINT --profile nfyio

Compare “Total Objects” and “Total Size”.

Checksum Verification (rclone)

rclone check aws:my-app-assets nfyio:my-app-assets

Spot-Check Critical Files

# Download a few files from both and compare
aws s3 cp s3://my-app-assets/config.json /tmp/aws_config.json
aws s3 cp s3://my-app-assets/config.json /tmp/nfyio_config.json \
  --endpoint-url $NFYIO_ENDPOINT --profile nfyio

diff /tmp/aws_config.json /tmp/nfyio_config.json
# No output = files identical

Step 6: Update Application Configs

Update your applications to use the NFYio endpoint:

Environment variables:

# Before (AWS)
AWS_REGION=us-east-1
AWS_S3_BUCKET=my-app-assets

# After (NFYio)
AWS_REGION=us-east-1
AWS_S3_BUCKET=my-app-assets
AWS_ENDPOINT_URL=https://storage.yourdomain.com
# Or: S3_ENDPOINT=https://storage.yourdomain.com

SDK examples:

// Node.js (AWS SDK v3)
import { S3Client, ListObjectsCommand } from "@aws-sdk/client-s3";

const client = new S3Client({
  region: "us-east-1",
  endpoint: process.env.AWS_ENDPOINT_URL || "https://storage.yourdomain.com",
  forcePathStyle: true,  // Required for NFYio
});
# Python (boto3)
import boto3

s3 = boto3.client(
    "s3",
    endpoint_url="https://storage.yourdomain.com",
    region_name="us-east-1",
)

Step 7: Cutover

  1. Schedule maintenance — Brief read-only or downtime window
  2. Final sync — Run one last incremental sync to catch any changes
  3. Switch configs — Deploy updated application configs pointing to NFYio
  4. Smoke test — Verify upload, download, list operations
  5. Monitor — Watch logs and metrics for errors

Rollback Plan

If issues arise, roll back to AWS S3:

Immediate Rollback

  1. Revert application configs to use AWS S3 endpoint
  2. Redeploy or restart applications
  3. No data loss on AWS side (you copied, not moved)

Data Rollback

If you’ve written new data to NFYio during cutover and need to sync back to AWS:

# Sync NFYio → AWS (reverse direction)
aws s3 cp s3://my-app-assets/ s3://my-app-assets/ \
  --recursive \
  --endpoint-url $NFYIO_ENDPOINT \
  --profile nfyio
# Note: This uses NFYio as source; ensure profile/endpoint are correct for source

Use rclone for bidirectional sync if needed.

Rollback Checklist

  • Keep AWS buckets and data intact until migration is verified
  • Document exact config values for quick revert
  • Have rollback runbook and owner assigned
  • Test rollback procedure in staging

Common Issues

Issue: Sync Fails with 403

Cause: Invalid or expired credentials, or wrong endpoint.

Fix: Verify access keys, endpoint URL, and CORS if using browser-based uploads.

Issue: Large Files Timeout

Cause: Default timeouts too short for large object transfers.

Fix:

# Increase timeout for AWS CLI
export AWS_CLI_READ_TIMEOUT=3600
export AWS_CLI_CONNECT_TIMEOUT=60

Or use multipart upload (automatic for large files in AWS CLI).

Issue: Metadata or ACLs Not Preserved

Cause: NFYio may not support all S3 metadata or ACL features.

Fix: Use --metadata-directive COPY and --acl where supported. Document any unsupported features and adjust application logic.

Issue: Presigned URLs Break After Migration

Cause: Applications may cache or hardcode AWS domain.

Fix: Ensure all presigned URL generation uses the configured endpoint. Update any hardcoded s3.amazonaws.com references.

Issue: CORS Errors in Browser

Cause: NFYio bucket CORS not configured for your frontend origin.

Fix: Configure CORS on the NFYio bucket via API or dashboard to allow your domain.

Post-Migration

  • Monitor — Watch error rates, latency, and storage usage
  • Optimize — Tune multipart thresholds, connection pooling
  • Document — Update runbooks with NFYio-specific procedures
  • Decommission — After a stable period (e.g., 30 days), consider archiving or deleting AWS data

What’s Next

Edit this page on GitHub