Migrating from AWS S3 to nfyio — A Complete Guide

nfyio’s storage layer is fully S3-compatible — the same SDKs, CLIs, and libraries you use with AWS work out of the box. This guide shows you how to migrate buckets, objects, and access policies from AWS S3 to nfyio with minimal disruption.

Prerequisites

• A running nfyio instance (see installation guide)
• AWS CLI v2 configured with source credentials
• mc (MinIO Client) for parallel sync
• Source S3 bucket names and regions

aws --version
mc --version

Step 1: Inventory Your AWS Buckets

List all buckets and sizes:

aws s3 ls

# Get total size per bucket
aws s3 ls s3://my-production-bucket --recursive --summarize | tail -2

Total Objects: 142,387
Total Size: 48.6 GiB

Export a manifest for large migrations:

aws s3api list-objects-v2 \
  --bucket my-production-bucket \
  --output json > manifest.json

# Count objects
jq '.Contents | length' manifest.json

Step 2: Create Target Buckets in nfyio

# Create matching buckets via the nfyio API
curl -X POST http://localhost:3000/api/v1/buckets \
  -H "Authorization: Bearer $NFYIO_JWT" \
  -H "Content-Type: application/json" \
  -d '{"name": "my-production-bucket", "region": "us-east-1"}'

{
  "id": "bucket_a1b2c3d4",
  "name": "my-production-bucket",
  "region": "us-east-1",
  "created_at": "2026-03-04T10:00:00Z",
  "endpoint": "http://localhost:7007"
}

Or use the AWS CLI with the nfyio endpoint:

aws s3 mb s3://my-production-bucket \
  --endpoint-url http://localhost:7007

Step 3: Configure MinIO Client for Both Endpoints

# AWS source
mc alias set aws https://s3.amazonaws.com $AWS_ACCESS_KEY $AWS_SECRET_KEY

# nfyio target
mc alias set nfyio http://localhost:7007 $NFYIO_ACCESS_KEY $NFYIO_SECRET_KEY

Verify connectivity:

mc ls aws/my-production-bucket --summarize
mc ls nfyio/my-production-bucket --summarize

Step 4: Sync Data

For small buckets (under 10 GB):

mc mirror aws/my-production-bucket nfyio/my-production-bucket \
  --overwrite --preserve

For large buckets, use parallel workers:

mc mirror aws/my-production-bucket nfyio/my-production-bucket \
  --overwrite --preserve \
  --watch \
  --json 2>&1 | tee migration-log.json

Monitor progress:

# In another terminal — watch object count grow
watch -n 5 'mc ls nfyio/my-production-bucket --recursive --summarize | tail -2'

Step 5: Verify Data Integrity

Compare checksums:

mc diff aws/my-production-bucket nfyio/my-production-bucket

If mc diff reports zero differences, the migration is clean.

Spot-check a few objects:

# Download from both and compare
mc cat aws/my-production-bucket/data/report-2026.pdf | md5sum
mc cat nfyio/my-production-bucket/data/report-2026.pdf | md5sum

Step 6: Migrate Access Keys

Map your AWS IAM access keys to nfyio access keys:

curl -X POST http://localhost:3000/api/v1/access-keys \
  -H "Authorization: Bearer $NFYIO_JWT" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "production-app",
    "permissions": ["read", "write", "list"],
    "buckets": ["my-production-bucket"],
    "expires_at": "2027-03-04T00:00:00Z"
  }'

{
  "access_key": "nfyio_ak_x9y8z7w6",
  "secret_key": "nfyio_sk_a1b2c3d4e5f6g7h8",
  "name": "production-app",
  "created_at": "2026-03-04T12:00:00Z"
}

Step 7: Update Application Config

Replace your AWS SDK configuration:

// Before (AWS)
const s3 = new S3Client({
  region: 'us-east-1',
  credentials: {
    accessKeyId: process.env.AWS_ACCESS_KEY_ID,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY,
  },
});

// After (nfyio)
const s3 = new S3Client({
  region: 'us-east-1',
  endpoint: process.env.NFYIO_STORAGE_URL, // http://localhost:7007
  forcePathStyle: true,
  credentials: {
    accessKeyId: process.env.NFYIO_ACCESS_KEY,
    secretAccessKey: process.env.NFYIO_SECRET_KEY,
  },
});

Python (boto3):

import boto3

s3 = boto3.client(
    "s3",
    endpoint_url="http://localhost:7007",
    aws_access_key_id="nfyio_ak_x9y8z7w6",
    aws_secret_access_key="nfyio_sk_a1b2c3d4e5f6g7h8",
)

# Works exactly like AWS S3
response = s3.list_objects_v2(Bucket="my-production-bucket")
for obj in response.get("Contents", []):
    print(obj["Key"], obj["Size"])

Step 8: Enable Embeddings (Bonus)

Once your data is in nfyio, enable automatic embeddings for semantic search:

curl -X POST http://localhost:3000/api/v1/buckets/my-production-bucket/embeddings \
  -H "Authorization: Bearer $NFYIO_JWT" \
  -H "Content-Type: application/json" \
  -d '{
    "enabled": true,
    "model": "text-embedding-3-small",
    "file_types": ["pdf", "md", "txt", "docx"]
  }'

This is something AWS S3 alone cannot do — every uploaded document is automatically chunked, embedded, and indexed in pgvector.

Migration Comparison

Aspect	AWS S3	nfyio
API Compatibility	Native	S3-compatible
Hosting	AWS-managed	Self-hosted
Embeddings	Not built-in	Built-in (pgvector)
AI/RAG Pipeline	Separate service	Integrated
Cost	Per-request + storage	Infrastructure only
Data Residency	AWS regions	Your servers
Vendor Lock-in	High	None

Key Takeaways

• nfyio is a drop-in S3 replacement — same API, same SDKs, same CLI
• mc mirror handles parallel, resumable data transfer with integrity checks
• Access keys in nfyio support fine-grained bucket-level permissions
• After migration, you get built-in embedding and semantic search at no extra cost
• Zero vendor lock-in: your data stays on your infrastructure

For the full API reference, see API docs. For access key management, see access keys.