Keep the files off the app servers

Large application and batch files do not belong on Spring Boot heaps or Kubernetes ephemeral disks. The pattern we used on Eligibility & Enrollment is the one I would ship again: the compute server never sees the bytes. It asks an API for a short-lived URL. The browser or batch worker talks to S3. When the object lands, S3 tells SQS, a processor Lambda updates PostgreSQL, and every later download, list, or delete goes through the same API.

The diagram above is the whole design.

Two IAM roles, not one “app role”

A single role that can run the UI and write the documents bucket is how files leak onto the compute plane. Split identity the way the runtime is split.

compute-server is assumed by the application runtime (ECS task, EKS service account, or EC2 instance profile). It may invoke API Gateway (execute-api:Invoke), read configuration, and write application logs. It does not get s3:PutObject / s3:GetObject / s3:DeleteObject on the documents bucket. If a pod is compromised, it still cannot mint object access except by going through the API the same way a client does.

functions is assumed by both Lambdas. That is the only identity allowed to:

• call GetObject, PutObject, and DeleteObject (including the permissions needed to presign upload and download URLs)
• consume the landing queue (sqs:ReceiveMessage, DeleteMessage, GetQueueAttributes)
• reach PostgreSQL (typically via Secrets Manager + TLS, or IAM DB auth)
• write processor and API logs

Same role, two functions, one blast radius for file I/O. The compute role stays boring on purpose.

API Gateway is the only front door

A single HTTP API (or REST API) fronts the API Lambda. Four routes:

Route	What it does
POST /upload-url	Authenticate the caller, insert a pending row in PostgreSQL, return a short-lived presigned PUT (key, bucket, expiry, required headers).
GET /download-url	Authorize against metadata + RBAC, return a short-lived presigned GET.
GET /files	List metadata from PostgreSQL (status, size, content type, timestamps). No S3 ListObjects in the hot path.
DELETE /delete-file	Authorize, delete the object, mark the row deleted or remove it, and keep an audit stamp.

The compute server is just another client of these routes. Angular/React never embeds long-lived AWS keys.

PostgreSQL is the system of record for files

S3 is the blob store. Postgres is what the product queries:

• file_id (UUID the API issued before upload)
• s3_bucket, s3_key
• content_type, byte_size, optional checksum
• status: pending → available / failed
• uploaded_by, business reference (case id, batch job id)
• created_at, available_at, retry_count
• audit fields for delete and replace

GET /files and authorization for download/delete read this table, not the bucket listing. That is how you keep “the file exists in S3” from meaning “the user may see it.”

Landing path: S3 → SQS → processor Lambda

Direct S3-to-Lambda is tempting and wrong for this workload. A queue gives you:

1. S3 event (ObjectCreated) on the documents prefix.
2. SQS with a dead-letter queue, visibility timeout longer than the processor, and retry.
3. Processor Lambda (same functions role): HeadObject to confirm size and type, match s3_key to the pending row, set available, record checksum if you required one on the presigned POST/PUT.

Until the processor succeeds, /download-url refuses the object even if S3 already has it. Idempotency is the matching key: a duplicate S3 event must not create a second product file.

Failed processing stays failed with a reason; the DLQ is how you find poison keys without blocking the queue.

Why the compute server still exists

The Spring Boot (or whatever) compute plane still owns session, JWT/RBAC, case context, and “this user may upload to this program.” It calls /upload-url with that context. It does not stream the file through the pod, so a 400 MB batch extract cannot knock over the application JVM.

That is the architecture: two roles, API Gateway + API Lambda for /upload-url, /download-url, /files, and /delete-file, S3 for bytes, SQS + processor Lambda for “the object is real,” PostgreSQL for metadata. Comments welcome if you have run the same split with a different queue or with EventBridge instead of S3 notifications.