Stretching the Limits of S3 Backend for OpenTofu
Talkat OpenTofu Day North America 2025 · on Monday, 2025-11-10· at 14:40· Building B | Level 2 | B203·
summary
OpenTaco layers a stateless API on top of an S3 state bucket to add managed-backend features without extra storage, implementing a remote backend API proxy, versioning and rollbacks, cross-state dependency tracking, and RBAC with cloud-block–compatible auth. Early experiments using JSON files on S3 hit performance limits (28 records takes 8s), a Trino read cluster works but is complex, and a pragmatic fix uses SQLite in-memory with WAL that writes back to S3 and cuts that 8s to under 1s while sacrificing easy replication.
AI-written summary · about
related talks
50mA Homegrown S3 Server: Challenges of Building Stable Storage
31mScaling Autonomous-Driving R&D: A 100 PB, 100 Billion-File Open-Source Elastic File...
29mAccelerate your cloud architectures with the right storage options
44mOptimizing storage price and performance with Amazon S3
40mSupercharging Cloud Orchestration with Temporal Open Source
33mNext Gen of Datastores: Hot Queries, Cold Storage
32mLow Latency Serving of Offline Data: Efficient, Safe, and Reliable Data...
36mAn Exploration in Storing Telemetry in Cloud Object Storage