18th USENIX Symposium on Operating Systems Design and Implementation (OSDI '24)
2024-07-10..2024-07-12
Santa Clara, United States
who it's for
ml infrastructure engineers, systems researchers, cloud datacenter operators, platform engineers, operating systems developers, storage and filesystems engineers, security and verification engineers, hardware architecture designers
topics
large language model serving; serverless microVM snapshot compression; transactional page migration; CXL memory tiering; KV-cache offload and prefetching; disaggregated prefill and decoding; dynamic scheduling for LLM serving; hybrid far-memory data planes; in-network lock management; file-system crash consistency checking; verified security modules for confidential VMs; global scheduling of ML training
| title | author | length |
|---|---|---|
| Joint Keynote Address: Scaling AI Sustainably: An Uncharted Territory | Carole-Jean Wu | 01h 02m |
Claims that AI growth shifts emissions from operations to manufacturing, supported by specific measurements: global data‑center compute rises 550× (2010–2018)… | ||
| Sabre: Hardware-Accelerated Snapshot Compression for Serverless MicroVMs | Nikita Lazarev, Varun Gohil, James Tsai, Andy Anderson, Bhushan Chitlur, Zhiru Zhang, Christina Delimitrou | 15m |
Presents a system that uses a near‑memory IA accelerator to do lossless page (de)compression and zero‑copy prefetching for Firecracker microVMs, building a… | ||
| Nomad: Non-Exclusive Memory Tiering via Transactional Page Migration | Lingfeng Xiang, Zhen Lin, Weishu Deng, Hui Lu, Jia Rao, Yifan Yuan, Ren Wang | 15m |
Nomad implements transactional page migration and page shadowing to keep migrated pages accessible and retain shadow copies in the slow tier, turning exclusive… | ||
| Managing Memory Tiers with CXL in Virtualized Environments | Yuhong Zhong, Daniel S. Berger, Carl A. Waldspurger, Ryan Wee, Ishwar Agarwal, Rajat Agarwal, Frank Hady, Karthik Kumar, Mark D. Hill, Mosharaf Chowdhury, Asaf Cidon | 16m |
Introduces Intel Flat Memory Mode, a hardware-managed, cacheline-granular CXL tier that acts as a direct-map exclusive cache with zero CPU overhead and… | ||
| Harvesting Memory-bound CPU Stall Cycles in Software with MSH | Zhihong Luo, Sam Son, Sylvia Ratnasamy, Scott Shenker | 16m |
A software system called MSH instruments binaries via profile-guided binary instrumentation, adds prefetching and bounded scavenger yields, and uses runtime… | ||
| A Tale of Two Paths: Toward a Hybrid Data Plane for Efficient Far-Memory Applications | Lei Chen, Shi Liu, Chenxi Wang, Haoran Ma, Yifan Qiao, Zhe Wang, Chenggang Wu, Youyou Lu, Xiaobing Feng, Huimin Cui, Shan Lu, Harry Xu | 16m |
Atlas implements a hybrid data plane that picks per‑page transfer mode using always‑on profiling (a per‑page card table measuring card access ratio) and a… | ||
| DRust: Language-Guided Distributed Shared Memory with Fine Granularity, Full... | Haoran Ma, Yifan Qiao, Shi Liu, Shan Yu, Yuanjiang Ni, Qingda Lu, Jiesheng Wu, Yiying Zhang, Miryung Kim, Harry Xu | 16m |
DRust is a Rust-based DSM that exposes Rust’s ownership and borrow semantics to the runtime and uses an ownership-based coherence protocol to avoid… | ||
| Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve | Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, Ramachandran Ramjee | 16m |
Describes a scheduler that splits large prefill passes into chunked-prefills and constructs stall-free schedules that mix those chunks with decode batches to… | ||
| ServerlessLLM: Low-Latency Serverless Inference for Large Language Models | Yao Fu, Leyang Xue, Yeqi Huang, Andrei-Octavian Brabete, Dmitrii Ustiugov, Yuvraj Patel, Luo Mai | 16m |
ServerlessLLM is a distributed system that actually stores checkpoints on GPU servers, implements a loading-optimized checkpoint format and a multi‑tier… | ||
| InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV... | Wonbeom Lee, Jungi Lee, Junghwan Seo, Jaewoong Sim | 16m |
Framework that offloads the KV cache to CPU memory and speculative KV prefetching reduces PCIe transfers by predicting and fetching only the small set of… | ||
| Llumnix: Dynamic Scheduling for Large Language Model Serving | Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, Wei Lin | 16m |
Llumnix implements live migration of per-request KV caches to reschedule running LLM inferences across instances with near-zero downtime, using a distributed… | ||
| DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language... | Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang | 15m |
System disaggregates prefill and decoding onto separate GPUs, assigns independent parallelism and resource plans for each phase, and places instances to cut… | ||
| ACCL+: an FPGA-Based Collective Engine for Distributed Applications | Zhenhao He, Dario Korolija, Yu Zhu, Benjamin Ramhorst, Tristan Laan, Lucian Petrica, Michaela Blott, Gustavo Alonso | 15m |
ACCL+ is an open-source FPGA collective library that enables direct FPGA-to-FPGA collectives over UDP TCP and RDMA, claims firmware-updatable collective… | ||
| Beaver: Practical Partial Snapshots for Distributed Cloud Services | Liangcheng Yu, Xiao Zhang, Haoran Zhang, John Sonchack, Dan R. K. Ports, Vincent Liu | 15m |
Beaver presents a practical partial snapshot protocol that captures causally consistent states when only a subset of machines participates, by instrumenting… | ||
| Fast and Scalable In-network Lock Management Using Lock Fission | Hanze Zhang, Ke Cheng, Rong Chen, Haibo Chen | 14m |
FISSLOCK implements lock fission by running the grant decision in a programmable switch pipeline and keeping holders/waiters in server-side agent pools, with… | ||
| Chop Chop: Byzantine Atomic Broadcast to the Network Limit | Martina Camaioni, Rachid Guerraoui, Matteo Monti, Pierre-Louis Roman, Manuel Vidigueira, Gauthier Voron | 12m |
Chop Chop claims Byzantine atomic broadcast at line rate, matching unordered protocol cost even for 8‑byte messages by converting full batches into distilled… | ||
| Enabling Tensor Language Model to Assist in Generating High-Performance Tensor... | Yi Zhai, Sijia Yang, Keyu Pan, Renwei Zhang, Shuo Liu, Chao Liu, Zichun Ye, Jianmin Ji, Jie Zhao, Yu Zhang, Yanyong Zhang | 15m |
A compiler-side framework converts tensor-program search into a generation task and implements a tensor language model that samples decision paths rather than… | ||
| Ladder: Enabling Efficient Low-Precision Deep Learning Computing through... | Lei Wang, Lingxiao Ma, Shijie Cao, Quanlu Zhang, Jilong Xue, Yining Shi, Ningxin Zheng, Ziming Miao, Fan Yang, Ting Cao, Yuqing Yang, Mao Yang | 16m |
Ladder is a compiler that represents evolving low‑bit formats as a first‑class tType and exposes four concrete tensor transforms (slice/map/convert/pad) to… | ||
| Caravan: Practical Online Learning of In-Network ML Models with Labeling Agents | Qizheng Zhang, Ali Imran, Enkeleda Bardhi, Tushar Swamy, Nathan Zhang, Muhammad Shahbaz, Kunle Olukotun | 18m |
CARAVAN is a prototype system that uses labeling agents (off‑the‑shelf foundation models like GPT‑4 via API) to generate noisy labels (70–80% accuracy) for… | ||
| nnScaler: Constraint-Guided Parallelization Plan Generation for Deep Learning Training | Zhiqi Lin, Youshan Miao, Quanlu Zhang, Fan Yang, Yi Zhu, Cheng Li, Saeed Maleki, Xu Cao, Ning Shang, Yilei Yang, Weijiang Xu, Mao Yang, Lintao Zhang, Lidong Zhou | 14m |
A plan-generation framework implements three primitives op-trans, op-assign, op-order plus constraint filters so experts compose custom parallelization spaces… | ||
| ChameleonAPI: Automatic and Efficient Customization of Neural Networks for ML... | Yuhan Liu, Chengcheng Wan, Kuntai Du, Henry Hoffmann, Junchen Jiang, Shan Lu, Michael Maire | 16m |
Paper builds ChameleonAPI that parses app code to extract a decision-making summary, then uses it to construct an application-specific loss function and… | ||
| SquirrelFS: using the Rust compiler to check file-system crash consistency | Hayley LeBlanc, Nathan Taylor, James Bornholt, Vijay Chidambaram | 16m |
SquirrelFS is a 7,500‑line persistent‑memory file system implemented in Rust that encodes crash‑ordering rules in types so compilation proves certain crash… | ||
| High-throughput and Flexible Host Networking for Accelerated Computing | Athinagoras Skiadopoulos, Zhiqiang Xie, Mark Zhao, Qizhe Cai, Saksham Agarwal, Jacob Adelmann, David Ahern, Carlo Contavalli, Michael Goldflam, Vitaly Mayatskikh, Raghu Raja, Daniel Walton, Rachit Agarwal, Shrijeet Mukherjee, Christos Kozyrakis | 14m |
ZeroNIC is a co-designed FPGA NIC plus software stack that physically separates control/data separation to deliver a zero-copy data path to CPU and GPU memory… | ||
| IntOS: Persistent Embedded Operating System and Language Support for Multi-threaded... | Yilun Wu, Byounguk Min | 18m |
An embedded OS and language runtime called INTOS implements undo-logging of persistent objects and a novel replay-and-bypass recovery to avoid checkpointing… | ||
| Data-flow Availability: Achieving Timing Assurance in Autonomous Systems | Ao Li, Ning Zhang | 17m |
A survey of 189 timing bugs across 7 open-source CPS projects finds only 9% use real-time theory. The paper defines Data-flow Availability (DFA) to encode… | ||
| Microkernel Goes General: Performance and Compatibility in the HongMeng Production... | Haibo Chen, Xie Miao, Ning Jia, Nan Wang, Yu Li, Nian Liu, Yutao Liu, Fei Wang, Qiang Huang, Kun Li, Hongyang Yang, Hui Wang, Jie Yin, Yu Peng, Fengwei Xu | 16m |
HongMeng (HM) is a Linux-ABI-compatible commercial microkernel with a 90k-line core and ~1M lines of user services, deployed on tens of millions of devices and… | ||
| When will my ML Job finish? Toward providing Completion Time Estimates through... | Abdullah Bin Faisal, Noah Martin, Hafiz Mohsin Bashir, Swaminathan Lamelas, Fahad R. Dogar | 19m |
Claims a system that gives reliable completion-time estimates for ML jobs by configuring Weighted-Fair-Queueing so preemption is bounded and predictable. Uses… | ||
| Optimizing Resource Allocation in Hyperscale Datacenters: Scalability, Usability, and... | Neeraj Kumar, Pol Mauri Ruiz, Vijay Menon, Igor Kabiljo, Mayank Pundir, Andrew Newell, Daniel Lee, Liyuan Wang, Chunqiang Tang | 16m |
Meta builds Rebalancer, a reusable resource‑allocation framework with a high‑level spec language of 28 specs that translates rules into a compact expression… | ||
| μSlope: High Compression and Fast Search on Semi-Structured Logs | Rui Wang, Devin Gibson, Kirk Rodrigues, Yu Luo, Yun Zhang, Kaibo Wang, Yupeng Fu, Ting Chen, Ding Yuan | 16m |
μSlope losslessly compresses semi-structured logs by extracting and storing schema trees once, grouping records with the same schema into encoded record… | ||
| ServiceLab: Preventing Tiny Performance Regressions at Hyperscale through... | Mike Chow, Yang Wang, William Wang, Ayichew Hailu, Rohan Bopardikar, Bin Zhang, Jialiang Qu, David Meisner, Santosh Sonawane, Yunqi Zhang, Rodrigo Paim, Mack Ward, Ivor Huang, Matt McNally, Daniel Hodges, Zoltan Farkas, Caner Gocmen, Elvis Huang, Chunqiang Tang | 16m |
ServiceLab is a pre-production A/B performance platform that replays sampled RPCs across on-demand heterogeneous machines to catch tiny regressions before… | ||
| MAST: Global Scheduling of ML Training across Geo-Distributed Datacenters at Hyperscale | Arnab Choudhury, Yang Wang, Tuomas Pelkonen, Kutta Srinivasan, Abha Jain, Shenghao Lin, Delia David, Siavash Soleimanifard, Michael Chen, Abhishek Yadav, Ritesh Tijoriwala, Denis Samoylov, Chunqiang Tang | 16m |
MAST is a global scheduler that automatically places ML input tables and training across Meta regions, modeling table placement as a mixed-integer optimization… | ||
| Automatically Reasoning About How Systems Code Uses the CPU Cache | Rishabh Iyer, Katerina Argyraki, George Candea | 15m |
CFAR extracts a compact program called a memory distillate that outputs symbolic memory accesses, built by combining symbolic execution of source paths with… | ||
| VeriSMo: A Verified Security Module for Confidential VMs | Ziqiao Zhou, Weiteng Chen, Sishuai Gong, Chris Hawblitzel, Weidong Cui | 17m |
VERISMO implements a Rust security module for AMD SEV-SNP that is formally verified for functional correctness, secure information flow, and VM confidentiality… | ||
| Validating the eBPF Verifier via State Embedding | Hao Sun, Zhendong Su | 15m |
State embedding validates the eBPF verifier by taking concrete runtime states from profiling and folding, encoding them into an eBPF program so the verifier… | ||
| Using Dynamically Layered Definite Releases for Verifying the RefFS File System | Mo Zou, Dong Du, Mingkai Dong, Haibo Chen | 15m |
RefFS is a concurrent in-memory file system with a machine-checkable proof that guarantees both safety and liveness by enforcing dynamically layered definite… | ||
| Anvil: Verifying Liveness of Cluster Management Controllers | Xudong Sun, Wenjie Ma, Jiawei Tyler Gu, Zicheng Ma, Tej Chajed, Jon Howell, Andrea Lattuada, Oded Padon, Lalith Suresh, Adriana Szekeres, Tianyin Xu | 17m |
Anvil presents eventually stable reconciliation as a concise temporal liveness spec and provides a Rust framework that verifies controllers against it using… | ||
| DSig: Breaking the Barrier of Signatures in Data Centers | Marcos K. Aguilera, Clément Burgelin, Rachid Guerraoui, Antoine Murat, Athanasios Xygkis, Igor Zablotchi | 16m |
Claims a hybrid OTS+EdDSA scheme that moves expensive work to a background plane and pre-distributes OTS public keys over RDMA, achieving 0.7 μs signing and… | ||
| Ransom Access Memories: Achieving Practical Ransomware Protection in Cloud with DeftPunk | Zhongyu Wang, Yaheng Song, Erci Xu, Haonan Wu, Guangxun Tong, Shizhuo Sun, Haoran Li, Jincheng Liu, Lijun Ding, Rong Liu, Jiaji Zhu, Jiesheng Wu | 15m |
DeftPunk is a block-level ransomware detection and recovery system for cloud block stores that pairs a two-layer classifier (fast decision tree front end… | ||
| Secret Key Recovery in a Global-Scale End-to-End Encryption System | Graeme Connell, Vivian Fang, Rolfe Schmidt, Emma Dauterman, Raluca Ada Popa | 15m |
System recovers end‑to‑end encryption keys by splitting keys into shares across three heterogeneous enclave trust domains (deployment uses Intel SGX on Azure… | ||
| Flock: A Framework for Deploying On-Demand Distributed Trust | Darya Kaviani, Sijun Tan, Pravein Govindan Kannan, Raluca Ada Popa | 17m |
Flock claims an on-demand distributed trust architecture that lets each client programmatically deploy serverless instances across three major clouds so an app… | ||
| FairyWREN: A Sustainable Cache for Emerging Write-Read-Erase Flash Interfaces | Sara McAllister, Yucong "Sherry" Wang, Benjamin Berg, Daniel S. Berger, George Amvrosiadis, Nathan Beckmann, Gregory R. Ganger | 14m |
FairyWREN is a flash cache built for emerging Write-Read-Erase (WREN) interfaces that gives the cache control over garbage collection and data placement… | ||
| Massively Parallel Multi-Versioned Transaction Processing | Shujian Qian, Ashvin Goel | 15m |
Epic is a GPU-first deterministic OLTP that precomputes version numbers and locations for each batched transaction set, using batched deterministic ordering… | ||
| Burstable Cloud Block Storage with Data Processing Units | Junyi Shu, Kun Qian, Ennan Zhai, Xuanzhe Liu, Xin Jin | 14m |
An end-to-end system called BurstCBS runs on DPUs to balance per-core load with added FPGA queues, uses a two-tier memory pool to prefill buffers without… | ||
| Motor: Enabling Multi-Versioning for Distributed Transactions on Disaggregated Memory | Ming Zhang, Yu Hua, Zhijun Yang | 14m |
Motor enables multi-version distributed transactions on disaggregated memory by storing version metadata contiguously so a coordinator finds the right version… | ||
| Detecting Logic Bugs in Database Engines via Equivalent Expression Transformation | Zu-Ming Jiang, Zhendong Su | 14m |
Proposes equivalent expression transformation that rewrites expressions (using CASE-based rewrites and boolean identities) so semantics stay the same while… | ||
| Inductive Invariants That Spark Joy: Using Invariant Taxonomies to Streamline... | Tony Nuda Zhang, Travis Hance, Manos Kapritsos, Tej Chajed, Bryan Parno | 16m |
Claim: Kondo auto-generates and proves the network-level lemmas that make host-level invariants inductive, so developers only prove the synchronous proof about… | ||
| Performance Interfaces for Hardware Accelerators | Jiacheng Ma, Rishabh Iyer, Sahand Kashani, Mahyar Emami, Thomas Bourgeat, George Candea | 14m |
Proposes a precise accelerator performance IR called LPN and an open toolchain ltc that converts RTL-derived IR into executable performance interfaces and… | ||
| IronSpec: Increasing the Reliability of Formal Specifications | Eli Goldweber, Weixin Yu, Seyed Armin Vakil Ghahani, Manos Kapritsos | 15m |
IronSpec combines automated sanity checker heuristics, a manual methodology for writing SpecTesting Proofs (STPs), and automated mutation testing as an… | ||
| Identifying On-/Off-CPU Bottlenecks Together with Blocked Samples | Minwoo Ahn, Jeongmin Han, Youngjin Kwon, Jinkyu Jeong | 17m |
Proposes blocked samples that simultaneously record on- and off-CPU events by capturing the last instruction pointer and call chain and an off-CPU subclass… | ||
| dLoRA: Dynamically Orchestrating Requests and Adapters for LoRA LLM Serving | Bingyang Wu, Ruidong Zhu, Zili Zhang, Peng Sun, Xuanzhe Liu, Xin Jin | 15m |
dLoRA is a LoRA-serving system that dynamically merges and unmerges adapters and migrates both requests and adapters to reduce latency and boost GPU… | ||
| Parrot: Efficient Serving of LLM-based Applications with Semantic Variable | Chaofan Lin, Zhenhua Han, Chengruidong Zhang, Yuqing Yang, Fan Yang, Chen Chen, Lili Qiu | 13m |
Parrot is an LLM-serving system that exposes Semantic Variable annotations so the service sees the dataflow across multiple prompt requests and can do… | ||
| USHER: Holistic Interference Avoidance for Resource Optimized ML Inference | Sudipta Saha Shubha, Haiying Shen, Anand Iyer | 15m |
System claims a concrete 2.6× higher goodput and 3.5× better cost-efficiency on large-scale tests by combining a fast kernel-based estimator that predicts… | ||
| Fairness in Serving Large Language Models | Ying Sheng, Shiyi Cao, Dacheng Li, Banghua Zhu, Zhuohan Li, Danyang Zhuo, Joseph E. Gonzalez, Ion Stoica | 16m |
Defines fairness as a token-cost function (input vs decode tokens) and implements a scheduler that updates per-client virtual counters at token granularity… | ||
| MonoNN: Enabling a New Monolithic Optimization Space for Neural Network Inference... | Donglin Zhuang, Zhen Zheng, Haojun Xia, Xiafei Qiu, Junjie Bai, Wei Lin, Shuaiwen Leon Song | 22m |
MonoNN compiles an entire static neural network into a monolithic kernel on a single GPU to slash CPU-to-GPU launch and off‑chip traffic, empirically… | ||
| nothing matches | ||