Making Sense Out of Flight Test Data with Python
summary
Aggregates 20 GB per sorty into a messy iRIG Chapter 10 container, decodes hundreds of messages with a machine‑readable XML “decoder ring,” and still wrestles with misaligned timestamps and coordinate frames across aircraft; total archive runs ~6–7 TB. Workflow currently uses pandas, numexpr, Cython/Numba experiments, CSVs and HDF5, a Makefile pipeline that takes days, and plotting with Matplotlib/pyqtgraph/Cesium. Concrete wins: automated aggregation revealed a consistent sensor bias (elevation up and left). Planned moves: Airflow/Luigi or Dask and better metadata/pedigree tracking.
AI-written summary · about
chapters (13)
- 0:00GTRI and electronic warfare
- 3:50Flight test data challenges
- 7:41Excel to Python evolution
- 10:30Binary data decoding
- 13:59Coordinate frame alignment
- 17:34NumPy and Numba optimization
- 20:02Data storage solutions
- 24:15Visualization and dashboards
- 27:49Pipeline automation
- 31:00Metadata and data pedigree
- 33:50Big data solutions
- 37:18Clustering and ML analysis
- 39:44Q&A
related talks
42mPython flying at 40,000 feet
42mFlying Python at 40,000 feet
03h 07mData of Unusual Size: Interactive Visualization
01h 25mA practical guide to analysis and interactive visualization of massive datasets
40mPython and pandas as back end to real-time data driven applications
40mScaling Python - Bank Edition
18mSit back, relax, and enjoy the talk: a journey through FlightRadar24 with Python
30mOn the Fly and On the Flight: Scientific Data Analysis Beyond the Beamline