Building a content-focused, scientific document authoring workflow
summary
A reproducible authoring workflow based on pandoc + XeLaTeX build and git produces a versioned PDF artifact built by CI (Drone) and Make, created in about two weeks and reused across teams; the 45‑page paper compiles in ~12 seconds (single-section edits in ~2 seconds). Tooling includes tikz diagrams, Lua/python pandoc filters, git/GitHub reviews, and optional tectonic to reduce downloads; install size hits ~1.5–2 GB and equation editing remains the biggest pain point.
AI-written summary · about
chapters (23)
- 0:00Intro
- 2:36A BIG CHANGE
- 3:20DRAMATIS PERSONAE A team of seven colocateed data scientists and engineers
- 4:00REAL NEED A content-focused, scientific document authoring workflow for Data Scientists and Engineers alike
- 4:17THINKING ARCHITECTURALLY
- 4:41PRIMARY VALUES • Reviewable content: prose, diagrams, equations . Content-focused with minimal markup • Minimize structural exceptions with standardized styling and typesetting
- 5:09SECONDARY VALUES • Accomodate some preferences for LaTeX over simpler formats (Markdown) • Easy to use: one command to generate output • Automation: artifact built from versioned "code"
- 5:36Treat documentation as source code.
- 6:46SOLUTION
- 7:22BIGGEST BENEFIT? LaTeX typesetting without suffering writing LaTeX
- 8:06pandoc, BRIEFLY "A universal document converter"
- 8:38pandoc BASICS
- 10:11CLI metadata options can be put into the YAML front-matter of the document
- 12:21include-code of embedding pandoc
- 18:20MANAGING CONTENT One chapter per file - enables extraction • Transforms necessitate a build directory
- 18:53COMMITTING Use git commits to tell a story about the changes.
- 19:22REVIEWING
- 20:03CONTINUOUS INTEGRATION Block PR merging with Cl system automation • Ensure valid markup and view changes compiled • Run proselint or a grammar/spelling tool
- 21:23PAIN POINTS
- 21:33Dependency installation • Learning Pandoc's flavor of Markdown • "Why can't I just use LaTeX?" . Converting from Word or LaTeX loses cross- references
- 25:12GREATEST RISK OF ADDITIONAL TRANSFORMATION TOOLS?
- 26:15Leveled the playing field for contributions, great for collaborating and building documents with all of the features of LaTeX
- 27:02CURRENT USE
related talks
29mConquering PDFs- document understanding beyond plain text
13mVisualizing authorship and contribution of collaborative writing in e-learning environments
45mOpen Source for Open Science with Frictionless Data
42mJSON data + RML template = PDF report
54mLightning Talks + Closing Remarks DAY 1
43mKeynote
38mLightning Talks
06mUse Your Data Skills for Good