«Математика — это та часть физики, где эксперименты дешевы.»
— В.И. Арнольд
N.B. LaTeX rendering is trash on Github, vs code works great.
- Cheap Experiments for Leverage University 🔧🌎
- Be the second coming of Kazushige Goto-san 🙇🥷
- HPC 🏎️💨
- Modeling 🚂
- Scientific Computing 🧑🔬
- Compiler work 🛠️
North Star: Ronin Perf LLC. ⚔️🥷
- Conquer the Math Dragon 🧮🐉: persistent nerves about doing higher level math need to be confronted with exposure therapy. Don't be fooled, math is half intuition cultivation and half a bag of tricks 🪄. It is learnable.
- Find Leverage: the longer I ignore the closed form solutions lurking in math the more pathetic my programming gets. δῶς μοι πᾶ στῶ καὶ τὰν γᾶν κινάσω 🌍🔧.
- Don't Get Bored 🥱: it's easy to get the gist of every subject halfway through a book but math always goes deeper. It won't be boring.
I'm a software developer with minimal experience in math (some linear algebra and calculus). Now I want to make a career step change.
This isn't ultralearning in the traditional sense, I'm not speedrunning an MIT CS degree, I'm trying to surpass it. Building slow and deep and foundational knowledge and intuition.
I am seeking a basis for the goal space: linearly independent and yet still spans the goal space.
I seek mathematical rigor to conquer the math dragon 🐉. No half assed computationally focussed "this is how you do a matrix multiply" books.
This booklist is meant to reflect the actual best and most rigorous texts that form the basis. MIT will have stuff like Strang, but I choose Axler. I choose Spivak for the disguised real analysis rather than a cookbook of derivatives. Soviet math is of particular interest for its reputation for blending intuition and rigor. Soviet books get ⚒️.
No bridge books 🌉📚! LLMs solve the difficulty cold start wall problem. Sometimes I'll dip into supplemental works.
I am also interested in domain specific goals that can add dimensionality and give a real leg up in a particularly interesting domain (Durbin's BSA, Arnold's CM, Hecht etc.).
This syllabus is STRICTLY about the math and text books not about manuals and hands on work. That is all covered in parallel documents.
Interleave 1-3 books at a time.
Anki helps to retain familiarity with definitions and challenging proofs encountered in work.
A Leuchtturm 1916 notebook contains a "Commonplace Book" of encountered proof tricks like "Completing the Square." The book will be refined with experience.
Work through the problems in a paper pad and nice notebook and also write them up in LaTeX. Solve the problems independently multiple times.
This process will take years. But every book completed should add to performance abilities at work. There won't be one day after completing this list where "Now you can work on HPC! 🧑🎓." Likely the list will be refined as the books are completed and you identify paths yourself rather than with help.
- 🛍️: Still to buy
- 🌐: Available Online
- 📖: Currently reading
- Axler — Linear Algebra Done Right
- Supplement: Shilov
- Supplement: Gelfand’s Lectures
- Spivak — Calculus
- Supplement: Schaum’s Outline
- Supplement: Shilov's Elementary Real & Complex Analysis
- Supplement: Courant & John Introduction to Calculus & Analysis I
- Knuth et al. — Concrete Mathematics
- Trefethen & Bau — Numerical Linear Algebra
- Supplement: Golub & Van Loan — Matrix Computations
- Higham — Accuracy and Stability of Numerical Algorithms
- Saad — Iterative Methods for Sparse Linear Systems
- 🛍️ Boyd & Vandenberghe — Convex Optimization
- Nocedal & Wright — Numerical Optimization
- 🛍️ Rivlin — Chebyshev Polynomials
- Spivak — Calculus on Manifolds
- Arnold — Ordinary Differential Equations
- 🛍️ Durrett — Probability: Theory and Examples
- Kolmogorov & Fomin — Elements of the Theory of Functions and Functional Analysis
- Supplement: Kreyszig
- 🛍️ Samarskii — Theory of Difference Schemes
- Tikhonov & Samarskii — Equations of Mathematical Physics
- 🛍️ Körner — Fourier Analysis
- 🛍️ Ahlfors — Complex Analysis
- 🛍️ Trefethen — Spectral Methods in MATLAB
- Bryant & O'Hallaron CSAPP
- Sipser
- CLRS – Introduction to Algorithms
- Engineering a Compiler [Alternative:Modern Compiler Design by Grune et al]
- Supplement: 🛍️ Muchnick's Advanced Compiler Design and Implementation
- 🛍️ Pierce – Types and Programming Languages
- 🛍️ Hennessy & Patterson – Computer Architecture: A Quantitative Approach
- 🛍️ Herlihy & Shavit – Art of Multiprocessor Programming
- 🛍️ Hwu & Kirk – Programming Massively Parallel Processors
- 🛍️ Gropp, Lusk & Skjellum – Using MPI
- 🛍️ Hastie & Tibshirani – Elements of Statistical Learning
- 🛍️ Warren – Hacker's Delight
- 🛍️ OSTEP
- 🛍️ Brendan Gregg – Systems Performance
- 🛍️ Fog Agner – Optimizing Software in C++
- Thurston — “On Proof and Progress in Mathematics”: philosophy of mathematics, discovery, and pedagogy.
- Wigner — “The Unreasonable Effectiveness of Mathematics in the Natural Sciences”: reflections on the mathematics–physics relationship.
- Williams, Waterman & Patterson — “Roofline: An Insightful Visual Performance Model for Multicore Architectures”: performance modeling of memory- vs compute-bound workloads.
- Hong & Kung — “I/O Complexity: The Red–Blue Pebble Game” (1981): foundational lower bounds on communication complexity.
- Culler et al. — “LogP: Towards a Realistic Model of Parallel Computation” (1993): classic model for distributed-memory performance.
- Gustafson — “Reevaluating Amdahl’s Law” (1988): scalability model (weak scaling vs Amdahl’s pessimism).
- Goto & van de Geijn — “Anatomy of High-Performance Matrix Multiplication”: cache optimization, BLAS kernel design.
- Whaley & Dongarra — “Automatically Tuned Linear Algebra Software (ATLAS)”: systematic search-based kernel optimization.
- Yotov et al. — “Is Search Really Necessary to Generate High-Performance BLAS?”: prediction vs empirical tuning.
- Frigo & Johnson — “The Design and Implementation of FFTW (‘The Fastest Fourier Transform in the West’)”: auto-tuning in FFT kernels.
- Volkov & Demmel — “LU, QR, and Cholesky Factorizations Using Vector Capabilities of GPUs” (2008): GPU kernel optimization, a Goto-style paper for accelerators.
- Lam, Wolf & Lam — “The Cache Performance and Optimizations of Blocked Algorithms”: cache blocking theory.
- Frigo, Leiserson, Prokop & Ramachandran — “Cache-Oblivious Algorithms”: recursive cache-friendly algorithms without tuning parameters.
- Lattner & Adve — “LLVM: A Compilation Framework for Lifelong Program Analysis & Transformation”: modular compiler infrastructure.
- Bastoul — “Code Generation in the Polyhedral Model”: loop transformations and dependence analysis.
- Allen & Kennedy — “Optimizing Compilers for Modern Architectures” (overview paper): classic introduction to compiler optimizations, ties into Muchnick.
- Baydin et al. — “Automatic Differentiation in Machine Learning: A Survey”: AD foundations and systems.
- Vaswani et al. — “Attention Is All You Need”: the Transformer architecture, parallelism-friendly ML.
- Dean & Ghemawat — “MapReduce: Simplified Data Processing on Large Clusters”: distributed data processing at scale.
These might help with my current job directly.
- Streetman & Banerjee – Solid State Electronic Devices
- Arnold's Mathematical Methods of Classical Mechanics ⚒️
- 🛍️ Hecht's Optics
- 🛍️ Griffiths – Introduction to Electrodynamics
It is fine if these seem completely ancillary or redundant. This is a subscription list to expand and pick through after the main curriculum.
- Durbin's Biological Sequence Analysis
- 🛍️ Cover & Joy – Elements of Information Theory
- 🛍️ Nielsen & Chuang – Quantum Computation and Quantum Information
- 🛍️ Shreve – Stochastic Calculus for Finance II
- 🛍️ Anderson – Computational Fluid Dynamics: The Basics with Applications: Navier-Stokes is the perfect PDE playground for HPC. Ties together everything: PDEs, numerical methods, parallel computing. Plus, fluid flow shows up in semiconductor manufacturing.
- 🛍️ Pharr, Jakob & Humphreys – Physically Based Rendering (PBRT): Combines Monte Carlo methods (Durrett), optics (Hecht), and serious optimization. Rendering is embarrassingly parallel - perfect HPC domain. Ray tracing for metrology simulation.
- 🛍️ Diestel – Graph Theory