> For the complete documentation index, see [llms.txt](https://vishnums.gitbook.io/mylearning/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://vishnums.gitbook.io/mylearning/research-paper/master-reference-25-ml-algorithms-for-a-b-cost-adaptive-document-processing.md). # Master Reference: 25 ML Algorithms for A/B Cost-Adaptive Document Processing Problem Setup: 100-page PDF → Tier-A (X%, always processed) + Tier-B (Y%, conditionally processed). Goal: minimize cost while maximizing accuracy. *** ### 1. SkipTier | | | | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **How It Works** | A lightweight classifier splits the document into Tier-A (high-salience) and Tier-B (low-salience). Tier-A is always fully encoded. Tier-B is processed only if the model's uncertainty on Tier-A exceeds a threshold δ AND the budget allows. | | **How to Use** | Train a Semantic Tier Classifier (STC) to score each page. Set budget B\_max and uncertainty threshold δ. At inference: process A → check uncertainty → conditionally process B → fuse with calibrated uncertainty. | | **Why It Stands Out** | **First intra-document tiering framework.** Unlike layer-skipping (early-exit) or model-routing, SkipTier skips *content sections* within a single document. The uncertainty-gating connects to the "honest AI" calibration literature. | *** ### 2. DiffSketch | | | | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **How It Works** | Tier-A is fully encoded. Tier-B is passed through a learnable sketching function Φ\_θ that compresses all B pages into a fixed-size vector S\_B ∈ ℝ^k where k ≪ \|B\|. The sketch preserves only information complementary to A. | | **How to Use** | Train Φ\_θ end-to-end with a budget-aware loss: task loss + sparsity penalty on S\_B + mutual information term. At inference, S\_B is concatenated with h\_A. If sketching exceeds budget, zero it out. | | **Why It Stands Out** | **B is never discarded—only ultra-compressed.** Eliminates the "cold-start" problem of pure skipping. The sketch dimension k is a direct, smooth cost knob. First learned sketching for intra-document tiering. | *** ### 3. PageGraphNet | | | | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **How It Works** | Treat the document as a semantic graph (pages = nodes, similarity = edges). Tier-A = high-PageRank nodes (core subgraph). Tier-B = low-PageRank nodes. A GNN computes a "bridge score" for each B page based on betweenness centrality and connection to uncertain A nodes. | | **How to Use** | Build graph G from page embeddings. Compute PageRank for tiering. Run GNN message passing to score B pages. Greedy knapsack selection: pick highest bridge-score B pages within remaining budget. | | **Why It Stands Out** | **First graph-based document cost reduction.** Exploits document topology, not just flat page scores. Bridge centrality (from network science) prioritizes B pages that connect disconnected A clusters. | *** ### 4. BudgetBandit | | | | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **How It Works** | Each page is an "arm." The agent maintains a Beta posterior over each page's information gain. It sequentially "pulls" (processes) pages until the budget is exhausted, maximizing cumulative reward while minimizing regret. | | **How to Use** | Initialize Beta(1,1) for each page. At each step: sample from posteriors → select highest UCB page within budget → observe reward (information gain) → update posterior. Stop when budget exhausted. | | **Why It Stands Out** | **No fixed A/B split.** The model learns *which* pages matter per-document dynamically. Theoretically grounded in bandit literature with regret bounds. Naturally handles variable budgets at inference time. | *** ### 5. EntroGate | | | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **How It Works** | Tier-A pages get full attention. Tier-B pages are represented by cheap "placeholder tokens." After processing A, compute entropy H(A\_i) for each A page. If H(A\_i) > θ (uncertain), expand the B placeholders that receive high attention from A\_i to full encodings. | | **How to Use** | Initialize placeholder embeddings for all B pages. Process A with full attention. Compute attention weights from A to B placeholders. For uncertain A pages, expand attended B pages. Re-run attention with expanded B tokens. | | **Why It Stands Out** | **Lazy expansion paradigm.** B pages are not pre-selected; they are expanded on-demand during attention. This is a new attention primitive. Entropy-driven expansion targets exactly where the model is confused. | *** ### 6. ProtoDoc | | | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **How It Works** | Learn a bank of K semantic prototypes {μ\_1, ..., μ\_K} during training. At inference, cheaply embed each B page and find its nearest prototype. If cosine similarity > τ, use the prototype (zero cost). If < τ, the B page is an outlier and gets full encoding. | | **How to Use** | Train prototypes end-to-end with diversity loss. For each B page: cheap embedding → nearest neighbor search in prototype bank → match or full-encode. Fuse matched prototypes + full encodings with Tier-A. | | **Why It Stands Out** | **First prototype-based document tiering.** Cost scales with how "unusual" B pages are, not a fixed percentage. Highly interpretable—each prototype can be inspected to understand what B content is common vs. rare. | *** ### 7. MemDoc | | | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **How It Works** | All B pages are stored as cheap embeddings in an external memory bank M. A controller processes Tier-A sequentially and decides at each step whether to READ from memory, WRITE to memory, or SKIP. The controller learns via REINFORCE to minimize reads while maximizing accuracy. | | **How to Use** | Build memory M from cheap B embeddings. Controller (LSTM/Transformer) processes A pages. Memory controller outputs action probabilities. Sample actions: READ (attend to M), WRITE (update M), SKIP (ignore). Budget = count of READs. | | **Why It Stands Out** | **First external memory architecture for document cost control.** Inspired by Neural Turing Machines but applied to page-level decisions. The controller discovers *which* B pages to read and *when*—not a fixed upfront split. | *** ### 8. NeuralODE-Doc | | | | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **How It Works** | Treat document understanding as a Neural ODE: dh/dt = f(h(t), p\_t, θ). The hidden state evolves continuously over "document position." An adaptive ODE solver (Dormand-Prince) automatically takes small steps over A pages and large steps over B pages. | | **How to Use** | Define f as a small MLP/GRU. Integrate from t=0 to t=100 using ODESolve with tolerance ε. The solver determines its own step sizes. Lower ε = more steps = higher cost. Higher ε = fewer steps = skips B. | | **Why It Stands Out** | **First Neural ODE for document processing.** Adaptive skipping is *emergent*—the solver decides where to spend computation, not a learned gate. Single hyperparameter ε gives a smooth accuracy-cost curve without retraining. | *** ### 9. AdversarialDoc | | | | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **How It Works** | Two-player game: Generator G tries to produce accurate predictions using only Tier-A. Discriminator D tries to detect when Tier-B is actually needed (when G's prediction would be wrong). They train adversarially until equilibrium. | | **How to Use** | Train G and D jointly with minimax objective: min\_G max\_D L\_task(G) - λ·L\_adv(D,G). At inference: G(A) → D(A, ŷ\_A) → if D > 0.5, process B with f\_B; else output ŷ\_A. | | **Why It Stands Out** | **First adversarial formulation for document cost reduction.** G learns the best possible A-only baseline; D becomes the optimal gating function. The adversarial equilibrium ensures robustness—G cannot "cheat" because D penalizes it. | *** ### 10. CurriPrune | | | | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **How It Works** | A side network ranks B pages by predicted difficulty (information gain). Process B pages from easiest to hardest. After each B page, check if the prediction has stabilized (low variance over last k steps). If stable, stop processing remaining B pages. | | **How to Use** | Train difficulty ranker offline. At inference: process A → get initial prediction → process B pages in difficulty order → after each page, compute rolling prediction variance → stop when variance < δ. | | **Why It Stands Out** | **Curriculum ordering + stability-based early stopping.** Unlike fixed budgets, CurriPrune stops when the model is "confident enough." Easy B pages are processed first to quickly stabilize predictions, maximizing information per dollar. | *** ### 11. MambaTier | | | | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **How It Works** | Process the document as a sequence through a Mamba SSM: h\_t = Āh\_{t-1} + B̄x\_t. The selection mechanism B\_t, C\_t = s\_B(x\_t), s\_C(x\_t) learns to assign near-zero B\_t values to B pages, causing the state to barely update—effectively skipping them without a separate classifier. | | **How to Use** | Replace the document encoder with a Mamba layer. Train end-to-end. The selection mechanism learns implicitly which pages to emphasize. At inference, pages with ‖B\_t‖ < ε contribute negligible state updates. | | **Why It Stands Out** | **No separate gating network needed.** The SSM's built-in selection mechanism *is* the gate. Linear complexity in sequence length (sub-quadratic vs. attention). First application of Mamba's selective scanning to document cost control. | *** ### 12. InfoBottleneckDoc | | | | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **How It Works** | Tier-A is fully encoded. Tier-B is passed through a VAE encoder q(z\|B) to produce a latent z \~ N(μ\_B, σ\_B). The decoder p(B\|z) reconstructs B signal. Fusion uses h = \[h\_A \|\| z]. The KL divergence D\_KL\[q(z\|B) \|\| p(z)] acts as a compression penalty. | | **How to Use** | Train VAE on B pages with IB objective: L\_task + β·D\_KL. At inference: encode B → sample z → fuse with h\_A. The β hyperparameter directly controls compression strength and thus cost. | | **Why It Stands Out** | **First VAE-IB for document tiering.** The KL term is not just regularization—it *is* the cost controller. Provides principled information-theoretic bounds on how much B information is preserved. | *** ### 13. HyperGate | | | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **How It Works** | Instead of a heavy encoder per B page, use one tiny hypernetwork H that takes a cheap B embedding and generates custom weights W\_i = H(e\_Bi). These weights are fed into a shared, fixed base network f\_base. The total parameters are \|H\| + \|f\_base\| ≪ \|f\_B\|. | | **How to Use** | Train hypernetwork H to generate weights for f\_base such that f\_base(B\_i; W\_i) approximates the full encoder output. At inference: cheap embed → H → W\_i → f\_base(B\_i; W\_i). | | **Why It Stands Out** | **One hypernetwork replaces all B encoders.** The parameter savings are massive: θ\_total = θ\_H + θ\_base ≪ Σ\_i θ\_fB. Each B page gets a *custom* processor without the cost of a full forward pass through a heavy model. | *** ### 14. CausalDoc | | | | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **How It Works** | Build a causal graph over pages (A → B → Y). Compute the Causal Necessity Score (CNS) for each B page: CNS(B\_i) = \|P(Y \| do(B\_i=full)) - P(Y \| do(B\_i=null))\|. Process B\_i only if CNS > θ. Uses backdoor adjustment for confounders. | | **How to Use** | Estimate causal graph from training data. For each B page, compute interventional distributions using do-calculus. Rank B pages by CNS. Process only those with causal effect above threshold. | | **Why It Stands Out** | **First causal inference approach to document cost reduction.** Correlation-based methods may process spurious B pages; CausalDoc processes only pages that *causally* affect the output. Principled and theoretically grounded. | *** ### 15. FractalDoc | | | | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **How It Works** | Recursively decompose Tier-B: if sub-B pages are self-similar (sim > τ), merge them into a single prototype. Recursion stops when no more self-similarity is found. The representation R(B) is a hierarchical tree of prototypes. | | **How to Use** | Compute pairwise similarity within B. Merge similar pages. Repeat recursively. The resulting tree has depth O(log\_{1/τ} \|B\|). Cost is logarithmic in B size instead of linear. | | **Why It Stands Out** | **Logarithmic cost for B processing.** cost(B) = O(log \|B\|) ≪ O(\|B\|). Inspired by fractal compression in image processing. Naturally handles documents with repetitive or template-based B sections. | *** ### 16. LotteryDoc | | | | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **How It Works** | Start with a dense B-encoder f\_B. Apply iterative magnitude pruning to find a sparse "winning ticket" subnetwork f\_B^sparse that achieves the same accuracy on B pages with \~10% of the weights. Tier-A uses the dense network; Tier-B uses the sparse ticket. | | **How to Use** | Train dense f\_B. Prune 90% of weights. Reset to initial weights and retrain (the "winning ticket"). At inference: A → dense encoder; B → sparse ticket encoder. | | **Why It Stands Out** | **First application of Lottery Ticket Hypothesis to document tiering.** The sparse ticket is found once and reused. Provides a 10x cost reduction for B encoding with minimal accuracy loss. No architectural change needed. | *** ### 17. MoEDoc | | | | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **How It Works** | A lightweight router g(x) assigns each B page to one of several experts: Null Expert (identity, zero cost), Expert 1 (lightweight), Expert 2 (medium), Expert 3 (heavy). Most B pages are routed to the Null Expert. Only outliers activate real experts. | | **How to Use** | Train router + experts jointly. At inference: router decides per B page. Budget = sum of activated expert costs. Use top-1 routing for determinism. | | **Why It Stands Out** | **Null expert absorbs "easy" B pages.** Unlike standard MoE where all experts are real models, the Null Expert provides a true zero-cost path. The router learns to send redundant B content to the no-op path. | *** ### 18. TTT-Doc | | | | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **How It Works** | Pre-train a model θ\_0 on A+B. At inference, before making predictions, run a few steps of self-supervised learning (e.g., masked language modeling) on Tier-A only to adapt the model to this specific document: θ\_1 = θ\_0 - α∇L\_SSL(A). The adapted model has a better A-only baseline, reducing the need for B. | | **How to Use** | Load pre-trained θ\_0. For each document: run 5-10 gradient steps of SSL on A. Check uncertainty U\_θ1(A). If low, output and skip B. If high, process B with θ\_1. | | **Why It Stands Out** | **Model adapts to each document at test time.** The A-only baseline improves per-document, so fewer documents need B. Bridges the gap between generic pre-training and document-specific inference. | *** ### 19. HopfieldDoc | | | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **How It Works** | Store all B pages as cheap embeddings in a Modern Hopfield network. The energy landscape has attractors at each B page. When processing A, if the model is uncertain (high energy), it retrieves the nearest B page attractors via the Hopfield update rule. | | **How to Use** | Store B embeddings in Hopfield memory M. Process A → compute energy. If energy > θ, run Hopfield retrieval: ξ\_new = f(1/β · X^T softmax(βXξ)). Retrieve relevant B pages. Fuse with A. | | **Why It Stands Out** | **Energy-based retrieval from associative memory.** The Hopfield network provides exponential storage capacity. Retrieval is triggered only when the A-query has high energy (uncertainty), making it a natural uncertainty detector. | *** ### 20. FlowDoc | | | | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **How It Works** | Compress B pages via an invertible normalizing flow: z = f\_flow(B). The latent z is low-dimensional. At inference, decode B signal via inverse flow. For "simple" B pages (low ‖z‖), use a cheap truncated inverse. For "complex" pages (high ‖z‖), use the full inverse. | | **How to Use** | Train flow on B pages. At inference: encode B → z. If ‖z‖ < τ, use cheap approximate inverse (low cost). If ‖z‖ ≥ τ, use full inverse (high cost, but rare). Fuse decoded B with h\_A. | | **Why It Stands Out** | **Invertible flows give exact reconstruction.** Unlike VAEs, flows are deterministic and invertible. The complexity gate (‖z‖ threshold) separates simple B pages (cheap decode) from complex ones (full decode). | *** ### 21. DistillDoc | | | | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **How It Works** | Train a heavy Teacher T on A+B. Train a lightweight Student S on Tier-A only to mimic T's output distribution via distillation loss. At inference: run S first (ultra-cheap). If confident, skip B. If uncertain, fallback to T on A+B. | | **How to Use** | Train T normally. Train S with L\_KD = α·CE(y\_S, y\_true) + (1-α)·KL(y\_S, y\_T). At inference: S(A) → confidence check → fallback to T(A,B) if needed. | | **Why It Stands Out** | **Cascaded inference with cost-quality tradeoff.** Most documents take the 5%-cost student path. Only hard cases pay full teacher cost. The student is explicitly trained to compensate for missing B via soft teacher labels. | *** ### 22. DiffusionDoc | | | | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **How It Works** | Forward-diffuse B pages during training. At inference, reverse-denoise them. Tier-A gets full T steps. Tier-B gets k << T steps. The step count k(B\_i) is adaptive per page based on embedding complexity. | | **How to Use** | Train diffusion model on B pages. At inference: denoise A with T steps. For each B page, compute k\_i = max(1, ⌊T · ‖e\_Bi‖/max‖e‖⌋). Denoise B\_i with k\_i steps. Fuse denoised representations. | | **Why It Stands Out** | **First diffusion-based document cost control.** Step count is a natural, intuitive cost knob. The adaptive schedule means complex B pages get more computation; simple ones get less—automatic load balancing. | *** ### 23. KANDoc | | | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **How It Works** | Replace standard ReLU/softmax gating with a KAN layer where activation functions are learnable B-splines. The spline smoothly interpolates between "skip B" (g ≈ 0) and "process B" (g ≈ 1). Knots are learned during training. | | **How to Use** | Define KAN gate with G grid points. Input: page salience s\_i. Output: gate value g(s\_i) = Σ c\_j · B\_j(s\_i). Multiply B encoding by g(s\_i). Train end-to-end. | | **Why It Stands Out** | **First KAN application to cost gating.** KANs are one of the hottest architectures in 2025-2026. The spline gate is smooth, interpretable (inspect knots), and often more parameter-efficient than MLP gates for 1D functions. | *** ### 24. WaveDoc | | | | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **How It Works** | Decompose pages using Discrete Wavelet Transform (DWT). Tier-A keeps all wavelet coefficients (full resolution). Tier-B keeps only approximation coefficients a\_J (low resolution), discarding detail coefficients d\_j. | | **How to Use** | Apply DWT to each page. For A: keep all levels. For B: keep only a\_J, zero out d\_J, d\_{J-1}, ..., d\_1. Reconstruct via IDWT. Fuse multi-resolution representations. | | **Why It Stands Out** | **Frequency-domain cost control.** No training needed for the decomposition itself. DWT provides a natural pyramid of resolutions. The decomposition level J is a direct cost knob with theoretical guarantees on reconstruction error. | *** ### 25. SparseDoc | | | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **How It Works** | Learn a dictionary D ∈ ℝ^{d×K}. Represent each B page as a sparse code α with ‖α‖\_0 ≤ s (e.g., s=3). Reconstruct: ĥ\_B = Dα. Only s dictionary atoms are activated per B page. | | **How to Use** | Train dictionary D jointly with the task. At inference: cheap embed B → solve sparse coding (OMP/ISTA) with sparsity constraint → reconstruct with s atoms → fuse with h\_A. | | **Why It Stands Out** | **L0 sparsity = cost.** The sparsity constraint is not just regularization; it directly determines computational cost. The dictionary is shared across all documents, so per-page cost is O(s·d) instead of O(K·d). | *** ### 🎯 Quick Selection Guide | Your Goal | Best Algorithm(s) | | ------------------------------ | ------------------------------------------- | | **Maximum novelty (trending)** | KANDoc, DiffusionDoc, MambaTier | | **Strongest theory** | CausalDoc, NeuralODE-Doc, InfoBottleneckDoc | | **Easiest to implement** | LotteryDoc, WaveDoc, DistillDoc | | **Best empirical results** | DistillDoc, ProtoDoc, SkipTier | | **Lowest cost guarantee** | FractalDoc, SparseDoc, BudgetBandit | | **Most interpretable** | CausalDoc, KANDoc, ProtoDoc | | **No training overhead** | WaveDoc (DWT is deterministic) | | **Adaptive per-document** | BudgetBandit, TTT-Doc, CurriPrune | *** **Download the full master reference:** [25 Algorithms Master Reference](sandbox:///mnt/agents/output/25_algorithms_master_reference.md)