Daily SOTA Pulse Feed
High-velocity intelligence scanned from 550+ daily preprints and pruned by Apple Silicon local ML.
FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-Precision
Exploits asynchronous Hopper Tensor Memory Accelerator (TMA) and warp-group GEMMs to hide memory latency. Delivers 1.8x speedup over FlashAttention-2 at FP16 and up to 740 TFLOPS at FP8.
Dual-Memory Linear Recurrence with Test-Time Associative Binding
Replaces quadratic multi-head attention with a dual-chunk recurrent linear attention kernel. Evaluated on 128k context windows with sub-quadratic O(chunk_size) memory scaling. 3.2x faster decode speed and reduces KV cache from 18.4GB to 1.9GB on 70B parameter models.
DeepSeek-V3 Multi-Head Latent Attention & FP8 Triton Kernel Deconstruction
Exhaustive reverse-engineering of Multi-Head Latent Attention (MLA) and low-precision FP8 Mixed Precision training kernels. Compresses KV-cache by 73.2% while matching standard MHA expressivity via low-rank joint key-value projection.
Speculative Token Decoupling: Asynchronous Verification via Flash-Decoding
Asynchronous speculative draft verification decoupling small draft model token proposals from large model verification passes, unlocking 2.8x wall-clock latency reduction on frontier reasoning tasks.
BitNet b1.58: 1-Bit LLM Architectures with Dynamic Precision Scaling
Practical post-training quantization pipeline mapping dense weights to ternary {-1, 0, 1} states with dynamic FP8 activations. Enables 70B models to execute within 36GB unified memory without perplexity degradation.
Moonlight: Low-Rank MoE Architecture with Dynamic Routing Calibration
Sparse mixture-of-experts model enforcing dynamic auxiliary-loss-free routing. Keeps 94% of weights inactive during inference while preserving dense model reasoning performance at 3.5x lower FLOPS per token.
KAN: Kolmogorov-Arnold Networks as Interpretable Alternatives to MLPs
Replaces fixed linear weight matrices in MLPs with learnable spline-parameterized activation functions on edges. Yields dramatic parameter efficiency on physical PDE simulation and symbolic mathematical discovery tasks.
HyperAttention-2: Exact Linear Attention with Bounded Precision Loss
Approximates attention matrices via randomized locality-sensitive hashing with non-uniform row sampling. Establishes theoretical bounds on spectral distortion under INT8 precision constraints.
Dual-Memory Linear Recurrence with Test-Time Associative Binding
We introduce Dual-Memory Linear Recurrence (DMLR), replacing quadratic softmax self-attention with an orthogonal dual-state memory accumulator. DMLR achieves constant O(1) inference VRAM footprint during auto-regressive generation, fitting a 131k context window in 24GB VRAM while outperforming Llama-3.3-70B on SWE-bench Verified (+4.2%) and MATH-500 (+6.1%). Model weights and Triton kernels are fully open-sourced.