I am an open source contributor, working on load balancing for Bifrost (LLM gateway) and ran into some interesting challenges with Go implementation. Standard weighted round-robin works fine for static loads, but LLM providers behave weirdly. OpenAI might be fast at 9am, slow at 2pm. Azure rate limits kick in unexpectedly. One region degrades while others stay healthy. Built adaptive routing that adjusts weights based on live metrics - latency, error rates, throughput. Used EWMAs (exponentially weighted moving averages) to smooth out spikes without overreacting to noise. The Go part that was tricky: tracking per-provider metrics without locks becoming a bottleneck at high RPS. Ended up using atomic operations for counters and a separate goroutine that periodically reads metrics and recalculates weights. Keeps the hot path lock-free. Also had to handle provider health scoring. Not just "up or down" but scoring based on recent performance. A provider recovering from issues should gradually earn traffic back, not get slammed immediately. Connection pooling matters more than expected. Go's http.Transport reuses connections well, but tuning MaxIdleConnsPerHost made a noticeable difference under sustained load. Running this at 5K RPS with sub-microsecond overhead now. The concurrency primitives in Go made this way easier than Python would've been. Anyone else built adaptive routing in Go? What patterns worked for you? submitted by /u/dinkinflika0 [link] [comments]

My boss wants to have an LLM in house for document searches. I've convinced him that we'll only use it for identifying relevant documents due to the risk of hallucinations, and not perform calculations and the like. So for example, finding all PDF files related to customer X, product Y between 2023-2025. Because of legal concerns it'll have to be hosted locally and air gapped. I've only used Gemini. Does anyone have experience or suggestions about picking a vendor for this type of application? I'm familiar with CNNs but have zero interest in building or training a LLM myself. submitted by /u/Few-Strawberry2764 [link] [comments]

I spent a few days working on a case study for a company and they completely ghosted me after I submitted it. It’s incredibly frustrating because I could have used that time for something more productive. With how bad the job market is, it feels like there’s no real choice but to go along with these ridiculous interview processes. The funniest part is that I didn’t even apply for the role. They reached out to me on LinkedIn. I’ve decided that from now on I’m not doing case studies as part of interviews. Do any of you say no to case studies too? submitted by /u/Lamp_Shade_Head [link] [comments]

Hi everyone! Please bear with me with this question 🫣 I’m looking for someone in research to pick their brain about the similarities and differences between statistical learning in cognitive science and in machine learning, so definition, conceptual differences/similarities, predictions, testing…. Hope it makes sense, I’m doing research in cognitive sciences and I’d love to learn more about this term’s use in ML for a review I’m working on 🙂 thanks! submitted by /u/Ok_Fudge1993 [link] [comments]

Just read through this new systematic review (arXiv:2601.09334) on Serverless for HPC/AI. It’s a solid read if you're dealing with infrastructure scaling. The TL;DR: Static Allocation is breaking: The paper argues that rigid GPU clusters can't handle modern "bursty" AI workloads efficiently. You either over-provision (waste money) or under-provision (crash during spikes). Serverless is the fix: The industry is moving toward elastic, serverless execution models to survive the efficiency gap. We've been seeing this exact pattern in production. We actually built our engine specifically to solve that Cold Start problem via state snapshotting, so it's validating to see the academic side converging on the same architecture. Paper link: https://arxiv.org/abs/2601.09334 Anyone seeing this shift from static -> serverless in their own clusters? submitted by /u/pmv143 [link] [comments]

Results out for ISBI 2026 - London a few days back. Just want to check with fellow medical imaging peeps on how did it go for all. Results were delayed by a month and I see a pretty high acceptance rate this time. submitted by /u/ade17_in [link] [comments]

submitted by /u/idan_huji [link] [comments]

Have a Google Sr. DS interview coming up in a month. Has anyone taken it? tips? submitted by /u/No-Mud4063 [link] [comments]

TL;DR: The paper describes a mechanism that essentially turns the context window into a training dataset for a "fast weight" update loop: Inner Loop: The model runs a mini-gradient descent on the context during inference. It updates specific MLP layers to "learn" the current context. Outer Loop: The model's initial weights are meta-learned during training to be "highly updateable" or optimized for this test-time adaptation From the Paper: "Overall, our empirical observations strongly indicate that TTT-E2E should produce the same trend as full attention for scaling with training compute in large-budget production runs." Abstract: We formulate long-context language modeling as a problem in continual learning rather than architecture design. Under this formulation, we only use a standard architecture a Transformer with sliding-window attention. However, our model continues learning at test time via next-token prediction on the given context, compressing the context it reads into its weights. In addition, we improve the model's initialization for learning at test time via meta-learning at training time. Overall, our method, a form of Test-Time Training (TTT), is End-to-End (E2E) both at test time (via next-token prediction) and training time (via meta-learning), in contrast to previous forms. We conduct extensive experiments with a focus on scaling properties. In particular, for 3B models trained with 164B tokens, our method (TTT-E2E) scales with context length in the same way as Transformer with full attention, while others, such as Mamba 2 and Gated DeltaNet, do not. However, similar to RNNs, TTT-E2E has constant inference latency regardless of context length, making it 2.7x faster than full attention for 128K context. Our code is publicly available. Layman's Explanation: Think of this paper as solving the memory bottleneck by fundamentally changing how a model processes information. Imagine you are taking a massive open-book exam. A standard Transformer (like GPT-4) is the student who frantically re-reads every single page of the textbook before answering every single question. This strategy guarantees they find the specific details (perfect recall), but as the textbook gets thicker, they get exponentially slower until they simply cannot finish the test in time. On the other hand, alternatives like RNNs or Mamba try to summarize the entire textbook onto a single index card. They can answer questions instantly because they don't have to look back at the book, but for long, complex subjects, they eventually run out of space on the card and start forgetting crucial information. This new method, Test-Time Training (TTT), changes the paradigm from retrieving information to learning it on the fly. Instead of re-reading the book or summarizing it onto a card, the TTT model treats the context window as a dataset and actually trains itself on it in real-time. It performs a mini-gradient descent update on its own neural weights as it reads. This is equivalent to a student who reads the textbook and physically rewires their brain to master the subject matter before the test. Because the information is now compressed into the model's actual intelligence (its weights) rather than a temporary cache, the model can answer questions instantly (matching the constant speed of the fast index-card models) but with the high accuracy and scaling capability of the slow, page-turning Transformers. This effectively decouples intelligence from memory costs, allowing for massive context lengths without the usual slowdown. Link to the Paper: https://arxiv.org/pdf/2512.23675 Link to the Open-Sourced Official Implementation of End-to-End Test Time Training for Long Context: https://github.com/test-time-training/e2e submitted by /u/44th--Hokage [link] [comments]

I have limited python proficiency but I can code well with R. I want to design a project that’ll require me to collect patient data from the All of Us database. Does this sound like an unrealistic plan with my limited python proficiency? submitted by /u/phymathnerd [link] [comments]

I Work on Bifrost (been posting a lot here lol) and wanted to share what we learned building multi-provider routing, since it's messier than it seems. Github: https://github.com/maximhq/bifrost Initially thought weighted routing would be the main thing - like send 80% of traffic to Azure, 20% to OpenAI. Pretty straightforward. Configure weights, distribute requests proportionally, done. But production is messier. Providers go down regionally. Rate limits hit unexpectedly. Azure might be healthy in US-East but degraded in EU-West. Or you hit your tier limit mid-day and everything starts timing out. So we built automatic fallback chains. When you configure multiple providers on a virtual key, Bifrost sorts them by weight and creates fallbacks automatically. Primary request goes to Azure, fails, immediately retries with OpenAI. Happens transparently - your app doesn't see it. The health monitoring part was interesting. We track success rates, response times, error patterns per provider. When issues get detected, requests start routing to backup providers within milliseconds. No manual intervention needed. Also handles rate limits differently now. If a provider hits TPM/RPM limits, it gets excluded from routing temporarily while other providers stay available. Prevents cascading failures. One thing that surprised us - weighted routing alone isn't enough. You need adaptive load balancing that actually looks at real-time metrics (latency, error rates, throughput) and adjusts on the fly. Static weights don't account for degradation. The tricky part was making failover fast enough that it doesn't add noticeable latency. Had to optimize connection pooling, timeout handling, and how we track provider health. how are you folks handling multi-provider routing in production. Static configs? Manual switching? Something else? submitted by /u/dinkinflika0 [link] [comments]

Hi everyone I’m a fellowship-trained neurosurgeon / spine surgeon. I’ve been discussing a persistent problem in our field with other surgeons for a while, and I wanted to run it by people who think about ML systems, not just model performance. I’m trying to pressure-test whether a particular approach is even technically sound, where it would break, and what I’m likely underestimating. Id love to find an interested person to have a discussion with to get a 10000 feet level understanding of the scope of what I am trying to accomplish. The clinical problem: For the same spine pathology and very similar patient presentations, you can see multiple reputable surgeons and get very different surgical recommendations. anything from continued conservative management to decompression, short fusion, or long multilevel constructs. Costs and outcomes vary widely. This isn’t because surgeons are careless. It’s because spine surgery operates with: Limited prospective evidence Inconsistent documentation Weak outcome feedback loops Retrospective datasets that are biased, incomplete, and poorly labeled EMRs are essentially digital paper charts. PACS is built for viewing images, not capturing decision intent. Surgical reasoning is visual, spatial, and 3D, yet we reduce it to free-text notes after the fact. From a data perspective, the learning signal is pretty broken. Why I’m skeptical that training on existing data works: “Labels” are often inferred indirectly (billing codes, op notes) Surgeon decision policies are non-stationary Available datasets are institution-specific and access-restricted Selection bias is extreme (who gets surgery vs who doesn’t is itself a learned policy) Outcomes are delayed, noisy, and confounded Even with access, I’m not convinced retrospective supervision converges to something clinically useful. The idea I’m exploring: Instead of trying to clean bad data later, what if the workflow itself generated structured, high-fidelity labels as a byproduct of doing the work, or at least the majority of it? Concretely, I’m imagining an EMR-adjacent, spine-specific surgical planning and case monitoring environment that surgeons would actually want to use. Not another PACS viewer, but a system that allows: 3D reconstruction from pre-op imaging Automated calculation of alignment parameters Explicit marking of anatomic features tied to symptoms Surgical plan modeling (levels, implants, trajectories, correction goals) Structured logging of surgical cases (to derive patterns and analyze for trends) Enable productivity (generate note, auto populate plans ect.) Enable standardized automated patient outcomes data collection. The key point isn’t the UI, but UI is also an area that currently suffers. It’s that surgeons would be forced (in a useful way) to externalize decision intent in a structured format because it directly helps them plan cases and generate documentation. Labeling wouldn’t feel like labeling it would almost just be how you work. The data used for learning would explicitly include post-operative outcomes. PROMs collected at standardized intervals, complications (SSI, reoperation), operative time, etc, with automated follow-up built into the system. The goal would not be to replicate surgeon decisions, but to learn decision patterns that are associated with better outcomes. Surgeons could specify what they want to optimize for a given patient (eg pain relief vs complication risk vs durability), and the system would generate predictions conditioned on those objectives. Over time, this would generate: Surgeon-specific decision + outcome datasets Aggregate cross-surgeon data Explicit representations of surgical choices, not just endpoints Learning systems could then train on: Individual surgeon decision–outcome mappings Population-level patterns Areas of divergence where similar cases lead to different choices and outcomes Where I’m unsure, and why I’m posting here: From an ML perspective, I’m trying to understand: Given delayed, noisy outcomes, is this best framed as supervised prediction or closer to learning decision policies under uncertainty? How feasible is it to attribute outcome differences to surgical decisions rather than execution, environment, or case selection? Does it make sense to learn surgeon-specific decision–outcome mappings before attempting cross-surgeon generalization? How would you prevent optimizing for measurable metrics (PROMs, SSI, etc) at the expense of unmeasured but important patient outcomes? Which outcome signals are realistically usable for learning, and which are too delayed or confounded? What failure modes jump out immediately? I’m also trying to get a realistic sense of: The data engineering complexity this implies Rough scale of compute once models actually exist The kind of team required to even attempt this (beyond just training models) I know there are a lot of missing details. If anyone here has worked on complex ML systems tightly coupled to real-world workflows (medical imaging, decision support, etc) and finds this interesting, I’d love to continue the discussion privately or over Zoom. Maybe we can collaborate on some level! Appreciate any critique especially the uncomfortable kind!! submitted by /u/LaniakeaResident [link] [comments]

Methodology: 10 frontier models (Claude Opus/Sonnet 4.5, o1, GPT-4o, Gemini 3 Pro, Grok 4, DeepSeek V3.2, Llama 4 Scout, Mistral Large, Command A) Each answers identical prompt blindly All 10 judge all 10 responses (100 judgments) Self-judgments excluded from final scores 5 criteria: Correctness (30%), Completeness (20%), Clarity (20%), Depth (15%), Usefulness (15%) CODE-001 Results (Async Python Debugging): Claude Opus 4.5: 9.49 o1: 9.48 Claude Sonnet 4.5: 9.41 DeepSeek V3.2: 9.39 Grok 4: 9.37 Command A: 9.23 Gemini 3 Pro: 9.19 Mistral Large: 9.10 GPT-4o: 8.79 Llama 4 Scout: 8.04 REASON-001 Results (Two Envelope Paradox): Claude Opus 4.5: 9.24 o1: 9.23 Claude Sonnet 4.5: 9.09 DeepSeek V3.2: 8.93 Grok 4: 8.88 GPT-4o: 8.75 Gemini 3 Pro: 8.68 Mistral Large: 8.64 Command A: 8.38 Llama 4 Scout: 7.92 Judge Bias Patterns: Strictest: Claude Opus (avg 7.10-8.76 depending on task) Most lenient: Mistral Large (9.22-9.73) Correlation: Strict judges tend to score higher themselves Open questions for feedback: Is 5-point rubric weighting optimal for different task types? Should we normalize for judge harshness before aggregating? Are 9 judgments per response sufficient for statistical validity? Full data + prompts: https://themultivac.substack.com Daily evals at themultivac.com — currently in Phase 2 (peer matrix format). submitted by /u/Silver_Raspberry_811 [link] [comments]

I know most will wonder why I’m wasting my time training at only 19k tok a sec. It’s because I can. I’m doing this in my living room in my spare time. 0 formal ML experience. The absurd amount I’ve learned in the last few months made me realize I really picked the wrong career. My Mixture of Experts is 2.36B parameter with 8 routed experts plus a shared expert using top-2 routing. Attention is Grouped Query Attention with QK-normalization and RoPE positional embeddings. All feed-forward layers use SwiGLU activation with RMSNorm throughout. Load balancing follows DeepSeek V3’s auxiliary-loss-free approach using bias-based routing. I monitor coefficient of variation and maximum violation per step. Training runs on TorchAO FP8 quantization with the Muon optimizer and a multi-stage learning rate schedule (warmup, constant, cosine decay). The backend is optimized for Blackwell architecture with cuBLASLt. The data pipeline implements MeCo (Metadata Conditioning then Cooldown) with ledger-based deterministic sampling. I have document-aware attention masking and cross-document loss masking but was disabled for the initial MeCo run. I have since disabled MeCo and curated a clean corpus with no tagging of any kind. MeCo worked but it worked too well and with only 8 experts, it became very problematic. My two biggest early mistakes were not using symmetric router initialization (std=0.006) and not having a dense first layer. Cost me a lot of time and sleep. So what did I do? I cheated. I used aux loss of .003 snd ema smoothing at the beginning. I just didn’t know better. I paid a price later on for that. DO NOT use router scaling on a small MoE. DeepSeek used 2.5. Kimi K2 used 2.446. I tried 1.2 and it was horribly unstable and violation blew up to over .500. 24 batch 6 Grad LR 3e-4 AdamW+Muon Scaled. Bias .001 Aux .0001. I update every step. As of yesterday: 2026-01-13 20:53:06 step 41915 | lr 3.00e-04 | loss 1.8867 | gnorm 0.13 | 19,415 tok/s (ema 19,553) | 75.9s/5 steps | cv 0.022 | bias -0.001708±0.179996 | rel_max=0.036 maxvio=0.027 ent=1.203 applied=True | seq_aux 2.444 2026-01-13 20:54:20 [moe] token counts: [150018, 148422, 155402, 147966, 145236, 146724, 144358, 141522] 2026-01-13 20:54:20 step 41920 | lr 3.00e-04 | loss 1.9263 | gnorm 0.13 | 20,102 tok/s (ema 19,828) | 73.4s/5 steps | cv 0.026 | bias -0.001708±0.179920 | rel_max=0.054 maxvio=0.054 ent=1.211 applied=True | seq_aux 2.515 I got a long ways to go 🙂 I’ll gladly answer any question. No gate keeping here. submitted by /u/exhorder72 [link] [comments]

TL;DR: The paper introduces Spectral Sphere Optimizer, which takes steepest descent under spectral norm (Muon) and forces the weights & updates onto a spectral sphere. Paper: https://www.arxiv.org/pdf/2601.08393 Repo: https://github.com/Unakar/Spectral-Sphere-Optimizer Abstract: Scaling large models requires optimization strategies that ensure rapid convergence grounded in stability. Maximal Update Parametrization ( muP) provides a theoretical safeguard for width-invariant theta(1) activation control, whereas emerging optimizers like Muon are only ``half-aligned'' with these constraints: they control updates but allow weights to drift. To address this limitation, we introduce the Spectral Sphere Optimizer (SSO), which enforces strict module-wise spectral constraints on both weights and their updates. By deriving the steepest descent direction on the spectral sphere, SSO realizes a fully muP-aligned optimization process. To enable large-scale training, we implement SSO as an efficient parallel algorithm within Megatron. Through extensive pretraining on diverse architectures, including Dense 1.7B, MoE 8B-A1B, and 200-layer DeepNet models, SSO consistently outperforms AdamW and Muon. Furthermore, we observe significant practical stability benefits, including improved MoE router load balancing, suppressed outliers, and strictly bounded activations. Algorithm: https://preview.redd.it/f1bvi7yd1cdg1.png?width=1197&format=png&auto=webp&s=88a15a375316f54b092e8101e492a2574dc2ace1 Evals: https://preview.redd.it/5hefuy7g1cdg1.png?width=1503&format=png&auto=webp&s=8a0864c5279654a1c9a29b7aae57d2a1b160aa4d https://preview.redd.it/0sy8ih8h1cdg1.png?width=1517&format=png&auto=webp&s=ffd675a60192908ed95652b89540cce8d2110088 https://preview.redd.it/rz6bhc6i1cdg1.png?width=1585&format=png&auto=webp&s=50cd471c7805517d0279877fee235dea3e42954e https://preview.redd.it/fu5wd7zi1cdg1.png?width=1524&format=png&auto=webp&s=5bfb7668a76ceefa320d7325b6abdb731d985e45 submitted by /u/StartledWatermelon [link] [comments]

submitted by /u/idan_huji [link] [comments]

I recently started doing LeetCode to prep for coding interviews. So far I’ve mostly been focusing on arrays, hash maps, strings, and patterns like two pointers, sliding window, and binary search. Should I move on to other topics like stacks, queues, and trees, or is this enough for now? submitted by /u/Lamp_Shade_Head [link] [comments]

Hi everyone, I’m trying to make a deliberate choice between two paths for machine learning and AI development, and I’d really value input from people who’ve used both CUDA GPUs and Apple Silicon. Context I already own a MacBook Pro M1, which I use daily for coding and general work. I’m now considering adding a local CUDA workstation mainly for: Local LLM inference (30B–70B models) Real-time AI projects (LLM + TTS + RVC) Unreal Engine 5 + AI-driven characters ML experimentation and systems-level learning I’m also thinking long-term about portfolio quality and employability (FAANG / ML infra / quant-style roles). Option A — Apple Silicon–first Stick with the M1 MacBook Pro Use Metal / MPS where possible Offload heavy jobs to cloud GPUs (AWS, etc.) Pros I see: efficiency, quiet, great dev experience Concerns: lack of CUDA, tooling gaps, transferability to industry infra Option B — Local CUDA workstation Used build (~£1,270 / ~$1,700): RTX 3090 (24GB) i5-13600K 32GB DDR4 (upgradeable) Pros I see: CUDA ecosystem, local latency, hands-on GPU systems work Concerns: power, noise, cost, maintenance What I’d love feedback on For local LLMs and real-time pipelines, how limiting is Apple Silicon today vs CUDA? For those who’ve used both, where did Apple Silicon shine — and where did it fall short? From a portfolio / hiring perspective, does CUDA experience meaningfully matter in practice? Is a local 3090 still a solid learning platform in 2025, or is cloud-first the smarter move? Is the build I found a good deal ? I’m not anti-Mac (I use one daily), but I want to be realistic about what builds strong, credible ML experience. Thanks in advance — especially interested in responses from people who’ve run real workloads on both platforms. submitted by /u/Individual-School-07 [link] [comments]

I have been trying to solve classification problem on a low resource language. I am doing comparative analysis, LinearSVC and Logistic regression performed the best and the only models with 80+ accuracy and no overfitting. I have to classify it using deep learning model as well. I applied BERT on the dataset, model is 'bert-base-multilingual-cased', and I am fine tuning it, but issue is overfitting. Training logs: Epoch 6/10 | Train Loss: 0.4135 | Train Acc: 0.8772 | Val Loss: 0.9208 | Val Acc: 0.7408 Epoch 7/10 | Train Loss: 0.2984 | Train Acc: 0.9129 | Val Loss: 0.8313 | Val Acc: 0.7530 Epoch 8/10 | Train Loss: 0.2207 | Train Acc: 0.9388 | Val Loss: 0.8720 | Val Acc: 0.7505 this was with default dropout of the model, when I change dropout to 0.3, or even 0.2, model still overfits but not this much, but with dropout I don't go near 60% accuracy, long training introduces overfitting, early stopping isn't working as val loss continuous to decrease. On 10 epoch, I trained patience of 2 and 3. It doesn't stops. To prevent this I am not doing warmup step, my optimizer is below: optimizer = AdamW([ {'params': model.bert.parameters(), 'lr': 2e-5}, {'params': model.classifier.parameters(), 'lr': 3e-5} ], weight_decay=0.01) About my dataset, I have 9000 training samples and 11 classes to train, data is imbalanced but not drastically, to cater this I have added class weights to loss function. 17 words per training sample on average. I set the max_length to 120 for tokens ids and attention masks. How can I improve my training, I am trying to achieve atleast 75% accuracy without overfitting, for my comparative analysis. What I am doing wrong? Please guide me. Data Augmentation didn't work too. I did easy data augmentation. Mixup Augmentation also didn't work. If you need more information about my training to answer questions, ask in the comment, thanks. submitted by /u/Sikandarch [link] [comments]

Hey hey. Like the title says, we are currently building some pretty weird and ambitious systems (think hive-mind/swarm-like collective) and we are growing these to be able to create great RL environments. And we are starting with pufferlib envs. It is doing a pretty damn good job atm. We are currently bootstrapped and we are limited on compute. Even a small batch of gpus (of decent size chips) would be pretty great. If you have any extra gpus laying around, or would potentially want to sponsor us, would love to chat. I am open to any questions in the thread as well. I'm also down to do a decent amount of discovery (need nda ideally). submitted by /u/cobalt1137 [link] [comments]