Hi All, I am brainstorming some kind of a nomenclature for our team so that theres a standard way of naming ML models like their pickle files . Any inputs will be appreciated. thanks submitted by /u/BravoZero6 [link] [comments]

Hi r/MachineLearning community, I'm currently working on the second part of my thesis focused on Named Entity Recognition (NER) for detecting Adverse Drug Events (ADE) in clinical texts. In my first thesis project, I tried to replicate a paper but had to pivot to the n2c2 dataset, which led to challenges in model performance. I've fine-tuned a DeBERTa model with standard practices, but I'm struggling with achieving high accuracy, particularly with precision and recall. This is my first deep dive into a thesis and the world of NLP, and any guidance would be immensely appreciated. Also, any common pitfalls for thesis work or useful resources on this topic would be extremely helpful. I'm eager to learn from the community and improve my research. Thank you so much for your time! submitted by /u/Popsuga [link] [comments]

Through our experiments, we found that LLM agents have a certain ability to understand probability distributions, the LLM agent's sampling ability for probability distributions is lacking and it is difficult to give a behavior sequence that conforms to a certain probability distribution through LLMs alone. We are looking forward to your thoughts, critiques, and discussions on this topic. Full Paper & Citation: You can access the full paper https://arxiv.org/abs/2404.09043. Please cite our work if it contributes to your research. https://preview.redd.it/ai7uks7nluwc1.png?width=935&format=png&auto=webp&s=891dd57ef50d1ee99b1a8b2372b9a460397754d6 submitted by /u/GYX-001 [link] [comments]

Main aim: Style transfer between two discrete timeseries signals. Here are the details: Dataset: Discrete time series. 1700 rows, with 97 percent of it with zeroes. Cannot remove these zeroes as it means something. Values ranging from 0-32 for one of the features in Domain A needs to translated to another feature with same range in domain B. Another feature from 0-5000 from domain A, translated to a different domain B with same range. I can recreate the same dataset multiple times with small variations, so we can have larger datasets. I would create sequences of size 20 or 30 and batch: 32 or 64 initially. Generator Network: A simple encoder with linear layer first hidden size:16 , relu, 2nd linear layer :8 and relu again . A symmetric Decoder . Discriminator: 2 linear layers with hidden size 8 and leaky Relu between them. And sigmoid as final layer. Loss function : BCEloss . Also experimented BCE + MSE loss for generator. Training: I'm using pytorch. Only trained with one feature/signal and tried to generate this feature from noise. Didn't move to cycle consistency yet. With the small dataset training, the discriminator becomes too strong, I even tried to set reduce the learning rate for discriminator as 0.0001 and generator as 0.01 , it didn't work. Tried to add/complicate the layer of generator, still didn't work. Tried to train discriminator every 10th epoch, while the generator trained more. Didn't work. Also tried to normalize the data. I want to explore Adversarial autoencoder /cycle Gan , but the generator is unable to learn anything with vanilla GAN as well. Can someone help or give me some ideas on what I can do ? Thanks submitted by /u/investmentwholesome [link] [comments]

Hi guys, I just tried to run Llama 3 on my Colab(free version) and seems that I ran out of the memory: OutOfMemoryError: CUDA out of memory. Tried to allocate 32.00 MiB. GPU 0 has a total capacity of 14.75 GiB of which 9.06 MiB is free. Process 8863 has 14.74 GiB memory in use. Of the allocated memory 14.60 GiB is allocated by PyTorch, and 22.06 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) Anyone have the same experience? Has anyone managed to run Llama 3 on free version of Colab (or similar platform)? Thanks! submitted by /u/ReputationMindless32 [link] [comments]

Hi everyone, I'm trying to build my own test set in order to make an initial fast evaluation of the huge number of models that pop up on huggingface.co every week, and I'm searching for a starting point or suggestions. If someone would share some questions that they use to test LLM abilities, even as high-level concepts, or simply give me some tips or suggestions, I would really appreciate that! Thanks in advance to everyone for any kind of reply." submitted by /u/Distinct-Target7503 [link] [comments]

https://arxiv.org/abs/2404.16767 New deep RL algorithm that works with both language models and diffusion models. submitted by /u/athens117 [link] [comments]

I am attempting to train CLIP from scratch. However, there is a lack of available datasets. The one dataset that seemed quite diverse and clean seems to be taken down (laion-400m). Looking at HF datasets, these are the two datasets that are promising, but wondering if there has been anything better/ cleaner. - conceptual captions: uses alt-text. - red_caps: reddit threads, but these are mostly the first comment on the image than an actual caption. TIA submitted by /u/themathstudent [link] [comments]

Everywhere I look for the answer to this question, the responses do little more than anthropomorphize the model. They invariably make claims like: Without examples, the model must infer context and rely on its knowledge to deduce what is expected. This could lead to misunderstandings. One-shot prompting reduces this cognitive load by offering a specific example, helping to anchor the model's interpretation and focus on a narrower task with clearer expectations. The example serves as a reference or hint for the model, helping it understand the type of response you are seeking and triggering memories of similar instances during training. Providing an example allows the model to identify a pattern or structure to replicate. It establishes a cue for the model to align with, reducing the guesswork inherent in zero-shot scenarios. These are real excerpts, btw. But these models don’t “understand” anything. They don’t “deduce”, or “interpret”, or “focus”, or “remember training”, or “make guesses”, or have literal “cognitive load”. They are just statistical token generators. Therefore pop-sci explanations like these are kind of meaningless when seeking a concrete understanding of the exact mechanism by which in-context learning improves accuracy. Can someone offer an explanation that explains things in terms of the actual model architecture/mechanisms and how the provision of additional context leads to better output? I can “talk the talk”, so spare no technical detail please. I could make an educated guess - Including examples in the input which use tokens that approximate the kind of output you want leads the attention mechanism and final dense layer to weight more highly tokens which are similar in some way to these examples, increasing the odds that these desired tokens will be sampled at each generation step; like fundamentally I’d guess a similarity/distance thing, where explicitly exemplifying the output I want increases the odds that the output get will be similar to it - but I’d prefer to hear it from someone else with deep knowledge of these models and mechanisms. submitted by /u/synthphreak [link] [comments]

In a video about "A little guide to building Large Language Models in 2024" at 41:38 the author starts to talk about the limits of how big the batch size can be. ​ Well, if you start to have a very large batch size, the model for each optimization step makes less efficient use of each token, because the batch size is so big that each token is kind of washed out in the optimization step. And roughly, it's a little bit hard to measure this limit, which we call the critical batch size. I thought that the bigger batch size is always better for training LLMs, because: It better approximates the true gradient. We go through the dataset faster. To my knowledge the limits are only infrastructure, hardware, communication overhead etc. I found a paper that introduces the "critical batch size" concept - An Empirical Model of Large-Batch Training. It mostly talks about the speed/efficiency tradeoff for data parallelism of large batch sizes. Also another highly cited paper Scaling Laws for Neural Language Models: Training at the critical batch size provides a roughly optimal compromise between time and compute efficiency So I don't really understand what author of the video meant by saying: each token is kind of washed out in the optimization step Are there any other issues with large batch sizes other than infrastructure, hardware or implementation limits? submitted by /u/kiockete [link] [comments]

Hi guys,I have a problem for which I am not sure what would be the best approach (and I cannot really find any relevant literature).I have a small dataset (~100 measurements like the one attached) of a sensor value, for which I want to predict a certain relevant event. Here t_0 is the relevant moment in time which I want to predict. The problem is, that I need to trigger something when the event is reached. If I need too long to trigger after it was reached, it will not be a positive outcome.My initial idea was to basically chunk the time series before the event, and try to predict the remaining time from that segment until the event is reached. When it is below a threshold, I can trigger my action. I wanted to have a look at e.g. XGBoost and feed it small chunks of the timeseries and run this process continuously. I am not really sure if that is the correct approach there.Is that a known problem? What would be a good name for this problem to search for literature? Do you have suggestions how to solve it? Thanks. https://preview.redd.it/l59gsmswkswc1.png?width=1303&format=png&auto=webp&s=f9e83d7b87ce5227378de6a0805a916fd4f93314 submitted by /u/seboz12345 [link] [comments]

I'm starting to do some research for my graduation and I'm looking for some papers on text to speech synthesis. I'm doing some reproductions on a paper I found to be interesting called Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Prediction. Basically it's a model that receives text, turns it into a spectogram and the spectogram is used to build the audio file. Since I'm still at the start of reproduction, are there papers that you guys would recommend looking into? Did you work with speech synthesis (TTS)? What are good refferences I should look into? ​ I saw this post over here https://www.reddit.com/r/MachineLearning/comments/nxkuvn/d_what_is_actually_the_state_of_the_art_in_text/ But it already has 3 years. Maybe there is something newer than FastSpeech2? ​ submitted by /u/Zelun [link] [comments]

[D] Hey everyone, do you have any suggestions on what's the better option and the most effective way to dive into a hot topic these days? I stumbled upon a repository for Federated Learning at: ​ https://github.com/muditbhargava66/dropgrad https://github.com/adap/flower But can't seem to find anything similar for Meta Learning. Any advice on how to pick my PhD topic would be greatly appreciated! submitted by /u/Tight_Confusion_1695 [link] [comments]

My friend implemented the method of Multihead Mixture of Experts in this arxiv paper https://arxiv.org/pdf/2404.15045 and he wanted me to share it with you! https://github.com/lhallee/Multi_Head_Mixture_of_Experts__MH-MOE Try it out. Let me know what you think and I will pass it on to him. submitted by /u/Prudent_Student2839 [link] [comments]

Hello guys, I've been working on a sequential labelling using DNA sequences as inputs. Lately there have been 2 foundation models released HyenaDNA (Based on Hyena operator) and Caduceus (based on mamba), I used both pretrained and from scratch models and performances are terrible even with pretrained ones. Does anyone have experience with this type of models, and what are the potential causes for performance drop ? I am literally getting zero performance for the minority class ? Does mamba deal poorly with class imbalance ? submitted by /u/blooming17 [link] [comments]

This is a small drug toxicity prediction GNN model I wrote/trained repo: https://github.com/Null-byte-00/toxicity-prediction-gnn submitted by /u/Soroush_ra [link] [comments]

ML is very good at solving a niche set of problems, but most of the technical nuances are lost on tech bros and managers. What are some problems you have been told to solve which would be impossible (no data, useless data, unrealistic expectations) or a misapplication of ML (can you have this LLM do all of out accounting). submitted by /u/LanchestersLaw [link] [comments]

Does anyone know what datasets are out there for causal inference? I’d like to explore methods in the doubly robust ML literature, and I’d like to compensate my learning by working on some datasets and learn the econML software. Does anyone know of any datasets, specifically in the context of marketing/pricing/advertising that would be good sources to apply causal inference techniques? I’m open to other datasets as well. submitted by /u/Direct-Touch469 [link] [comments]

Dreambooth the MusicGen model suite on small consumer GPUs, in a matter of minutes, using this repository: https://github.com/ylacombe/musicgen-dreamboothing The aim of this project is to provide tools to easily fine-tune and dreambooth the MusicGen model suite, with little data and to leverage a series of optimizations and tricks to reduce resource consumption, thanks to LoRA adaptors. For example, the model can be fine-tuned on a particular music genre or artist to give a checkpoint that generates in that given style. The aim is also to easily share and build on these trained checkpoints, Specifically, this involves: using as few data and resources as possible. We're talking fine-tuning with as little as 15mn on an A100 and as little as 10GB to 16GB of GPU utilization. easily share and build models thanks to the Hugging Face Hub. optionally, generate automatic music descriptions optionally, training MusicGen in a Dreambooth-like fashion, where one key-word triggers generation in a particular style Wandb example of what the training run looks like here. submitted by /u/Sufficient-Tennis189 [link] [comments]

Paper: https://arxiv.org/abs/2404.08698 Abstract: While Large Language Models (LLMs) have shown remarkable abilities, they are hindered by significant resource consumption and considerable latency due to autoregressive processing. In this study, we introduce Adaptive N-gram Parallel Decoding (ANPD), an innovative and lossless approach that accelerates inference by allowing the simultaneous generation of multiple tokens. ANPD incorporates a two-stage approach: it begins with a rapid drafting phase that employs an N-gram module, which adapts based on the current interactive context, followed by a verification phase, during which the original LLM assesses and confirms the proposed tokens. Consequently, ANPD preserves the integrity of the LLM's original output while enhancing processing speed. We further leverage a multi-level architecture for the N-gram module to enhance the precision of the initial draft, consequently reducing inference latency. ANPD eliminates the need for retraining or extra GPU memory, making it an efficient and plug-and-play enhancement. In our experiments, models such as LLaMA and its fine-tuned variants have shown speed improvements up to 3.67x, validating the effectiveness of our proposed ANPD. submitted by /u/SeawaterFlows [link] [comments]