What are some ways to increase precision or recall in machine learning?
What are some ways to Boost Precision and Recall in Machine Learning?
Sensitivity vs Specificity?
In machine learning, recall is the ability of the model to find all relevant instances in the data while precision is the ability of the model to correctly identify only the relevant instances. A high recall means that most relevant results are returned while a high precision means that most of the returned results are relevant. Ideally, you want a model with both high recall and high precision but often there is a trade-off between the two. In this blog post, we will explore some ways to increase recall or precision in machine learning.
There are two main ways to increase recall:
by increasing the number of false positives or by decreasing the number of false negatives. To increase the number of false positives, you can lower your threshold for what constitutes a positive prediction. For example, if you are trying to predict whether or not an email is spam, you might lower the threshold for what constitutes spam so that more emails are classified as spam. This will result in more false positives (emails that are not actually spam being classified as spam) but will also increase recall (more actual spam emails being classified as spam).
Get 20% off Google Google Workspace (Google Meet) Standard Plan with the following codes: 96DRHDRA9J7GTN6 Get 20% off Google Workspace (Google Meet) Business Plan (AMERICAS): M9HNXHX3WC9H7YE (Email us for more codes)
you can increase your threshold for what constitutes a positive prediction. For example, going back to the spam email prediction example, you might raise the threshold for what constitutes spam so that fewer emails are classified as spam. This will result in fewer false negatives (actual spam emails not being classified as spam) but will also decrease recall (fewer actual spam emails being classified as spam).
There are two main ways to increase precision:
by increasing the number of true positives or by decreasing the number of true negatives. To increase the number of true positives, you can raise your threshold for what constitutes a positive prediction. For example, using the spam email prediction example again, you might raise the threshold for what constitutes spam so that fewer emails are classified as spam. This will result in more true positives (emails that are actually spam being classified as spam) but will also decrease precision (more non-spam emails being classified as spam).
To decrease the number of true negatives,
you can lower your threshold for what constitutes a positive prediction. For example, going back to the spam email prediction example once more, you might lower the threshold for what constitutes spam so that more emails are classified as spam. This will result in fewer true negatives (emails that are not actually spam not being classified as spam) but will also decrease precision (more non-spam emails being classified as spam).
To summarize,
there are a few ways to increase precision or recall in machine learning. One way is to use a different evaluation metric. For example, if you are trying to maximize precision, you can use the F1 score, which is a combination of precision and recall. Another way to increase precision or recall is to adjust the threshold for classification. This can be done by changing the decision boundary or by using a different algorithm altogether.
Sensitivity vs Specificity
In machine learning, sensitivity and specificity are two measures of the performance of a model. Sensitivity is the proportion of true positives that are correctly predicted by the model, while specificity is the proportion of true negatives that are correctly predicted by the model.
Google introduced computing units, which you can purchase just like any other cloud computing unit you can from AWS or Azure etc. With Pro you get 100, and with Pro+ you get 500 computing units. GPU, TPU and option of High-RAM effects how much computing unit you use hourly. If you don’t have any computing units, you can’t use “Premium” tier gpus (A100, V100) and even P100 is non-viable.
Google Colab Pro+ comes with Premium tier GPU option, meanwhile in Pro if you have computing units you can randomly connect to P100 or T4. After you use all of your computing units, you can buy more or you can use T4 GPU for the half or most of the time (there can be a lot of times in the day that you can’t even use a T4 or any kinds of GPU). In free tier, offered gpus are most of the time K80 and P4, which performs similar to a 750ti (entry level gpu from 2014) with more VRAM.
For your consideration, T4 uses around 2, and A100 uses around 15 computing units hourly. Based on the current knowledge, computing units costs for GPUs tend to fluctuate based on some unknown factor.
Considering those:
For hobbyists and (under)graduate school duties, it will be better to use your own gpu if you have something with more than 4 gigs of VRAM and better than 750ti, or atleast purchase google pro to reach T4 even if you have no computing units remaining.
For small research companies, and non-trivial research at universities, and probably for most of the people Colab now probably is not a good option.
Colab Pro+ can be considered if you want Pro but you don’t sit in front of your computer, since it disconnects after 90 minutes of inactivity in your computer. But this can be overcomed with some scripts to some extend. So for most of the time Colab Pro+ is not a good option.
If you have anything more to say, please let me know so I can edit this post with them. Thanks!
In machine learning, precision and recall trade off against each other; increasing one often decreases the other. There is no single silver bullet solution for increasing either precision or recall; it depends on your specific use case which one is more important and which methods will work best for boosting whichever metric you choose. In this blog post, we explored some methods for increasing either precision or recall; hopefully this gives you a starting point for improving your own models!
Hello r/MachineLearning, I'm excited to share a project that was initially intended to integrate automated AI maintenance features for Windows into an application I was building to sell commercially, but has now been open-sourced for community use and development. The project focuses on automated data sorting and could serve as a base for more advanced machine learning applications. You can explore the project here: [NazTech Automated Data Sorting Tools](https://github.com/nazpins/naztech-automated-data-sorting-tools) These tools are designed to quickly automate sorting large data dumps, employing python algorithms suitable for handling large datasets. While the project is no longer in active development from my end, the python scripts are functional and open for any adaptations or enhancements you might find interesting for your own ML projects. I started building the framework for the actual application but due to time constraints and a lot going on irl, I haven't had time to continue working on it. I am happy however to share these tools with the community and hopefully they can be beneficial to someone else down the road. Cheers! submitted by /u/Kilroy_GreyFox [link] [comments]
Hi All, I am brainstorming some kind of a nomenclature for our team so that theres a standard way of naming ML models like their pickle files . Any inputs will be appreciated. thanks submitted by /u/BravoZero6 [link] [comments]
Hi r/MachineLearning community, I'm currently working on the second part of my thesis focused on Named Entity Recognition (NER) for detecting Adverse Drug Events (ADE) in clinical texts. In my first thesis project, I tried to replicate a paper but had to pivot to the n2c2 dataset, which led to challenges in model performance. I've fine-tuned a DeBERTa model with standard practices, but I'm struggling with achieving high accuracy, particularly with precision and recall. This is my first deep dive into a thesis and the world of NLP, and any guidance would be immensely appreciated. Also, any common pitfalls for thesis work or useful resources on this topic would be extremely helpful. I'm eager to learn from the community and improve my research. Thank you so much for your time! submitted by /u/Popsuga [link] [comments]
Through our experiments, we found that LLM agents have a certain ability to understand probability distributions, the LLM agent's sampling ability for probability distributions is lacking and it is difficult to give a behavior sequence that conforms to a certain probability distribution through LLMs alone. We are looking forward to your thoughts, critiques, and discussions on this topic. Full Paper & Citation: You can access the full paper https://arxiv.org/abs/2404.09043. Please cite our work if it contributes to your research. https://preview.redd.it/ai7uks7nluwc1.png?width=935&format=png&auto=webp&s=891dd57ef50d1ee99b1a8b2372b9a460397754d6 submitted by /u/GYX-001 [link] [comments]
Main aim: Style transfer between two discrete timeseries signals. Here are the details: Dataset: Discrete time series. 1700 rows, with 97 percent of it with zeroes. Cannot remove these zeroes as it means something. Values ranging from 0-32 for one of the features in Domain A needs to translated to another feature with same range in domain B. Another feature from 0-5000 from domain A, translated to a different domain B with same range. I can recreate the same dataset multiple times with small variations, so we can have larger datasets. I would create sequences of size 20 or 30 and batch: 32 or 64 initially. Generator Network: A simple encoder with linear layer first hidden size:16 , relu, 2nd linear layer :8 and relu again . A symmetric Decoder . Discriminator: 2 linear layers with hidden size 8 and leaky Relu between them. And sigmoid as final layer. Loss function : BCEloss . Also experimented BCE + MSE loss for generator. Training: I'm using pytorch. Only trained with one feature/signal and tried to generate this feature from noise. Didn't move to cycle consistency yet. With the small dataset training, the discriminator becomes too strong, I even tried to set reduce the learning rate for discriminator as 0.0001 and generator as 0.01 , it didn't work. Tried to add/complicate the layer of generator, still didn't work. Tried to train discriminator every 10th epoch, while the generator trained more. Didn't work. Also tried to normalize the data. I want to explore Adversarial autoencoder /cycle Gan , but the generator is unable to learn anything with vanilla GAN as well. Can someone help or give me some ideas on what I can do ? Thanks submitted by /u/investmentwholesome [link] [comments]
Hi guys, I just tried to run Llama 3 on my Colab(free version) and seems that I ran out of the memory: OutOfMemoryError: CUDA out of memory. Tried to allocate 32.00 MiB. GPU 0 has a total capacity of 14.75 GiB of which 9.06 MiB is free. Process 8863 has 14.74 GiB memory in use. Of the allocated memory 14.60 GiB is allocated by PyTorch, and 22.06 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) Anyone have the same experience? Has anyone managed to run Llama 3 on free version of Colab (or similar platform)? Thanks! submitted by /u/ReputationMindless32 [link] [comments]
Hi everyone, I'm trying to build my own test set in order to make an initial fast evaluation of the huge number of models that pop up on huggingface.co every week, and I'm searching for a starting point or suggestions. If someone would share some questions that they use to test LLM abilities, even as high-level concepts, or simply give me some tips or suggestions, I would really appreciate that! Thanks in advance to everyone for any kind of reply." submitted by /u/Distinct-Target7503 [link] [comments]
https://arxiv.org/abs/2404.16767 New deep RL algorithm that works with both language models and diffusion models. submitted by /u/athens117 [link] [comments]
I am attempting to train CLIP from scratch. However, there is a lack of available datasets. The one dataset that seemed quite diverse and clean seems to be taken down (laion-400m). Looking at HF datasets, these are the two datasets that are promising, but wondering if there has been anything better/ cleaner. - conceptual captions: uses alt-text. - red_caps: reddit threads, but these are mostly the first comment on the image than an actual caption. TIA submitted by /u/themathstudent [link] [comments]
Everywhere I look for the answer to this question, the responses do little more than anthropomorphize the model. They invariably make claims like: Without examples, the model must infer context and rely on its knowledge to deduce what is expected. This could lead to misunderstandings. One-shot prompting reduces this cognitive load by offering a specific example, helping to anchor the model's interpretation and focus on a narrower task with clearer expectations. The example serves as a reference or hint for the model, helping it understand the type of response you are seeking and triggering memories of similar instances during training. Providing an example allows the model to identify a pattern or structure to replicate. It establishes a cue for the model to align with, reducing the guesswork inherent in zero-shot scenarios. These are real excerpts, btw. But these models don’t “understand” anything. They don’t “deduce”, or “interpret”, or “focus”, or “remember training”, or “make guesses”, or have literal “cognitive load”. They are just statistical token generators. Therefore pop-sci explanations like these are kind of meaningless when seeking a concrete understanding of the exact mechanism by which in-context learning improves accuracy. Can someone offer an explanation that explains things in terms of the actual model architecture/mechanisms and how the provision of additional context leads to better output? I can “talk the talk”, so spare no technical detail please. I could make an educated guess - Including examples in the input which use tokens that approximate the kind of output you want leads the attention mechanism and final dense layer to weight more highly tokens which are similar in some way to these examples, increasing the odds that these desired tokens will be sampled at each generation step; like fundamentally I’d guess a similarity/distance thing, where explicitly exemplifying the output I want increases the odds that the output get will be similar to it - but I’d prefer to hear it from someone else with deep knowledge of these models and mechanisms. submitted by /u/synthphreak [link] [comments]
In a video about "A little guide to building Large Language Models in 2024" at 41:38 the author starts to talk about the limits of how big the batch size can be. Well, if you start to have a very large batch size, the model for each optimization step makes less efficient use of each token, because the batch size is so big that each token is kind of washed out in the optimization step. And roughly, it's a little bit hard to measure this limit, which we call the critical batch size. I thought that the bigger batch size is always better for training LLMs, because: It better approximates the true gradient. We go through the dataset faster. To my knowledge the limits are only infrastructure, hardware, communication overhead etc. I found a paper that introduces the "critical batch size" concept - An Empirical Model of Large-Batch Training. It mostly talks about the speed/efficiency tradeoff for data parallelism of large batch sizes. Also another highly cited paper Scaling Laws for Neural Language Models: Training at the critical batch size provides a roughly optimal compromise between time and compute efficiency So I don't really understand what author of the video meant by saying: each token is kind of washed out in the optimization step Are there any other issues with large batch sizes other than infrastructure, hardware or implementation limits? submitted by /u/kiockete [link] [comments]
Hi guys,I have a problem for which I am not sure what would be the best approach (and I cannot really find any relevant literature).I have a small dataset (~100 measurements like the one attached) of a sensor value, for which I want to predict a certain relevant event. Here t_0 is the relevant moment in time which I want to predict. The problem is, that I need to trigger something when the event is reached. If I need too long to trigger after it was reached, it will not be a positive outcome.My initial idea was to basically chunk the time series before the event, and try to predict the remaining time from that segment until the event is reached. When it is below a threshold, I can trigger my action. I wanted to have a look at e.g. XGBoost and feed it small chunks of the timeseries and run this process continuously. I am not really sure if that is the correct approach there.Is that a known problem? What would be a good name for this problem to search for literature? Do you have suggestions how to solve it? Thanks. https://preview.redd.it/l59gsmswkswc1.png?width=1303&format=png&auto=webp&s=f9e83d7b87ce5227378de6a0805a916fd4f93314 submitted by /u/seboz12345 [link] [comments]
I'm starting to do some research for my graduation and I'm looking for some papers on text to speech synthesis. I'm doing some reproductions on a paper I found to be interesting called Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Prediction. Basically it's a model that receives text, turns it into a spectogram and the spectogram is used to build the audio file. Since I'm still at the start of reproduction, are there papers that you guys would recommend looking into? Did you work with speech synthesis (TTS)? What are good refferences I should look into? I saw this post over here https://www.reddit.com/r/MachineLearning/comments/nxkuvn/d_what_is_actually_the_state_of_the_art_in_text/ But it already has 3 years. Maybe there is something newer than FastSpeech2? submitted by /u/Zelun [link] [comments]
[D] Hey everyone, do you have any suggestions on what's the better option and the most effective way to dive into a hot topic these days? I stumbled upon a repository for Federated Learning at: https://github.com/muditbhargava66/dropgrad https://github.com/adap/flower But can't seem to find anything similar for Meta Learning. Any advice on how to pick my PhD topic would be greatly appreciated! submitted by /u/Tight_Confusion_1695 [link] [comments]
My friend implemented the method of Multihead Mixture of Experts in this arxiv paper https://arxiv.org/pdf/2404.15045 and he wanted me to share it with you! https://github.com/lhallee/Multi_Head_Mixture_of_Experts__MH-MOE Try it out. Let me know what you think and I will pass it on to him. submitted by /u/Prudent_Student2839 [link] [comments]
Hello guys, I've been working on a sequential labelling using DNA sequences as inputs. Lately there have been 2 foundation models released HyenaDNA (Based on Hyena operator) and Caduceus (based on mamba), I used both pretrained and from scratch models and performances are terrible even with pretrained ones. Does anyone have experience with this type of models, and what are the potential causes for performance drop ? I am literally getting zero performance for the minority class ? Does mamba deal poorly with class imbalance ? submitted by /u/blooming17 [link] [comments]
This is a small drug toxicity prediction GNN model I wrote/trained repo: https://github.com/Null-byte-00/toxicity-prediction-gnn submitted by /u/Soroush_ra [link] [comments]
ML is very good at solving a niche set of problems, but most of the technical nuances are lost on tech bros and managers. What are some problems you have been told to solve which would be impossible (no data, useless data, unrealistic expectations) or a misapplication of ML (can you have this LLM do all of out accounting). submitted by /u/LanchestersLaw [link] [comments]
Does anyone know what datasets are out there for causal inference? I’d like to explore methods in the doubly robust ML literature, and I’d like to compensate my learning by working on some datasets and learn the econML software. Does anyone know of any datasets, specifically in the context of marketing/pricing/advertising that would be good sources to apply causal inference techniques? I’m open to other datasets as well. submitted by /u/Direct-Touch469 [link] [comments]
Dreambooth the MusicGen model suite on small consumer GPUs, in a matter of minutes, using this repository: https://github.com/ylacombe/musicgen-dreamboothing The aim of this project is to provide tools to easily fine-tune and dreambooth the MusicGen model suite, with little data and to leverage a series of optimizations and tricks to reduce resource consumption, thanks to LoRA adaptors. For example, the model can be fine-tuned on a particular music genre or artist to give a checkpoint that generates in that given style. The aim is also to easily share and build on these trained checkpoints, Specifically, this involves: using as few data and resources as possible. We're talking fine-tuning with as little as 15mn on an A100 and as little as 10GB to 16GB of GPU utilization. easily share and build models thanks to the Hugging Face Hub. optionally, generate automatic music descriptions optionally, training MusicGen in a Dreambooth-like fashion, where one key-word triggers generation in a particular style Wandb example of what the training run looks like here. submitted by /u/Sufficient-Tennis189 [link] [comments]
Today I Learned (TIL) You learn something new every day; what did you learn today? Submit interesting and specific facts about something that you just found out here.
submitted by /u/ColeBelthazorTurner [link] [comments]
Reddit Science This community is a place to share and discuss new scientific research. Read about the latest advances in astronomy, biology, medicine, physics, social science, and more. Find and submit new publications and popular science coverage of current research.