What are some ways to increase precision or recall in machine learning?
What are some ways to Boost Precision and Recall in Machine Learning?
Sensitivity vs Specificity?
In machine learning, recall is the ability of the model to find all relevant instances in the data while precision is the ability of the model to correctly identify only the relevant instances. A high recall means that most relevant results are returned while a high precision means that most of the returned results are relevant. Ideally, you want a model with both high recall and high precision but often there is a trade-off between the two. In this blog post, we will explore some ways to increase recall or precision in machine learning.
There are two main ways to increase recall:
by increasing the number of false positives or by decreasing the number of false negatives. To increase the number of false positives, you can lower your threshold for what constitutes a positive prediction. For example, if you are trying to predict whether or not an email is spam, you might lower the threshold for what constitutes spam so that more emails are classified as spam. This will result in more false positives (emails that are not actually spam being classified as spam) but will also increase recall (more actual spam emails being classified as spam).
Get 20% off Google Google Workspace (Google Meet) Standard Plan with the following codes: 96DRHDRA9J7GTN6 Get 20% off Google Workspace (Google Meet) Business Plan (AMERICAS): M9HNXHX3WC9H7YE (Email us for more codes)
you can increase your threshold for what constitutes a positive prediction. For example, going back to the spam email prediction example, you might raise the threshold for what constitutes spam so that fewer emails are classified as spam. This will result in fewer false negatives (actual spam emails not being classified as spam) but will also decrease recall (fewer actual spam emails being classified as spam).
There are two main ways to increase precision:
by increasing the number of true positives or by decreasing the number of true negatives. To increase the number of true positives, you can raise your threshold for what constitutes a positive prediction. For example, using the spam email prediction example again, you might raise the threshold for what constitutes spam so that fewer emails are classified as spam. This will result in more true positives (emails that are actually spam being classified as spam) but will also decrease precision (more non-spam emails being classified as spam).
To decrease the number of true negatives,
you can lower your threshold for what constitutes a positive prediction. For example, going back to the spam email prediction example once more, you might lower the threshold for what constitutes spam so that more emails are classified as spam. This will result in fewer true negatives (emails that are not actually spam not being classified as spam) but will also decrease precision (more non-spam emails being classified as spam).
To summarize,
there are a few ways to increase precision or recall in machine learning. One way is to use a different evaluation metric. For example, if you are trying to maximize precision, you can use the F1 score, which is a combination of precision and recall. Another way to increase precision or recall is to adjust the threshold for classification. This can be done by changing the decision boundary or by using a different algorithm altogether.
Sensitivity vs Specificity
In machine learning, sensitivity and specificity are two measures of the performance of a model. Sensitivity is the proportion of true positives that are correctly predicted by the model, while specificity is the proportion of true negatives that are correctly predicted by the model.
Google introduced computing units, which you can purchase just like any other cloud computing unit you can from AWS or Azure etc. With Pro you get 100, and with Pro+ you get 500 computing units. GPU, TPU and option of High-RAM effects how much computing unit you use hourly. If you don’t have any computing units, you can’t use “Premium” tier gpus (A100, V100) and even P100 is non-viable.
Google Colab Pro+ comes with Premium tier GPU option, meanwhile in Pro if you have computing units you can randomly connect to P100 or T4. After you use all of your computing units, you can buy more or you can use T4 GPU for the half or most of the time (there can be a lot of times in the day that you can’t even use a T4 or any kinds of GPU). In free tier, offered gpus are most of the time K80 and P4, which performs similar to a 750ti (entry level gpu from 2014) with more VRAM.
For your consideration, T4 uses around 2, and A100 uses around 15 computing units hourly. Based on the current knowledge, computing units costs for GPUs tend to fluctuate based on some unknown factor.
Considering those:
For hobbyists and (under)graduate school duties, it will be better to use your own gpu if you have something with more than 4 gigs of VRAM and better than 750ti, or atleast purchase google pro to reach T4 even if you have no computing units remaining.
For small research companies, and non-trivial research at universities, and probably for most of the people Colab now probably is not a good option.
Colab Pro+ can be considered if you want Pro but you don’t sit in front of your computer, since it disconnects after 90 minutes of inactivity in your computer. But this can be overcomed with some scripts to some extend. So for most of the time Colab Pro+ is not a good option.
If you have anything more to say, please let me know so I can edit this post with them. Thanks!
In machine learning, precision and recall trade off against each other; increasing one often decreases the other. There is no single silver bullet solution for increasing either precision or recall; it depends on your specific use case which one is more important and which methods will work best for boosting whichever metric you choose. In this blog post, we explored some methods for increasing either precision or recall; hopefully this gives you a starting point for improving your own models!
I am undergradute student learning machine learning. What I got to know while reading few papers that we try to reduce search space by imposing inductive bias in machine learning models. And the success in creating useful models comes when inductive bias matches with the underlying data. In heriarchical models like NVAE how they instilled inductive bias by specifing the way data gets computed? (I thinks it's called algorithmic bias, not sure though) But how people think such inductive bias will be helpful, what is step by step procedure they go through to insist such inductive bias. I took a lot of class in machine learning and statistics but didn't got any lectures explaing such stuff. Did I missed any course/lecture? Please provide my with papers/lectures/talks related to it if possible Thankyou submitted by /u/binny_sarita [link] [comments]
Generalization of the popular training method of CLIP to be better suited for search and recommendations. Paper: https://arxiv.org/pdf/2404.08535.pdf Github: https://github.com/marqo-ai/GCL Generalises CLIP: Use any number of text and/or images to represent documents. Better text understanding by having both inter- and intra-modal losses. Can encode rank/importance/relevance, a.k.a “rank-tune”. Works with pretrained, text, CLIP models. Can learn uni- or multi-vector representations for documents. Works with binary and Matryoshka methods. Open source 10M row multi-modal dataset with 100k queries and ~5M products. Why? The prevailing methods for training embedding models are largely disconnected from the end use-case (like search), the vector database, the requirements of users, and a lack of representative datasets for development and evaluation, particularly when multiple modalities and ranking is involved. Limitations of current embedding models for vector search Although vector search is very powerful and enables searching across just about any data, the current methods have some limitations. The prevailing methods for training embedding models are largely disconnected from the end use-case (like search), the vector database, and the requirements of users. This means that a lot of the potential of vector search is being unmet. Some of the current challenges are described below. Restricted to using a single piece of information to represent a document Current models encode and represent one piece of information with one vector. The reality is that often there are multiple pieces of pertinent information for a document that may span multiple modalities. For example, in product search there may be a title, description, reviews, and multiple images, each with its own caption. GCL generalises embedding model training to use as many pieces of information as is desired. No notion of rank when dealing with degenerate queries When there are degenerate queries - multiple results that satisfy some criteria of relevance - the ordering of the results is only ever learned indirectly from the many binary relationships. In reality, the ordering of results matters, even for first stage retrieval. GCL allows for the magnitude of query-document specific relevance to be encoded in the embeddings and improves ranking of candidate documents. Poor text understanding when using CLIP like methods For multi-modal models like CLIP, these are trained to only work from image to text (and vice versa). The text-text understanding is not as good as text only models due to the text-text relationships being learned indirectly through images. For many applications, having both inter- and intra-modality understanding is required. GCL allows for any combination of inter- and intra-modal understanding by directly optimizing for this. Lack of representative datasets to develop methods for vector search In developing GCL, it became apparent there was a disconnect with publicly available datasets for embedding model training and evaluation for real-world use cases. Existing benchmarks are typically text only or inter-modal only and focus on the 1-1 query-result paradigm. Additionally, existing datasets have limited notions of relevance, with the majority encoding it as a binary relationship while several use (up-to) a handful of discrete categorizations often on the test set only. This differs from a typical real-world use cases where relevance can be both hard binary relationships or come from continuous variables. To help with this we compiled a dataset of 10M (ranked) product-query pairs, across ~100k queries, nearly 5M products, and four evaluation splits (available here). submitted by /u/Jesse_marqo [link] [comments]
How are people using AI inside companies (startups -> FAANG) to improve operations and processes? There is so much talk about leveraging LLM’s and GenAI but I’m struggling to find real concrete examples that are successful. The following areas come to mind first but this list isn’t exhaustive of course: Design (and handoff) Engineering Customer Support Sales Documentation Marketing What’s worked or shown promise? What hasn’t worked? submitted by /u/CJSF [link] [comments]
I'm surprised (or maybe not) to say this, but Meta (or Facebook) democratises AI/ML much more than OpenAI, which was originally founded and primarily funded for this purpose. OpenAI has largely become a commercial project for profit only. Although as far as Llama models go, they don't yet reach GPT4 capabilities for me, but I believe it's only a matter of time. What do you guys think about this? submitted by /u/ReputationMindless32 [link] [comments]
I've had a lot of success with Whisper when it comes to transcriptions, but word level timestamps seems to be slightly inaccurate. From my understanding ("Whisper cannot provide reliable word timestamps, because the END-TO-END models like Transformer using cross-entropy training criterion are not designed for reliably estimating word timestamps." https://www.youtube.com/watch?v=H576iCWt1Co&t=192s) For my use case, I need precise word level timestamps, because I'm doing audio insertion after specific words. This becomes problematic when I do an insertion and the back part of a word ends up on the other side. Example: Given an original audio file with speech that has been transcribed, If I want to insert a clip at the end of the word "France", and according to the timestamp, the word "France" starts at 19.26 and ends at 19.85, I will insert the clip at 19.85. However, if the actual end of France is at 19.92, then when I insert the laugher at 19.85, I will here the remaining "France", likely "ce" (0.07), at the end. I'm curious if anyone has been posed with a similar problem and what they did to get around this? I've experimented with a few open source variations of whisper, but still running into that issue. submitted by /u/Mindless-Ordinary485 [link] [comments]
Paper: https://arxiv.org/abs/2404.06405 Code: https://huggingface.co/datasets/bethgelab/simplegeometry Abstract: Proving geometric theorems constitutes a hallmark of visual reasoning combining both intuitive and logical skills. Therefore, automated theorem proving of Olympiad-level geometry problems is considered a notable milestone in human-level automated reasoning. The introduction of AlphaGeometry, a neuro-symbolic model trained with 100 million synthetic samples, marked a major breakthrough. It solved 25 of 30 International Mathematical Olympiad (IMO) problems whereas the reported baseline based on Wu's method solved only ten. In this note, we revisit the IMO-AG-30 Challenge introduced with AlphaGeometry, and find that Wu's method is surprisingly strong. Wu's method alone can solve 15 problems, and some of them are not solved by any of the other methods. This leads to two key findings: (i) Combining Wu's method with the classic synthetic methods of deductive databases and angle, ratio, and distance chasing solves 21 out of 30 methods by just using a CPU-only laptop with a time limit of 5 minutes per problem. Essentially, this classic method solves just 4 problems less than AlphaGeometry and establishes the first fully symbolic baseline strong enough to rival the performance of an IMO silver medalist. (ii) Wu's method even solves 2 of the 5 problems that AlphaGeometry failed to solve. Thus, by combining AlphaGeometry with Wu's method we set a new state-of-the-art for automated theorem proving on IMO-AG-30, solving 27 out of 30 problems, the first AI method which outperforms an IMO gold medalist. submitted by /u/SeawaterFlows [link] [comments]
Hey guys, I'm working on finishing up my data science degree, with a high possibility to go into a masters. I'm currently holding a good job in aero space with a lot of background in aero space manufacturing. I really like the job and it pays well, but I'm lost for what field or job I can incorporate my aero space experience and data science. If anyone can offer any guidance, that would be great. submitted by /u/oppapoocow [link] [comments]
Is there a method to generate the approximations of Shapely values (or something similar) for a data without using model object. Essentialy I input features and model predictions on benchmark data, and the same for test data, and output is contributions for each feature on test data submitted by /u/ozymandias_514 [link] [comments]
Just curious what everyone’s thoughts are on this. Seems like more DS postings are placing a larger emphasis on software development than statistics/model development. I’ve also noticed this trend at my company. There are even senior DS managers at my company saying stats are for analysts (which is a wild statement). DS is well paid, however, not as well paid as SWE, typically. Feels like shady HR tactics are at work to save dollars on software development. submitted by /u/nxp1818 [link] [comments]
I developed a very powerful Python Config Management Tool. It can make your json config as powerful as python code. And very friendly to humans. The most attractive feature is that it will analyze Python code and json config file in real time, provide document display, parameter completion, and goto Python definition from the json config. (Powered by LSP) Similar or better config inheritance, parameter reference and parameter grid search like hydra Data validation similar to pydantic, and the ability to convert dataclass to json-schema This project is still in its early stages, and everyone is welcome to provide some suggestions and ideas. git repo submitted by /u/cssunfu [link] [comments]
https://huggingface.co/microsoft/Phi-3-mini-128k-instruct The numbers in the technical report look really great, I guess need to be verified by 3rd parties. submitted by /u/topcodemangler [link] [comments]
I currently work as an MLE(~5 YOE) focused mainly in computer vision. I'm looking towards the future and trying to figure out next steps for my career and how to up-skill. I find myself gravitating towards working on product with ML as part of core-value-added and can't see myself working on ML/DS for analytics. Web-dev has always seemed interesting to me - specifically front-end dev (Eg. React, WebGL, Three.js). I have a limited amount of time to invest in up-skilling and am debating whether there's any justification for investing the time here. Is there a niche for an ML-Engineer with front-end knowledge? Maybe developing ML-based-tools? submitted by /u/answersareallyouneed [link] [comments]
Hey, I am currently doing a malware classification (malware,benign).Used the naive Bayes(Bernoulli) the accuracy was 67 at this point.After the tuning it goes straight up 100. Is this normal or not? I did Outlier removal using the IQR and feature selection using the correlation. submitted by /u/Saheenus [link] [comments]
I've been working as a DS in predictive analytics a good amount of years now, and recently I've been pushed to delve a bit more on the more data viz side, which eeeehh fortunately or unfortunately meant coding web-dev stuff. I have realized that in the web-dev side, there are a shit ton of people building and consuming boilerplates. Like for real, a mind-blowing amount of demand for such things that I would not have expected in my life. However, I've never seen anything similar for DS projects. Sure, there's decent documentation and examples in most libraries, but it's still far from the convenience of a boilerplate. Talking to a mate he was like, I'm sure in web-dev everything is more standard than DS, and I'm like... man have you seen how many frameworks, backends, styling clusterfuck of technologies is out there. So I don't think standardization is the reason here. Do you guys think there is a gap in DS when it comes to this kind of things? Any ideas why is not more widespread? Am I missing something all toghether? EDIT: By boilerplate I don't mean ready to go models, I mean skeleton code for things like data loading and processing or result analysis, so the repetitive stuff... NOT things like model and parameter turning. submitted by /u/AccomplishedPace6024 [link] [comments]
Many are trying to install and deploy their own LLaMA 3 model, so here is a tutorial I just made showing how to deploy LLaMA 3 on an AWS EC2 instance: https://nlpcloud.com/how-to-install-and-deploy-llama-3-into-production.html Deploying LLaMA 3 8B is fairly easy but LLaMA 3 70B is another beast. Given the amount of VRAM needed you might want to provision more than one GPU and use a dedicated inference server like vLLM in order to split your model on several GPUs. LLaMA 3 8B requires around 16GB of disk space and 20GB of VRAM (GPU memory) in FP16. As for LLaMA 3 70B, it requires around 140GB of disk space and 160GB of VRAM in FP16. I hope it is useful, and if you have questions please don't hesitate to ask! Julien submitted by /u/juliensalinas [link] [comments]
Hello Community, I am preparing a lecture on language models with a lot of hands on work in python. I am trying Llama 3 8b but it seems to run very slow to answer prompts. I have a laptop with a modern I7 and 32GB of RAM. I assume my students have something less powerful and I don’t want them to take hours just for a prompt. So do you know a model that is reasonably fast without giving up too much performance? submitted by /u/mbungee [link] [comments]
Hello! I've studied robotics engineering, and my approach to Al has primarily focused on computer vision. I've been deeply interested in this area and, of course, nowadays, I use GPT-4 on a daily basis for both work and personal projects. I'm starting an MSc in Al next year, and in the cover letter for the master's application, I would like to discuss potential research topics that combine robotics and Al. My first choice involves using reinforcement learning for navigation and path planning. I'm also interested in the potential application of vision transformers to visual and semantic SLAM, as well as deep learning-based feature extractors. I'm also impressed by what large companies are achieving with multimodal transformers and humanoid robot like the figure Al, although I understand that these projects might only be feasible for such organizations. I'd love to hear your opinion and insights on the trends in robotics and Al. And receive some guidance from experts on the field. What do you think are the current hot topics, and what will be the key areas of focus in the next 5, 10, and 15 years? Thanks! submitted by /u/navarrox99 [link] [comments]
Hi everybody, I've been working on a project for over two years now: Deepfake-Total.com, a website where you can analyze audio files for traces of text-to-speech or voice conversion, i.e. identify audio deepfakes and spoofs, which I feel is an important topic, especially given the number of important elections coming up. The tool is free-to-use and supports YouTube and twitter URLs, as well as the manual upload of audio files. I wanted to share this and get your feedback: What works for you and what does not, where can I improve? The model should be quite precise for most input, but similar to an anti-virus scanner, there'll always be the occasional outlier. If you find any, feedback is much appreciated, as well as ideas for (scientific) collaboration, etc. submitted by /u/mummni [link] [comments]
Is MoE architecture also applied in other ML areas, let’s say Computer Vision? Why aren’t they popular? Is it because we don’t scale vision transformers as much as LLMs, and MoE is best for scalability? submitted by /u/lime_52 [link] [comments]
I'm wondering what technical recruiters or seasoned DS/MLE have to say about people with profiles like mine: good theoretical and decent technical background but working solo for too long. Summary of my career for context: I've been working 8 years now as a DS, the first 3 in medium sized R&D and consulting teams (for a big tech company), then for the past 5 as a solo DS for relatively successful non-ai focused start-ups, mostly developing ML/NLP stuff to address specific issues or improve one specific feature of their product (i.e. never a whole product). In 5 years I designed. developed and deployed, say, 4 models (but experimented with many ofc) - along with a few dashboards and simple streamlit POCs). Recently attending to meetups and seeing how people that make part of actual teams work, discuss and exchange knowledge it suddenly stroke me: I'm missing out, I'm becoming obsolete. I dont feel sharp enough for technical interviews, I'm not sure the way I develop and maintain my projects are following good standards/best practices (heck, i hardly follow a kanban, mostly use my planner to report to my boss on progress). I do some version control and document what I put into prod, but not even that I'm sure I'm doing as it'd be expected within a team. submitted by /u/Melodic_Reality_646 [link] [comments]
Today I Learned (TIL) You learn something new every day; what did you learn today? Submit interesting and specific facts about something that you just found out here.
submitted by /u/Cactus_Jacks_Ear [link] [comments]
Reddit Science This community is a place to share and discuss new scientific research. Read about the latest advances in astronomy, biology, medicine, physics, social science, and more. Find and submit new publications and popular science coverage of current research.