submitted by /u/ElectrikMetriks [link] [comments]

Hi Everyone! Bit of a technical modeling question here. Apologies if this is very basic preprocessing stuff but I'm a younger data scientist working in industry and I'm still learning. Say you have a pretty standard binary classification model predicting 1 = we should market to this customer and 0 = we should not market to this customer (the exact labeling scheme is a bit proprietary). I have a few features that are in the style "days since last touchpoint". For example "days since we last emailed this person" or "days since we last sold to this person". However, a solid percentage of the rows are NULL, meaning we have never emailed or sold to this person. Any thoughts on how should I handle NULLs for this type of column? I've been imputing with MAX(days since we last sold to this person) + 1 but I'm starting to think that could be confusing my model. I think the reality of the situation is that someone with 1 purchase a long time ago is a lot more likely to purchase today than someone who has never purchased anything at all. The person with 0 purchases may not even be interested in our product, while we have evidence that the person with 1 purchase a long time ago is at least a fit for our product. Imputing with MAX(days since we last sold to this person) + 1 poses these two cases as very similar to the model. For reference I'm testing with several tree-based models (light GBM and random forest) and comparing metrics to pick between the architecture options. So far I've been getting the best results with light GBM. One thing I'm thinking about is whether I should just leave the people who have never sold as NULLs and have my model pick the direction to split for missing values. (I believe this would work with LightGBM but not RandomForest). Another option is to break down the "days since last sale" feature into categories, maybe quantiles with a special category for NULLS, and then dummy encode. Has anyone else used these types of "days since last touchpoint" features in propensity modeling/marketing modeling? submitted by /u/Ok-Needleworker-6122 [link] [comments]

that's pretty much it. i'm proficient in python already, but was wondering if, to be a better DS, i'd need to learn something else, or is it better to focus on studying something else rather than a new language. edit: yes, SQL is obviously a must. i already know it. sorry for the overlook. submitted by /u/vniversvs_ [link] [comments]

Welcome to this week's entering & transitioning thread! This thread is for any questions about getting started, studying, or transitioning into the data science field. Topics include: Learning resources (e.g. books, tutorials, videos) Traditional education (e.g. schools, degrees, electives) Alternative education (e.g. online courses, bootcamps) Job search questions (e.g. resumes, applying, career prospects) Elementary questions (e.g. where to start, what next) While you wait for answers from the community, check out the FAQ and Resources pages on our wiki. You can also search for answers in past weekly threads. submitted by /u/AutoModerator [link] [comments]

submitted by /u/brodrigues_co [link] [comments]

Hey folks! I’m on the hunt for trustworthy remote job boards or sites that regularly post real data science and data analyst roles—and more importantly, are open to hiring from anywhere in the world. I’ve noticed sites like Indeed don’t support my country, and while LinkedIn has plenty of remote listings, many seem sketchy or not legit. So, what platforms or communities do you recommend for finding genuine remote gigs in this field that are truly global? Any tips on spotting legit postings would also be super helpful! Thanks in advance for sharing your experiences! submitted by /u/Aftabby [link] [comments]

I’ve been occasionally working on this in my spare time and would appreciate feedback. Try the package in Colab The idea for ‘framecheck’ is to catch bad data in a data frame before it flows downstream in very few lines of code. You’d also easily isolate the records with problematic data. This isn’t revolutionary or new - what I wanted was a way to do this in fewer lines of code than other packages like great expectations and pydantic. Really I just want honest feedback. If people don’t find it useful, I won’t put more time into it. pip install framecheck Repo with reproducible examples: https://github.com/OlivierNDO/framecheck submitted by /u/MLEngDelivers [link] [comments]

Why I’m doing this I am low on karma. Plus, it just feels good to help. About me I’m currently a staff data scientist at a big tech company in Silicon Valley. I’ve been in the field for about 10 years since earning my PhD in Statistics. I’ve worked at companies of various sizes — from seed-stage startups to pre-IPO unicorns to some of the largest tech companies. A few caveats Anything I share reflects my personal experience and may carry some bias. My experience is based in the US, particularly in Silicon Valley. I have some people management experience but have mostly worked as an IC Data science is a broad term. I’m most familiar with machine learning scientist, experimentation/causal inference, and data analyst roles. I may not be able to respond immediately, but I’ll aim to reply within 24 hours. Update: Wow, I didn’t expect this to get so much attention. I’m a bit overwhelmed by the number of comments and DMs, so I may not be able to reply to everyone. That said, I’ll do my best to respond to as many as I can over the next week. Really appreciate all the thoughtful questions and discussions! submitted by /u/Federal_Bus_4543 [link] [comments]

As someone in MLOps, I am curious to hear how other companies and teams manage the MLOps process and workflow. My company (because it's a huge enterprise) has multiple teams doing some type of MLOps or MLOps-adjacent projects. But I know that other companies do this very differently. So does your team have a separate dedicated person or a group for MLOps and managing model lifecycle in production? If not, how do you manage it? Is the data scientist / MLE expected to do all? submitted by /u/Illustrious-Pound266 [link] [comments]

Hey everyone! I started my journey in the data science world almost a year ago, and I'm wondering: What’s the best way to market myself so that I actually get noticed by recruiters and industry professionals? How do you build that presence and get on the radar of the right people? Any tips on networking, personal branding, or strategies that worked for you would be amazing to hear! submitted by /u/Aftabby [link] [comments]

Things are super slow at work due to economic uncertainty. I'm used to being super busy so I never had to think up my own problems/projects. Any ideas for useful projects I can do or sell to management? Thanks. submitted by /u/Trick-Interaction396 [link] [comments]

Help submitted by /u/marblesandcookies [link] [comments]

Picture this: You’re working in a place where every employee, contractor, and intern is plugged into a dense access matrix. Rows are users, columns are entitlements — approvals, roles, flags, mysterious group memberships with names like FIN_OPS_CONFIDENTIAL. Nobody really remembers why half of these exist. But they do. And people have them. Somewhere in there, someone has access they probably shouldn’t. Maybe they used to need it. Maybe someone clicked "approve" in 2019 and forgot. Maybe it’s just... weird. We’ve been exploring how to spot these anomalies before they turn into front-page incidents. The data looks like this: user_id → [access_1, access_2, access_3, ..., access_n] values_in_the_matrix -> [0, 1, 0 , ..., 0 This means this user has access_2 Flat. Sparse. Messy. Inherited from groups and roles sometimes. Assigned directly in other cases. Things I've tried or considered so far: LOF (Local Outlier Factor) Mixed with KNN: Treating the org as a social graph of access rights, and assuming most people should resemble their neighbors. Works okay, but choosing k (the number of neighbors) is tricky — too small and everything is an outlier; too big and nothing is. Then I tried to map each user to the nearest 10 peers and got the extra rights and missing rights they had, adding to the explainability of the solution. By telling this, [User x is an outlier because they have these [extra] rights or are missing these rights [missing] that their [peers] have. It's working, but I don't know if it is. All of that was done after I reduced the dimensionality of the matrix using SVD up to 90% explained variance to allow the Euclidean distance metric in LOF to somehow mimic cosine distance and avoid [the problem where all of the points are equally far because of the zeroes in the matrix] Clustering after SVD/UMAP: Embed people into a latent space and look for those floating awkwardly in the corner of the entitlement universe. Some light graph work: building bipartite graphs of users ↔ entitlements, then looking for rare or disconnected nodes. But none of it feels quite “safe” — or explainable enough for audit teams who still believe in spreadsheets more than scoring systems. Has anyone tackled something like this? I'm curious about: Better ways to define what “normal” access looks like. Handling inherited vs direct permissions (roles, groups, access policies). Anything that helped you avoid false positives and make results explainable. Treating access as a time series — worth it or not? Isolation Forest? Autoencoders? All I'm trying to do If you've wrangled a permission mess, cleaned up an access jungle, or just have thoughts on how to smell weirdness in high-dimensional RBAC soup — I'm all ears. How would you sniff out an access anomaly before it bites back? submitted by /u/Careful_Engineer_700 [link] [comments]

I built three MVP models for a client over 12 weeks. Nothing fancy: an LSTM, a prophet model, and XGBoost. The difficulty, as usual, was getting and understanding the data and cleaning it. The company is largely data illiterate. Turned in all 3 models, they loved it then all of a sudden canceled the pending contract to move them to production. Why? They had a devops person do in MS Copilot Analyst (a new specialized version of MS Copilot studio) and it took them 1 week! Would I like to sign a lesser contract to advise this person though? I finally looked at their code and it’s 40 lines of code using a subset of the California housing dataset run using a Random Forest regressor. They had literally nothing. My advice to them: go f*%k yourself. submitted by /u/melissa_ingle [link] [comments]

submitted by /u/bobo-the-merciful [link] [comments]

I’m based in the Bay Area with 5 YOE. A couple of months ago, I interviewed for a role I wasn’t too excited about, but the pay was super compelling. In the first recruiter call, they asked for my salary expectations. I asked for their range, as an example here, let’s say they said $150K–$180K. I said, “That works, I’m looking for something above $150K.” I think this was my first mistake, more on that later. I am a person with low self esteem(or serious imposter syndrome) and when I say I nailed all 8 rounds, I really must believe that. The recruiter followed up the day after 8th round saying team is interested in extending an offer. Then on compensation expectations the recruiter said, “You mentioned $150K earlier.” I clarified that I was targeting the upper end based on my fit and experience. They responded with, “So $180K?” and I just said yes. It felt a bit like putting words in my mouth. Next day, I got an email saying that I have to wait for the offer decision as they are interviewing other candidates. Haven’t heard back since. I don’t think I did anything fundamentally wrong or if I should have regrets but curious what others think. Edit: Just to clarify, in my mind I thought that’s how negotiations work. They will come back and say can’t do 150 but can do 140. But I guess not. submitted by /u/Lamp_Shade_Head [link] [comments]

A bit of context. I have taken charge of a project recently. It's a product in a client facing app. The implementation of the ML system is messy. The data pipelines consists of many sql codes. These codes contain rather complicated business knowledge. There is airflow that schedules them, so there is observability. This code has been used to run experiments for the past 2 months. I don't know how much firefighting has been going on. But in the past week that I picked up the project, I spent 3 days on firefighting. I understand that, at least theoretically, when scaling, everything that could go wrong goes wrong. But I want to hear real life experiences. When facing such issues, what have you done that worked? Could you find a way to fix code while helping with scaling? Did firefightings get in the way? Any past experience would help. Thanks! submitted by /u/furioncruz [link] [comments]

submitted by /u/sg6128 [link] [comments]

This specially sucks as a consultant. You get hired because some guy from Sales department of the consulting company convinced the client that they would give them a Data Scientist consultant that would solve all their problems and build perfect Machine Learning models. Then you join the client and quickly realize that is literary impossible to do any meaningful work with the poor data and the unjustified expectations they have. As an ethical worker, you work hard and to everything that is possible with the data at hand (and maybe some external data you magically gathered). You use everything that you know and don't know, take some time to study the state of the art, chat with some LLMs on their ideas for the project, run hundreds of different experiments (should I use different sets of features? Should I log transform some numerical features? Should I apply PCA? How many ML algorithms should I try?) And at the end of day... The model still sucks. You overfit the hell of the model, makes a gigantic boosting model with max_depth set as 1000, and you still don't match the dumb manager expectations. I don't know how common that it is in other professions, but an intrinsic thing of working in Data Science is that you are never sure that your work will eventually turn out to be something good, no matter how hard you try. submitted by /u/CadeOCarimbo [link] [comments]

Discussion idea inspired by that thread on tools. Bonus points if you've found anything that works on people who really think they understand GenAI but don't understand it's failure points or ways it could steer a company wrong, or those who think it's the solution to every problem. I'm currently a frustrato potato from this so any thoughts are very much appreciated submitted by /u/TaterTot0809 [link] [comments]