Hey folks, I’m working on a disaggregation framework that breaks down store-level forecasts into register-level predictions using historical proportions. I'm trying to figure out the best practice for tuning this disaggregation logic. Specifically: When tuning the register-level proportions (identifying the best approach for eg. 4 week rolling avg vs 8 week rolling avg), should I multiply them with store-level actuals (to isolate disaggregation quality), or with store-level forecasts (to reflect production-like conditions)? My thinking is: During tuning, we should use actuals to get the cleanest signal on how good our proportion logic is, without injecting forecast noise. Then during validation, we apply those tuned proportions to store-level forecasts to simulate end-to-end accuracy. A colleague argues that since we care about register-level accuracy, we should use forecasts even during tuning. I feel this risks overfitting to forecast errors, which won’t repeat in future weeks. Would love to hear how others approach this — especially in time-series setups. Do you separate tuning and validation like this, or just evaluate everything end-to-end from the start? submitted by /u/mugiwaraMorrison [link] [comments]

I’ve been feeling really confused since the time I started my current job: I was hired as a data engineer and wanted to go into DE bc over the course of several years in data analytics, I realized that I enjoy writing code to find solutions to streamline data processes. Even if “coding” itself eventually becomes “taken over” by AI, I feel finding the right method to accomplish certain tasks will always be there. However, I feel like my job has left my morale low b/c of the lack of clarity and organization as to what I’m supposed to do. Here’s what I struggle with: There’s little/no documentation standard or practices at my job. Business requirements are all over the place saved in tickets, email convos, different excel sheets, in shared folders that have tons of random documents I must sift through to understand things. This is probably the biggest reason a task assigned for few hours takes me few days There’s not much appreciation to use code like Python or SQL to find ways to streamline/automate our tasks. The people who code more are other teams, we usually end up opening excel sheets and eyeballing in terms of data validation and quality. I would love to write code to streamline our processes. However writing reasonable code takes time, so instead I find a BS interim solution where I make tons of mistakes and end up looking dumb in return My team seems very interested in “business” and to me that’s fine except I don’t even know what that means. They want to spend time in us understanding business requirements and translating it. Problem is there’s no standard as to what that process entails. Usually, business requirements given to me are email conversations and I have to wait on another person in between me and business to coordinate and set up follow up meetings that takes forever I hoped that my current role would make me more technical except I feel less technical, overworked and burned out. I feel it’s very much a team thing but I fear going to another team or another role and having the same thing happen. I wish I could have been a backend engineer or different role instead. I still like a role where there’s some domain knowledge, but I’m not getting a chance to use that knowledge or any engineering skills either. It also feels my team has no interest in me becoming more technical bc they seem to think technical skills can eventually be taken over by AI or given to a more technical team or handed off to offshore contractors. I’m confused where my career is going esp in light of this new AI hype which is making it feel like programming jobs are over now. I’m lost. Help!! submitted by /u/thro0away12 [link] [comments]

Had a phone screen with DoorDash recently for a DS Analytics role. First round was a product case study — the interviewer was super nice, gave good feedback throughout, and even ended with “Great job on this round,” so I felt pretty good about it. Second round was SQL with 4 questions. Honestly, the first one threw me off — it was more convoluted than I expected, so I struggled a bit but managed to get through it. The 2nd and 3rd were much easier and I finished those without issues. The 4th was a bonus question where I had to explain a SQL query — took me a moment, but I eventually explained what it was doing. Got a rejection email the next day. I thought it went decently overall, so I’m a bit confused. Any thoughts on what might’ve gone wrong or what I could do better next time submitted by /u/Substantial_Tank_129 [link] [comments]

Hello everyone 🙂 Serious question: Does anyone have any data science related metaphors/similes/analogies that you use regularly at work? (I want to sound smart.) Thanks! submitted by /u/idontknowotimdoing [link] [comments]

As the title suggests. I am trying to convert average wage data, from quarterly to monthly. I need to perform forecasting on that. What is the best ways to do that?? . I don’t want to go for a naive method and just divide by 3 as I will loose any trends or patterns. I have come across something called disproportionate aggregation but having a tough time grasping it. submitted by /u/NervousVictory1792 [link] [comments]

Huggingface just released an open-sourced robot named Reachy-Mini, which supports all Huggingface open-sourced AI models, be it text or speech or vision and is quite cheap. Check more details here : https://youtu.be/i6uLnSeuFMo?si=Wb6TJNjM0dinkyy5 submitted by /u/Technical-Love-8479 [link] [comments]

Im sure a lot of companies are rolling out AI products to help their business. Im curious how do people typically try to measure these AI products impacts. I guess it really depends on the domain but can we isolate and see if any uplift in the KPI is attributable to AI? Is AB testing always to gold standard? Use Quasi experimental methods? submitted by /u/Professional_Ball_58 [link] [comments]

I’m working on a project where all incoming data is provided via spreadsheets (Excel/CSV). The number of files is growing, and I need to manage them in a structured way that allows for: Easy access to different uploads over time Avoiding duplication or version confusion Interactive analysis (e.g., via Jupyter notebooks or a lightweight dashboard) I’m currently loading files manually, but I want a better system. Whether that means a file management structure, metadata tagging, or loading/parsing automation. Eventually I’d like to scale this up to support analysis across many uploads or clients. What are good patterns, tools, or Python-based workflows to support this? submitted by /u/Proof_Wrap_2150 [link] [comments]

Hi all, I am building an AI agent, similar to Github copilot / Cursor but very specialized on data science / ML. It is integrated in VSCode as an extension. Here is a few examples of use cases: - Combine different data sources, clean and preprocess for ML pipeline. - Refactor R&D notebooks into ready for production project: Docker, package, tests, documentation. We are approaching an MVP in the next few weeks and I am hesitating between 2 business models: 1- Closed source, similar to cursor, with fixed price subscription with limit by request. 2- Open source, pay per token. User can plug their own API or use our backend which offers all frontier models. Charge a topup % on top of token consumption (similar to Cline). The question is also whether the data science community would contribute to a vscode extension in React, Typescript. What do you think make senses as a data scientist / ML engineer? submitted by /u/SummerElectrical3642 [link] [comments]

I work in a data science field, and I bring this up because I think it's data science related. We have an internal website that is very bare bones. It's made to be simplistic, because it's the reference document for our end-users (1000 of them) use. Executives heard about a software that would be completely AI driven, build detailed statistical insights, and change the world as they know it. I had a demo with the company and they explained its RAG capabilities, but mentioned it doesn't really "learn" like the assumption AI does. Our repo is so small and not at all needed for AI. We have used a fuzzy search that has worked for the past three years. Additionally, I have already built out dashboards that retrieve all the information executives have asked for via API (who's viewing pages, what are they searching, etc.) I showed the c-suite executives our current dashboards in Tableau, and how the actual search works. I also explained what RAG is, and how AI/LLMs work at a high level. I explained to them that AI is a fantastic tool, but I'm not sure if we should be spending 100k a year on it. They also asked if I have built any predictive models. I don't think they quite understood what that was as well, because we don't have the amount of data or need to predict anything. Needless to say, they decided it was best not to move forward "for now". I am shocked, but also not, that executives want to change the structure of how my team and end-users digest information just because they heard "AI is awesome!" They had zero idea how anything works in our shop. Oh yeah, our company has already laid of 250 people this year due to "financial turbulence", and now they're wanting to spend 100k on this?! It just goes to show you how deep the AI train runs. Did I handle this correctly and can I put this on my resume? LOL submitted by /u/tits_mcgee_92 [link] [comments]

I’m a student interested in working as a product manager in tech. I know it’s tough to land a first role directly in PM, so I’m considering alternative paths that could lead there. My question is: how common is the transition from data scientist/product data scientist to product manager? Is it a viable path? Also would it make more sense to go down the software engineering route instead (even though I’m not particularly passionate about it) if it makes the transition to PM easier? submitted by /u/FinalRide7181 [link] [comments]

Hi everyone, I’m working on a challenge to predict the probability of a product becoming unavailable the next day. The dataset contains one row per product per day, with a binary target (failure or not) and 10 additional features. There are over 1 million rows without failure, and only 100 with failure — so it's a highly imbalanced dataset. Here are some key points I’m considering: The target should reflect the next day, not the current one. For example, if product X has data from day 1 to day 10, each row should indicate whether a failure will happen on the following day. Day 10 is used only to label day 9 and is not used as input for prediction. The features are on different scales, so I’ll need to apply normalization or standardization depending on the model I choose (e.g., for Logistic Regression or KNN). There are no missing values, so I won’t need to worry about imputation. To avoid data leakage, I’ll split the data by product, making sure that each product's full time series appears entirely in either the training or test set — never both. For example, if product X has data from day 1 to day 9, those rows must all go to either train or test. Since the output should be a probability, I’m planning to use models like Logistic Regression, Random Forest, XGBoost, Naive Bayes, or KNN. Due to the strong class imbalance, my main evaluation metric will be ROC AUC, since it handles imbalanced datasets well. Would it make sense to include calendar-based features, like the day of the week, weekend indicators, or holidays? How useful would it be to add rolling window statistics (e.g., 3-day averages or standard deviations) to capture recent trends in the attributes? Any best practices for flagging anomalies, such as sudden spikes in certain attributes or values above a specific percentile (like the 90th)? My questions: Does this approach make sense? I’m not entirely confident about some of these steps, so I’d really appreciate feedback from more experienced data scientists! submitted by /u/EducationalUse9983 [link] [comments]

Recently discovered pickup models that use reservation data to generate forecasts (see https://www.scitepress.org/papers/2016/56319/56319.pdf ) Seems used often in the hotel and airline industry. Is there a python package for this? Maybe it goes by a different name but I'm not seeing anything submitted by /u/GussieWussie [link] [comments]

submitted by /u/ElectrikMetriks [link] [comments]

Welcome to this week's entering & transitioning thread! This thread is for any questions about getting started, studying, or transitioning into the data science field. Topics include: Learning resources (e.g. books, tutorials, videos) Traditional education (e.g. schools, degrees, electives) Alternative education (e.g. online courses, bootcamps) Job search questions (e.g. resumes, applying, career prospects) Elementary questions (e.g. where to start, what next) While you wait for answers from the community, check out the FAQ and Resources pages on our wiki. You can also search for answers in past weekly threads. submitted by /u/AutoModerator [link] [comments]

So as the title suggest, last few years have been just Generative AI all over the place. Every new research is somehow focussed towards it. So does this mean other fields stands still ? Or eventually everything will merge into GenAI somehow? What's your thoughts submitted by /u/Technical-Love-8479 [link] [comments]

Hello all. To try and condense a lot of context for this question, I am an adult who went back to school to complete my bachelor's, in order to support myself and my partner on one income. Admittedly, I did this because I heard how good data science was as a field, but it seems I jumped in at the wrong time. Consequently, now that I am one year out from graduating with my bachelor's, I am starting to think about what fields would be best to apply in, beyond simply "data science" and "data analysis." Any leads on fields that are reliably hiring that are similar to data science but not exact? I am really open to anything that would pay the bills for two people. submitted by /u/fenrirbatdorf [link] [comments]

I’ve been stuck manually exporting post data from the LinkedIn analytics dashboard for months. Automating via API sounds ideal, but this is uncharted territory! submitted by /u/Proof_Wrap_2150 [link] [comments]

Hi everyone, I’ve been a job hopper throughout my career—never stayed at one place for more than 1-2 years, usually for various reasons. Now, I’m entering a phase where I want to get more settled. I’m about to start a new job and would love to hear from those who have successfully stayed long-term at a job. What’s the secret sauce besides just hard work and taking ownership? Lay your knowledge on me—your hacks, tips, rituals. Thanks in advance. submitted by /u/mlbatman [link] [comments]

Python has been largely devoid of easy to use environment and package management tooling, with various developers employing their own cocktail of pip, virtualenv, poetry, and conda to get the job done. However, it looks like uv is rapidly emerging to be a standard in the industry, and I'm super excited about it. In a nutshell uv is like npm for Python. It's also written in rust so it's crazy fast. As new ML approaches and frameworks have emerged around the greater ML space (A2A, MCP, etc) the cumbersome nature of Python environment management has transcended from an annoyance to a major hurdle. This seems to be the major reason uv has seen such meteoric adoption, especially in the ML/AI community. star history of uv vs poetry vs pip. Of course, github star history isn't necessarily emblematic of adoption. <ore importantly, uv is being used all over the shop in high-profile, cutting-edge repos that are governing the way modern software is evolving. Anthropic’s Python repo for MCP uses UV, Google’s Python repo for A2A uses UV, Open-WebUI seems to use UV, and that’s just to name a few. I wrote an article that goes over uv in greater depth, and includes some examples of uv in action, but I figured a brief pass would make a decent Reddit post. Why UV uv allows you to manage dependencies and environments with a single tool, allowing you to create isolated python environments for different projects. While there are a few existing tools in Python to do this, there's one critical feature which makes it groundbreaking: it's easy to use. Installing UV uv can be installed via curl curl -LsSf https://astral.sh/uv/install.sh | sh or via pip pipx install uv the docs have a more in-depth guide to install. Initializing a Project with UV Once you have uv installed, you can run uv init This initializes a uv project within your directory. You can think of this as an isolated python environment that's tied to your project. Adding Dependencies to your Project You can add dependencies to your project with uv add <dependency name> You can download all the dependencies you might install via pip: uv add pandas uv add scipy uv add numpy sklearn matplotlib And you can install from various other sources, including github repos, local wheel files, etc. Running Within an Environment if you have a python script within your environment, you can run it with uv run <file name> this will run the file with the dependencies and python version specified for this particular environment. This makes it super easy and convenient to bounce around between different projects. Also, if you clone a uv managed project, all dependencies will be installed and synchronized before the file is run. My Thoughts I didn't realize I've been waiting for this for a long time. I always found off the cuff quick implementation of Python locally to be a pain, and I think I've been using ephemeral environments like Colab as a crutch to get around this issue. I find local development of Python projects to be significantly more enjoyable with uv , and thus I'll likely be adopting it as my go to approach when developing in Python locally. submitted by /u/Daniel-Warfield [link] [comments]