Sharing my second ever blog post, covering experimental design and Hypothesis testing. I shared my first blog post here a few months ago and received valuable feedback, sharing it here so I can hopefully share some value and receive some feedback as well. submitted by /u/joshamayo7 [link] [comments]

I've been casually browsing job postings lately just to stay informed about the market, and honestly, I'm starting to wonder if the classic "Data Scientist" position is becoming a thing of the past. Most of what I'm seeing falls into these categories: Data Analyst/BI roles (lots of SQL, dashboards, basic reporting) Data Engineer positions (pipelines, ETL, infrastructure stuff) AI/ML Engineer jobs (but these seem more about LLMs and deploying models than actually building them) What I'm not seeing much of anymore is that traditional data scientist role - you know, the one where you actually do statistical modeling, design experiments, and work through complex business problems from start to finish using both programming and solid stats knowledge. It makes me wonder: are companies just splitting up what used to be one data scientist job into multiple specialized roles? Or has the market just moved on from needing that "unicorn" profile that could do everything? For those of you currently working as data scientists - what does your actual day-to-day look like? Are you still doing the traditional DS work, or has your role evolved into something more specialized? And for anyone else who's been keeping an eye on the job market - am I just looking in the wrong places, or are others seeing this same trend? Just curious about where the field is heading and whether that broad, stats-heavy data scientist role still has a place in today's market. submitted by /u/ImGallo [link] [comments]

title. Im a data analyst with ~3yoe currently work at a bank. lets say i have this golden time period where my work is low stress/pressure and I can put time into preparing for interviews. My goal is to get into FAANG/finance/similar companies in data science roles. How do I prepare for interviews? Did you follow a specific structure for certain companies? How/what did you allocate time into between analytics/sql/python, ML, GenAI(if at all) or other stuff and how did you prepare? Im good w sql, currently practicing ML and GenAI projects on python. I have very basic understanding of data engg from self projects. What metrics you use to determine where you stand? I get the job market is shit but Im not ready anyway. My aim is to start interviewing by fall, say august/september. I'd highly appreciate any help i can get. thx. submitted by /u/potatotacosandwich [link] [comments]

I recently built a model using a Tweedie loss function. It performed really well, but I want to understand it better under the hood. I'd be super grateful if someone could clarify this for me. I understand that using a "Tweedie loss" just means using the negative log likelihood of a Tweedie distribution as the loss function. I also already understand how this works in the simple case of a linear model f(x_i) = wx_i, with a normal distribution negative log likelihood (i.e., the RMSE) as the loss function. You simply write out the likelihood of observing the data {(x_i, y_i) | i=1, ..., N}, given that the target variable y_i came from a normal distribution with mean f(x_i). Then you take the negative log of this, differentiate it with respect to the parameter(s), w in this case, set it equal to zero, and solve for w. This is all basic and makes sense to me; you are finding the w which maximizes the likelihood of observing the data you saw, given the assumption that the data y_i was drawn from a normal distribution with mean f(x_i) for each i. What gets me confused is using a more complex model and loss function, like LightGBM with a Tweedie loss. I figured the exact same principles would apply, but when I try to wrap my head around it, it seems I'm missing something. In the linear regression example, the "model" is y_i ~ N(f(x_i), sigma^2). In other words, you are assuming that the response variable y_i is a linear function of the independent variable x_i, plus normally distributed errors. But how do you even write this in the case of LightGBM with Tweedie loss? In my head, the analogous "model" would be y_i ~ Tw(f(x_i), phi, p), where f(x_i) is the output of the LightGBM algorithm, and f(x_i) takes the place of the mean mu in the Tweedie distribution Tw(u, phi, p). Is this correct? Are we always just treating the prediction f(x_i) as the mean of the distribution we've assumed, or is that only coincidentally true in the special case of a linear model with normal distribution NLL? submitted by /u/_hairyberry_ [link] [comments]

For those of you working on personal data science projects, what comes after the .py script or Jupyter notebook? I’m trying to move beyond exploratory work into something more usable or shareable. Is building an app the natural next step? What paths have you taken to evolve your projects once the core analysis or modeling was done? submitted by /u/Proof_Wrap_2150 [link] [comments]

Hi everyone, This may be a bit of a vent post. I got a few years in DS experience as a data analyst and then got my MSc in well ranked US school. For some reason beyond my knowledge, I’ve never been able to get a DS job after the MS degree. I got a quant job where DS is the furthest thing from it even though some stats is used, and I am now headed to a data engineering fellowship with option to renew for one more year max. I just wonder if any of this effort was worth it sometimes . I’m open to any advice or suggestions because it feels like I can’t get any lower than this. Thanks everyone Edit : thank you everyone for all the insights and kind words!!! submitted by /u/Emuthusiast [link] [comments]

I am looking for a test which can test one’s math skills that are relevant for data science- that way I can understand which areas I’m weak in and how I measure relative to my peers. Is anybody aware of anything like that? submitted by /u/Beginning-Sport9217 [link] [comments]

I built a tool that scrapes fresh data science, machine learning, and data engineering roles from FAANG and other top tech companies’ official career pages — no LinkedIn noise or recruiter spam — and emails them straight to you. What it does: Scrapes jobs directly from sites like Google, Apple, Meta, Amazon, Microsoft, Netflix, Stripe, Uber, TikTok, Airbnb, and more Sends daily emails with newly scraped jobs Helps you find openings faster – before they hit job boards Lets you select different countries like USA, Canada, India, European countries, and more Check it out here: https://topjobstoday.com/data-scientist-jobs Would love to hear your thoughts or suggestions! submitted by /u/Flaky_Literature8414 [link] [comments]

submitted by /u/ElectrikMetriks [link] [comments]

I have a data pipeline that processes spreadsheets and generates outputs. What are smart next steps to take this further without overcomplicating it? I’m thinking of building a simple GUI or dashboard to make it easier to trigger batch processing or explore outputs. I want to support month-over-month comparisons e.g. how this month’s data differs from last and then generate diffs or trend insights. Eventually I might want to track changes over time, add basic versioning, or even push summary outputs to a web format or email report. Have you done something similar? What did you add next that really improved usefulness or usability? And any advice on building GUIs for spreadsheet based workflows? I’m curious how others have expanded from here submitted by /u/Proof_Wrap_2150 [link] [comments]

Welcome to this week's entering & transitioning thread! This thread is for any questions about getting started, studying, or transitioning into the data science field. Topics include: Learning resources (e.g. books, tutorials, videos) Traditional education (e.g. schools, degrees, electives) Alternative education (e.g. online courses, bootcamps) Job search questions (e.g. resumes, applying, career prospects) Elementary questions (e.g. where to start, what next) While you wait for answers from the community, check out the FAQ and Resources pages on our wiki. You can also search for answers in past weekly threads. submitted by /u/AutoModerator [link] [comments]

submitted by /u/CanYouPleaseChill [link] [comments]

Seems like there's a lot of overlap and maybe different experts do different jobs all within the data science field, but which background would you say is most prevalent in most data science positions? submitted by /u/officialcrimsonchin [link] [comments]

Currently learning GCP. Help me stay motivated by telling me about your first cloud-related DS/ML projects. submitted by /u/corgibestie [link] [comments]

Hi all, Just recently started as a data scientist, so I thought I could use the wisdom of this subreddit before I get up to speed and compare methodologies to see what can help my team better. So say I have a dataset for a classification problem with several features (not all) that are normally distributed, and for the sake of numerical stability I’m normalizing those values to their respective Z-values (using the training set’s means and std to prevent leakage). Now after I train the model and get some results I’m happy with using the test set (that was normalized also with the training’s mean and std), we trigger some of our tests and deploy pipelines (whatever they are) and later on we’ll use that model in production with new unseen data. My question is, what is your most popular go to choice to store those mean and std values for when you’ll need to normalize the unseen data’s features prior to the prediction? The same question applies for filling null values. “Simplest” thing I thought of (with an emphasis on the “”) is a wrapper class that stores all those values as member fields along with the actual model object (or pickle file path) and storing that class also with pickle, but it sounds a bit cumbersome, so maybe you can spread some light with more efficient ideas 🙂 Cheers. submitted by /u/indie-devops [link] [comments]

I am working on a demand forecasting model to accurately predict test slots across different areas. I have been following the Rob Hyndman book. But the book essentially deals with just one feature and predicting its future values. But my model takes into account a lot of variables. How can I deal with that ? What kind of EDA should I perform ?? Is it better to make every feature stationary ? submitted by /u/NervousVictory1792 [link] [comments]

I feel stuck in the middle where my notebook works well, but it’s growing, and I know clients will add new requirements. I don’t want to introduce infrastructure I don’t need yet, but I also don’t want to be caught off guard when it’s important. How do you know when it’s time to level up, and what lightweight steps help you prepare? Any books that can help me scale my jupyter notebooks into bigger solutions? submitted by /u/Proof_Wrap_2150 [link] [comments]

I’m trying to turn a Jupyter notebook that processes 100k rows in a spreadsheet into something that can be reused across multiple datasets. I’ve considered parameterized config files but I want to hear from folks who’ve built reusable pipelines in client facing or consulting setups. submitted by /u/Proof_Wrap_2150 [link] [comments]

I’m building an analysis that processes spreadsheets, transforms the data, and outputs HTML files. It works, but it’s hard to maintain. I’m not sure if I should start modularizing into scripts, introduce config files, or just reorganize inside the notebook. Looking for advice from others who’ve scaled up from this stage. It’s easy to make it work with new files, but I can’t help but wonder what the next stage looks like? EDIT: Really appreciate all the thoughtful replies so far. I’ve made notes with some great perspectives on refactoring, modularizing, and managing complexity without overengineering. Follow-up question for those further down the path: Let’s say I do what many of you have recommended and I refactor my project into clean .py files, introduce config files, and modularize the logic into a more maintainable structure. What comes after that? I’m self taught and using this passion project as a way to build my skills. Once I’ve got something that “works well” and is well organized… what’s the next stage? Do I aim for packaging it? Turning it into a product? Adding tests? Making a CLI? I’d love to hear from others who’ve taken their passion project to the next level! How did you keep leveling up? submitted by /u/Proof_Wrap_2150 [link] [comments]

How long are your data retention policies? How do you handle GDPR rules? My company is instituting a very, very conservative retention policy of <9months of raw event-level data (but storing 15-months worth of aggregated data). Additionally, the only way this company thinks about GDPR compliance is to delete user records instead of anonymizing. I'm curious how your companies deal with both, and what the risks would be with instituting such policies. submitted by /u/timusw [link] [comments]