Data science is an interdisciplinary field that uses scientific methods, processes, algorithms and systems to extract knowledge and insights from structured and unstructured data, and apply knowledge and actionable insights from data across a broad range of application domains.
In this blog, we are going to provide popular open source and public data sets, data visualization, data analytics and data lakes.
Researchers from IBM, MIT and Harvard Announced The Release Of DARPA “Common Sense AI” Dataset Along With Two Machine Learning Models At ICML 2021
Building machines that can make decisions based on common sense is no easy feat. A machine must be able to do more than merely find patterns in data; it also needs a way of interpreting the intentions and beliefs behind people’s choices.
At the 2021 International Conference on Machine Learning (ICML), Researchers from IBM, MIT, and Harvard University have come together to release a DARPA “Common Sense AI” dataset for benchmarking AI intuition. They are also releasing two machine learning models that represent different approaches to the problem that relies on testing techniques psychologists use to study infants’ behavior to accelerate the development of AI exhibiting common sense.
100 million protein structures Dataset by DeepMind
DeepMind creates ‘transformative’ map of human proteins drawn by AI. By the end of the year, DeepMind hopes to release predictions for 100 million protein structures, a dataset that will be “transformative for our understanding of how life works,”
Malware traffic dataset
Comprises 1914081 records created from all malware traffic analysis .net PCAP files, from 2013 to 2021. The logs are generated using Suricata and Zeek.
Percent of “foreign-born” population in each US and EU state or country.
For the EU, “foreign-born” mean being born outside of any of the EU countries. For the US, “foreign-born” mean being born outside of any US state 🇺🇸🇪🇺
Percent of “foreign-born” population in each US and EU state or country. For the EU, “foreign-born” mean being born outside of any of the EU countries. For the US, “foreign-born” mean being born outside of any US state.
Examples of “foreign-born” in this context:
Person born in Spain and living in France is NOT “foreign-born”
Person born in Turkey and living in France is “foreign-born”
Person born in Florida and living in Texas is NOT “foreign-born”
Person born in Mexico and living in Texas is “foreign-born”
Person born in Florida and living in France is “foreign-born”
Person born in France and living in Florida is “foreign-born”
Note: Poland, Ireland, Germany, Greece, Cyprus, Malta, Portugal uses Eurostat 2010 Migration data and Croatia has no data at all
Tools: MS Office
35% of “entry-level” jobs on LinkedIn require 3+ years of experience
Tool: Photoshop from my colleague
Latest complete Netflix movie dataset
Created from 4 APIs. 11K+ rows and 30+ attributes of Netflix (Ratings, earnings, actors, language, availability, movie trailers, and many more)
Explore this dataset using FlixGem.com (this dataset is powering this webapp)
A corpus of web crawl data composed of over 50 billion web pages. The Common Crawl corpus contains petabytes of data collected since 2008. It contains raw web page data, extracted metadata and text extractions.
AWS CLI Access (No AWS account required)
aws s3 ls s3://commoncrawl/ --no-sign-request
Dataset on protein prices
Data on Primary Commodity Prices are updated monthly based on the IMF’s Primary Commodity Price System.
The University of Chicago Project on Security and Threats presents the updated and expanded Database on Suicide Attacks (DSAT), which now links to Uppsala Conflict Data Program data on armed conflicts and includes a new dataset measuring the alliance and rivalry relationships among militant groups with connections to suicide attack groups. Access it here.
You can do a lot of aggregated analysis in a pretty straightforward way there.
11 TB dataset of drone imagery with annotations for small object detection and tracking
Download and more information are available here
Dataset License: CDLA-Sharing-1.0
Helper scripts for accessing the dataset: DATASET.md
Dataset Exploration: Colab
The HRRR is a NOAA real-time 3-km resolution, hourly updated, cloud-resolving, convection-allowing atmospheric model, initialized by 3km grids with 3km radar assimilation. Radar data is assimilated in the HRRR every 15 min over a 1-h period adding further detail to that provided by the hourly data assimilation from the 13km radar-enhanced Rapid Refresh.
This registry exists to help people discover and share datasets that are available via AWS resources. Learn more about sharing data on AWS.
See datasets from Digital Earth Africa, Facebook Data for Good, NASA Space Act Agreement, NIH STRIDES, NOAA Big Data Program, Space Telescope Science Institute, and Amazon Sustainability Data Initiative.
1,076 textbook lessons, 26,260 questions, 6229 images
The GDC Data Portal is a robust data-driven platform that allows cancer
researchers and bioinformaticians to search and download cancer data for analysis.
The Cancer Genome Atlas (TCGA), a collaboration between the National Cancer Institute (NCI) and National Human Genome Research Institute (NHGRI), aims to generate comprehensive, multi-dimensional maps of the key genomic changes in major types and subtypes of cancer.
AWS CLI Access (No AWS account required)
aws s3 ls s3://tcga-2-open/ --no-sign-request
The Therapeutically Applicable Research to Generate Effective Treatments (TARGET) program applies a comprehensive genomic approach to determine molecular changes that drive childhood cancers. The goal of the program is to use data to guide the development of effective, less toxic therapies. TARGET is organized into a collaborative network of disease-specific project teams. TARGET projects provide comprehensive molecular characterization to determine the genetic changes that drive the initiation and progression of childhood cancers. The dataset contains open Clinical Supplement, Biospecimen Supplement, RNA-Seq Gene Expression Quantification, miRNA-Seq Isoform Expression Quantification, miRNA-Seq miRNA Expression Quantification data from Genomic Data Commons (GDC), and open data from GDC Legacy Archive. Access it here.
The Genome Aggregation Database (gnomAD) is a resource developed by an international coalition of investigators that aggregates and harmonizes both exome and genome data from a wide range of large-scale human sequencing projects. The summary data provided here are released for the benefit of the wider scientific community without restriction on use. Downloads
Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable. Access it here.
The Pubmed Diabetes dataset consists of 19717 scientific publications from PubMed database pertaining to diabetes classified into one of three classes. The citation network consists of 44338 links. Each publication in the dataset is described by a TF/IDF weighted word vector from a dictionary which consists of 500 unique words. The README file in the dataset provides more details.
This dataset contains interactions between drugs and targets collected from DrugBank, KEGG Drug, DCDB, and Matador. It was originally collected by Perlman et al. It contains 315 drugs, 250 targets, 1,306 drug-target interactions, 5 types of drug-drug similarities, and 3 types of target-target similarities. Drug-drug similarities include Chemical-based, Ligand-based, Expression-based, Side-effect-based, and Annotation-based similarities. Target-target similarities include Sequence-based, Protein-protein interaction network-based, and Gene Ontology-based similarities. The original task on the dataset is to predict new interactions between drugs and targets based on different types of similarities in the network. Download link
PharmGKB data and knowledge is available as downloads. It is often critical to check with their curators at firstname.lastname@example.org before embarking on a large project using these data, to be sure that the files and data they make available are being interpreted correctly. PharmGKB generally does NOT need to be a co-author on such analyses; They just want to make sure that there is a correct understanding of our data before lots of resources are spent.
The dataset contains open RNA-Seq Gene Expression Quantification data and controlled WGS/WXS/RNA-Seq Aligned Reads, WXS Annotated Somatic Mutation, WXS Raw Somatic Mutation, and RNA-Seq Splice Junction Quantification. Documentation
AWS CLI Access (No AWS account required)
aws s3 ls s3://gdc-organoid-pancreatic-phs001611-2-open/ --no-sign-request
This dataset contains soil infrared spectral data and paired soil property reference measurements for georeferenced soil samples that were collected through the Africa Soil Information Service (AfSIS) project, which lasted from 2009 through 2018. Documentation
AWS CLI Access (No AWS account required)
aws s3 ls s3://afsis/ --no-sign-request
DAiSEE is the first multi-label video classification dataset comprising of 9068 video snippets captured from 112 users for recognizing the user affective states of boredom, confusion, engagement, and frustration “in the wild”. The dataset has four levels of labels namely – very low, low, high, and very high for each of the affective states, which are crowd annotated and correlated with a gold standard annotation created using a team of expert psychologists. Download it here.
NatureServe Explorer provides conservation status, taxonomy, distribution, and life history information for more than 95,000 plants and animals in the United States and Canada, and more than 10,000 vegetation communities and ecological systems in the Western Hemisphere.
The data available through NatureServe Explorer represents data managed in the NatureServe Central Databases. These databases are dynamic, being continually enhanced and refined through the input of hundreds of natural heritage program scientists and other collaborators. NatureServe Explorer is updated from these central databases to reflect information from new field surveys, the latest taxonomic treatments and other scientific publications, and new conservation status assessments. Explore Data here
Airline On-Time Performance and Causes of Flight Delays – On_Time Data.
This database contains scheduled and actual departure and arrival times, reason of delay. reported by certified U.S. air carriers that account for at least one percent of domestic scheduled passenger revenues. The data is collected by the Office of Airline Information, Bureau of Transportation Statistics (BTS).
FlightAware.com has data but you need to pay for a full dataset.
anyflights package supplies a set of functions to generate air travel data (and data packages!) similar to
nycflights13. With a user-defined year and airport, the
anyflights function will grab data on:
flights: all flights that departed a given airport in a given year and month
weather: hourly meterological data for a given airport in a given year and month
airports: airport names, FAA codes, and locations
airlines: translation between two letter carrier (airline) codes and names
planes: construction information about each plane found in
The U.S. Department of Transportation’s (DOT) Bureau of Transportation Statistics (BTS) tracks the on-time performance of domestic flights operated by large air carriers. Summary information on the number of on-time, delayed, canceled and diverted flights appears in DOT’s monthly Air Travel Consumer Report, published about 30 days after the month’s end, as well as in summary tables posted on this website. BTS began collecting details on the causes of flight delays in June 2003. Summary statistics and raw data are made available to the public at the time the Air Travel Consumer Report is released. Access it here
Download: airports.dat (Airports only, high quality)
Download: airports-extended.dat (Airports, train stations and ferry terminals, including user contributions)
Flightera.net seems to have a lot of good data for free. It has in-depth data on flights and doesn’t seem limited by date. I can’t speak on the validity of the data though.
flightradar24.com has lots of data, also historically, they might be willing to help you get it in a nice format.
Dataset with arrest in US by race and separate states. Download Excel here
Yahoo Answers DataSets
Yahoo is shutting down in 2021. This is Yahoo Answers datasets (300MB gzip) that is fairly extensive from 2015 with about 1.4m rows. This dataset has the best questions answers, I mean all the answers, including the most insane awful answers and the worst questions people put together. Download it here.
Another option here: According to the tracker, there are 77M done, 20M out(?), and 40M to go:
This is a dataset of about 55K Persian words with their phonetics. Each word is in a line and separated from its phonetic by a tab. Download it here
Air Quality Data Collected at Outdoor Monitors Across the US. This is a BigQuery Dataset. There are no files to download, but you can query it through Kernels using the BigQuery API. The AQS Data Mart is a database containing all of the information from AQS. It has every measured value the EPA has collected via the national ambient air monitoring program. It also includes the associated aggregate values calculated by EPA (8-hour, daily, annual, etc.). The AQS Data Mart is a copy of AQS made once per week and made accessible to the public through web-based applications. The intended users of the Data Mart are air quality data analysts in the regulatory, academic, and health research communities. It is intended for those who need to download large volumes of detailed technical data stored at EPA and does not provide any interactive analytical tools. It serves as the back-end database for several Agency interactive tools that could not fully function without it: AirData, AirCompare, The Remote Sensing Information Gateway, the Map Monitoring Sites KML page, etc.
Stack Exchange Dataset
This list of a topic-centric public data sources in high quality. They are collected and tidied from blogs, answers, and user responses. Most of the data sets listed below are free, however, some are not.
- The global dataset of historical yields for major crops 1981–2016 – The […]
- Hyperspectral benchmark dataset on soil moisture – This dataset was […]
- Lemons quality control dataset – Lemon dataset has been prepared to […]
- Optimized Soil Adjusted Vegetation Index – The IDB is a tool for working […]
- U.S. Department of Agriculture’s Nutrient Database
- U.S. Department of Agriculture’s PLANTS Database – The Complete PLANTS […]
- 1000 Genomes – The 1000 Genomes Project ran between 2008 and 2015, […]
- American Gut (Microbiome Project) – The American Gut project is the […]
- Broad Bioimage Benchmark Collection (BBBC) – The Broad Bioimage Benchmark […]
- Broad Cancer Cell Line Encyclopedia (CCLE)
- Cell Image Library – This library is a public and easily accessible […]
- Complete Genomics Public Data – A diverse data set of whole human genomes […]
- EBI ArrayExpress – ArrayExpress Archive of Functional Genomics Data […]
- EBI Protein Data Bank in Europe – The Electron Microscopy Data Bank […]
- ENCODE project – The Encyclopedia of DNA Elements (ENCODE) Consortium is […]
- Electron Microscopy Pilot Image Archive (EMPIAR) – EMPIAR, the Electron […]
- Ensembl Genomes
- Gene Expression Omnibus (GEO) – GEO is a public functional genomics data […]
- Gene Ontology (GO) – GO annotation files
- Global Biotic Interactions (GloBI)
- Harvard Medical School (HMS) LINCS Project – The Harvard Medical School […]
- Human Genome Diversity Project – A group of scientists at Stanford […]
- Human Microbiome Project (HMP) – The HMP sequenced over 2000 reference […]
- ICOS PSP Benchmark – The ICOS PSP benchmarks repository contains an […]
- International HapMap Project
- Journal of Cell Biology DataViewer [fixme]
- KEGG – KEGG is a database resource for understanding high-level functions […]
- MIT Cancer Genomics Data
- NCBI Proteins
- NCBI Taxonomy – The NCBI Taxonomy database is a curated set of names and […]
- NCI Genomic Data Commons – The GDC Data Portal is a robust data-driven […]
- NIH Microarray data
- OpenSNP genotypes data – openSNP allows customers of direct-to-customer […]
- Palmer Penguins – The goal of palmerpenguins is to provide a great […]
- Pathguid – Protein-Protein Interactions Catalog
- Protein Data Bank – This resource is powered by the Protein Data Bank […]
- Psychiatric Genomics Consortium – The purpose of the Psychiatric Genomics […]
- PubChem Project – PubChem is the world’s largest collection of freely […]
- PubGene (now Coremine Medical) – COREMINE™ is a family of tools developed […]
- Sanger Catalogue of Somatic Mutations in Cancer (COSMIC) – COSMIC, the […]
- Sanger Genomics of Drug Sensitivity in Cancer Project (GDSC)
- Sequence Read Archive(SRA) – The Sequence Read Archive (SRA) stores raw […]
- Stanford Microarray Data
- Stowers Institute Original Data Repository
- Systems Science of Biological Dynamics (SSBD) Database – Systems Science […]
- The Cancer Genome Atlas (TCGA), available via Broad GDAC
- The Catalogue of Life – The Catalogue of Life is a quality-assured […]
- The Personal Genome Project – The Personal Genome Project, initiated in […]
- UCSC Public Data
- Universal Protein Resource (UnitProt) – The Universal Protein Resource […]
- Rfam – The Rfam database is a collection of RNA families, each […]
- Actuaries Climate Index
- Australian Weather
- Aviation Weather Center – Consistent, timely and accurate weather […]
- Brazilian Weather – Historical data (In Portuguese) – Data related to […]
- Canadian Meteorological Centre
- Climate Data from UEA (updated monthly)
- Dutch Weather – The KNMI Data Center (KDC) portal provides access to KNMI […]
- European Climate Assessment & Dataset
- German Climate Data Center
- Global Climate Data Since 1929
- Charting The Global Climate Change News Narrative 2009-2020 – These four […]
- NASA Global Imagery Browse Services
- NOAA Bering Sea Climate [fixme]
- NOAA Climate Datasets
- NOAA Realtime Weather Models
- NOAA SURFRAD Meteorology and Radiation Datasets
- The World Bank Open Data Resources for Climate Change
- UEA Climatic Research Unit
- WU Historical Weather Worldwide
- Wahington Post Climate Change – To analyze warming temperatures in the […]
- WorldClim – Global Climate Data
- AMiner Citation Network Dataset
- CrossRef DOI URLs
- DBLP Citation dataset
- DIMACS Road Networks Collection
- NBER Patent Citations
- NIST complex networks data collection
- Network Repository with Interactive Exploratory Analysis Tools [fixme]
- Protein-protein interaction network
- PyPI and Maven Dependency Network
- Scopus Citation Database
- Small Network Data
- Stanford GraphBase
- Stanford Large Network Dataset Collection
- Stanford Longitudinal Network Data Sources [fixme]
- The Koblenz Network Collection
- The Laboratory for Web Algorithmics (UNIMI)
- UCI Network Data Repository
- UFL sparse matrix collection
- WSU Graph Database [fixme]
- Community Resource for Archiving Wireless Data At Dartmouth – Contains […]
- 3.5B Web Pages from CommonCrawl 2012
- 53.5B Web clicks of 100K users in Indiana Univ.
- CAIDA Internet Datasets
- CRAWDAD Wireless datasets from Dartmouth Univ. [fixme]
- ClueWeb09 – 1B web pages
- ClueWeb12 – 733M web pages
- CommonCrawl Web Data over 7 years
- Criteo click-through data
- Internet-Wide Scan Data Repository
- MIRAGE-2019 – MIRAGE-2019 is a human-generated dataset for mobile traffic […]
- OONI: Open Observatory of Network Interference – Internet censorship data
- Open Mobile Data by MobiPerf
- The Peer-to-Peer Trace Archive – Real-world measurements play a key role […]
- Rapid7 Sonar Internet Scans
- UCSD Network Telescope, IPv4 /8 net
- CCCS-CIC-AndMal-2020 – The dataset includes 200K benign and 200K malware […]
- Traffic and Log Data Captured During a Cyber Defense Exercise – This […]
- Bruteforce Database
- Challenges in Machine Learning
- CrowdANALYTIX dataX [fixme]
- D4D Challenge of Orange [fixme]
- DrivenData Competitions for Social Good
- ICWSM Data Challenge (since 2009)
- KDD Cup by Tencent 2012
- Kaggle Competition Data
- Localytics Data Visualization Challenge
- Netflix Prize
- Space Apps Challenge
- Telecom Italia Big Data Challenge [fixme]
- TravisTorrent Dataset – MSR’2017 Mining Challenge
- TunedIT – Data mining & machine learning data sets, algorithms, challenges [fixme]
- Yelp Dataset Challenge [fixme]
- 38-Cloud (Cloud Detection) – Contains 38 Landsat 8 scene images and their […]
- AQUASTAT – Global water resources and uses
- BODC – marine data of ~22K vars
- EOSDIS – NASA’s earth observing system data
- Earth Models [fixme]
- Global Wind Atlas – The Global Wind Atlas is a free, web-based […]
- Integrated Marine Observing System (IMOS) – roughly 30TB of ocean measurements
- Marinexplore – Open Oceanographic Data
- Alabama Real-Time Coastal Observing System
- National Estuarine Research Reserves System-Wide Monitoring Program – […]
- Oil and Gas Authority Open Data – The dataset covers 12,500 offshore […]
- Smithsonian Institution Global Volcano and Eruption Database
- USGS Earthquake Archives
- American Economic Association (AEA)
- EconData from UMD
- Economic Freedom of the World Data
- Historical MacroEconomic Statistics
- INFORUM – Interindustry Forecasting at the University of Maryland
- DBnomics – the world’s economic database – Aggregates hundreds of […]
- International Trade Statistics
- Internet Product Code Database
- Joint External Debt Data Hub
- Jon Haveman International Trade Data Links
- Long-Term Productivity Database – The Long-Term Productivity database was […]
- OpenCorporates Database of Companies in the World
- Our World in Data
- SciencesPo World Trade Gravity Datasets [fixme]
- The Atlas of Economic Complexity
- The Center for International Data
- The Observatory of Economic Complexity [fixme]
- UN Commodity Trade Statistics
- UN Human Development Reports
- College Scorecard Data
- New York State Education Department Data – The New York State Education […]
- Student Data from Free Code Camp
- AMPds – The Almanac of Minutely Power dataset
- BLUEd – Building-Level fUlly labeled Electricity Disaggregation dataset
- DBFC – Direct Borohydride Fuel Cell (DBFC) Dataset
- DEL – Domestic Electrical Load study datsets for South Africa (1994 – 2014)
- ECO – The ECO data set is a comprehensive data set for non-intrusive load […]
- Global Power Plant Database – The Global Power Plant Database is a […]
- HES – Household Electricity Study, UK
- PEM1 – Proton Exchange Membrane (PEM) Fuel Cell Dataset
- PLAID – The Plug Load Appliance Identification Dataset [fixme]
- The Public Utility Data Liberation Project (PUDL) – PUDL makes US energy […]
- SYND – A synthetic energy dataset for non-intrusive load monitoring – […]
- Smart Meter Data Portal – The Smart Meter Data Portal is part of the […]
- Ukraine Energy Centre Datasets
- UK-DALE – UK Domestic Appliance-Level Electricity
- BIS Statistics – BIS statistics, compiled in cooperation with central […]
- Blockmodo Coin Registry – A registry of JSON formatted information files […]
- CBOE Futures Exchange
- Complete FAANG Stock data – This data set contains all the stock data of […]
- Google Finance
- Google Trends
- NASDAQ [fixme]
- NYSE Market Data
- OSU Financial data [fixme]
- St Louis Federal
- Yahoo Finance
- Awesome 3D Semantic City Models – Collection of open 3D semantic city and […]
- ArcGIS Open Data portal
- Cambridge, MA, US, GIS data on GitHub
- Database of all continents, countries, States/Subdivisions/Provinces and […]
- Factual Global Location Data
- IEEE Geoscience and Remote Sensing Society DASE Website
- Geo Maps – High Quality GeoJSON maps programmatically generated
- Geo Spatial Data from ASU
- Geo Wiki Project – Citizen-driven Environmental Monitoring
- GeoFabrik – OSM data extracted to a variety of formats and areas
- GeoNames Worldwide
- Global Administrative Areas Database (GADM) – Geospatial data organized […]
- Homeland Infrastructure Foundation-Level Data
- Landsat 8 on AWS
- List of all countries in all languages
- National Weather Service GIS Data Portal
- Natural Earth – vectors and rasters of the world [fixme]
- OpenStreetMap (OSM)
- Pleiades – Gazetteer and graph of ancient places
- Reverse Geocoder using OSM data
- Robin Wilson – Free GIS Datasets
- TIGER/Line – U.S. boundaries and roads
- TZ Timezones shapefile
- TwoFishes – Foursquare’s coarse geocoder
- UN Environmental Data
- World boundaries from the U.S. Department of State
- World countries in multiple formats
- Alberta, Province of Canada
- Antwerp, Belgium
- Argentina (non official) [fixme]
- Datos Argentina – Portal de datos abiertos de la República Argentina. […]
- Austin, TX, US
- Australia (abs.gov.au)
- Australia (data.gov.au)
- Austria (data.gv.at)
- Baton Rouge, LA, US
- Beersheba, Israel – Open Data Portal (Smart7 OpenData)
- City of Berkeley Open Data
- Buenos Aires, Argentina
- Calgary, AB, Canada
- Cambridge, MA, US
- China [fixme]
- Dallas Open Data
- DataBC – data from the Province of British Columbia
- Debt to the Penny – The Debt to the Penny dataset provides information […]
- Denver Open Data
- Durham, NC Open Data
- Edmonton, AB, Canada
- England LGInform
- EveryPolitician – Ongoing project collating and sharing data on every […]
- Federal Committee on Statistical Methodology (FCSM) (formerly FedStats)
- Fredericton, NB, Canada
- Gatineau, QC, Canada
- Ghent, Belgium
- Glasgow, Scotland, UK [fixme]
- Guardian world governments
- Halifax, NS, Canada
- Helsinki Region, Finland
- Hong Kong, China
- Houston, TX, US [fixme]
- Indian Government Data
- Indonesian Data Portal
- Iowa – Welcome to the State of Iowa’s data portal. Please explore data […]
- Ireland’s Open Data Portal
- Israel’s Open Data Portal
- Istanbul Municipality Open Data Portal
- Italy – Il Portale dati.gov.it è il catalogo nazionale dei metadati […]
- Jail deaths in America – The U.S. government does not release jail by […]
- Laval, QC, Canada
- Lexington, KY
- London Datastore, UK
- London, ON, Canada [fixme]
- Los Angeles Open Data
- Luxembourg – Luxembourgish Open Data Portal
- MassGIS, Massachusetts, U.S.
- Metropolitan Transportation Commission (MTC), California, US
- Mexico [fixme]
- Mississauga, ON, Canada
- Moncton, NB, Canada
- Montreal, QC, Canada
- Mountain View, California, US (GIS)
- NYC Open Data [fixme]
- NYC betanyc
- New York Department of Sanitation Monthly Tonnage – DSNY Monthly Tonnage […]
- New Zealand
- Oakland, California, US [fixme]
- Open Data for Africa
- Open Government Data (OGD) Platform India
- OpenDataSoft’s list of 1,600 open data
- Ottawa, ON, Canada
- Palo Alto, California, US
- OpenDataPhilly – OpenDataPhilly is a catalog of open data in the […]
- Portland, Oregon
- Portugal – Pordata organization
- Puerto Rico Government
- Quebec City, QC, Canada [fixme]
- Quebec Province of Canada
- Regina SK, Canada
- Rio de Janeiro, Brazil
- Russia [fixme]
- San Diego, CA
- San Antonio, TX – Community Information Now – CI:Now is a nonprofit […] [fixme]
- San Francisco Data sets
- San Jose, California, US
- San Mateo County, California, US
- Saskatchewan, Province of Canada
- Singapore Government Data
- South Africa Trade Statistics
- South Africa
- State of Utah, US
- Taiwan gov
- Tel-Aviv Open Data
- Texas Open Data
- The World Bank [fixme]
- Toronto, ON, Canada [fixme]
- Tunisia [fixme]
- U.K. Government Data
- U.S. American Community Survey
- U.S. CDC Public Health datasets
- U.S. Census Bureau
- U.S. Department of Housing and Urban Development (HUD)
- U.S. Federal Government Agencies
- U.S. Federal Government Data Catalog
- U.S. Food and Drug Administration (FDA)
- U.S. National Center for Education Statistics (NCES)
- U.S. Open Government
- UK 2011 Census Open Atlas Project
- US Counties – This is a repository of various data, broken down by US […]
- U.S. Patent and Trademark Office (USPTO) Bulk Data Products
- Uganda Bureau of Statistics [fixme]
- United Nations
- Valley Transportation Authority (VTA), California, US
- Vancouver, BC Open Data Catalog [fixme]
- Victoria, BC, Canada
- Vienna, Austria
- Statistics from the General Statistics Office of Vietnam – Data in […] [fixme]
- U.S. Congressional Research Service (CRS) Reports
- AWS COVID-19 Datasets – We’re working with organizations who make […]
- COVID-19 Case Surveillance Public Use Data – The COVID-19 case […]
- 2019 Novel Coronavirus COVID-19 Data Repository by Johns Hopkins CSSE – […]
- Coronavirus (Covid-19) Data in the United States – The New York Times is […]
- COVID-19 Reported Patient Impact and Hospital Capacity by Facility – The […]
- Composition of Foods Raw, Processed, Prepared USDA National Nutrient Database for Standard […]
- The COVID Tracking Project – The COVID Tracking Project collects and […]
- EHDP Large Health Data Sets
- GDC – GDC supports several cancer genome programs for CCG, TCGA, TARGET etc.
- Gapminder World demographic databases
- MeSH, the vocabulary thesaurus used for indexing articles for PubMed
- MeDAL – A large medical text dataset curated for abbreviation […]
- Medicare Coverage Database (MCD), U.S.
- Medicare Data Engine of medicare.gov Data
- Medicare Data File
- Number of Ebola Cases and Deaths in Affected Countries (2014)
- Open-ODS (structure of the UK NHS)
- OpenPaymentsData, Healthcare financial relationship data
- PhysioBank Databases – A large and growing archive of physiological data.
- The Cancer Imaging Archive (TCIA)
- The Cancer Genome Atlas project (TCGA)
- World Health Organization Global Health Observatory
- Yahoo Knowledge Graph COVID-19 Datasets – The Yahoo Knowledge Graph team […]
- Informatics for Integrating Biology & the Bedside [fixme]
- 10k US Adult Faces Database
- 2GB of Photos of Cats
- Audience Unfiltered faces for gender and age classification
- Affective Image Classification
- Animals with attributes
- CADDY Underwater Stereo-Vision Dataset of divers’ hand gestures – […]
- Cytology Dataset – CCAgT: Images of Cervical Cells with AgNOR Stain […]
- Caltech Pedestrian Detection Benchmark
- Chars74K dataset – Character Recognition in Natural Images (both English […]
- Cube++ – 4890 raw 18-megapixel images, each containing a SpyderCube color […]
- Danbooru Tagged Anime Illustration Dataset – A large-scale anime image […]
- DukeMTMC Data Set – DukeMTMC aims to accelerate advances in multi-target […] [fixme]
- ETH Entomological Collection (ETHEC) Fine Grained Butterfly (Lepidoptra) Images
- Face Recognition Benchmark
- Flickr: 32 Class Brand Logos [fixme]
- GDXray – X-ray images for X-ray testing and Computer Vision
- HumanEva Dataset – The HumanEva-I dataset contains 7 calibrated video […]
- ImageNet (in WordNet hierarchy)
- Indoor Scene Recognition
- International Affective Picture System, UFL
- KITTI Vision Benchmark Suite
- Labeled Information Library of Alexandria – Biology and Conservation – […]
- MNIST database of handwritten digits, near 1 million examples [fixme]
- Multi-View Region of Interest Prediction Dataset for Autonomous Driving – […]
- Massive Visual Memory Stimuli, MIT
- Newspaper Navigator – This dataset consists of extracted visual content […]
- Open Images From Google – Pictures with segmentation masks for 2.8 […]
- RuFa – Contains images of text written in one of two Arabic fonts (Ruqaa […]
- SUN database, MIT
- SVIRO Synthetic Vehicle Interior Rear Seat Occupancy – 25.000 synthetic […]
- Several Shape-from-Silhouette Datasets [fixme]
- Stanford Dogs Dataset
- The Action Similarity Labeling (ASLAN) Challenge
- The Oxford-IIIT Pet Dataset
- Violent-Flows – Crowd Violence / Non-violence Database and benchmark
- Visual genome
- YouTube Faces Database
- All-Age-Faces Dataset – Contains 13’322 Asian face images distributed […]
- Audi Autonomous Driving Dataset – We have published the Audi Autonomous […]
- Context-aware data sets from five domains
- Delve Datasets for classification and regression
- Discogs Monthly Data
- Free Music Archive
- IMDb Database
- Iranis – A Large-scale Dataset of Farsi/Arabic License Plate Characters
- Keel Repository for classification, regression and time series
- Labeled Faces in the Wild (LFW)
- Lending Club Loan Data
- Machine Learning Data Set Repository [fixme]
- Million Song Dataset
- More Song Datasets
- MovieLens Data Sets
- New Yorker caption contest ratings
- RDataMining – “R and Data Mining” ebook data
- Registered Meteorites on Earth [fixme]
- Restaurants Health Score Data in San Francisco
- TikTok Dataset – More than 300 dance videos that capture a single person […]
- UCI Machine Learning Repository
- Yahoo! Ratings and Classification Data
- Youtube 8m
- eBay Online Auctions (2012)
- Canada Science and Technology Museums Corporation’s Open Data
- Cooper-Hewitt’s Collection Database
- Metropolitan Museum of Art Collection API
- Minneapolis Institute of Arts metadata
- Natural History Museum (London) Data Portal
- Rijksmuseum Historical Art Collection
- Tate Collection metadata
- The Getty vocabularies
- Automatic Keyphrase Extraction
- The Big Bad NLP Database
- Blizzard Challenge Speech – The speech + text data comes from […]
- Blogger Corpus
- CLiPS Stylometry Investigation Corpus [fixme]
- ClueWeb09 FACC
- ClueWeb12 FACC
- DBpedia – 4.58M things with 583M facts
- Dirty Words – With millions of images in our library and billions of […]
- Flickr Personal Taxonomies
- Freebase of people, places, and things [fixme]
- German Political Speeches Corpus – Collection of political speeches from […]
- Google Books Ngrams (2.2TB)
- Google MC-AFP – Generated based on the public available Gigaword dataset […]
- Google Web 5gram (1TB, 2006)
- Gutenberg eBooks List [fixme]
- Hansards text chunks of Canadian Parliament
- LJ Speech – Speech dataset consisting of 13,100 short audio clips of a […]
- M-AILabs Speech – The M-AILABS Speech Dataset is the first large dataset […] [fixme]
- Microsoft MAchine Reading COmprehension Dataset (or MS MARCO)
- Machine Comprehension Test (MCTest) of text from Microsoft Research
- Machine Translation of European languages
- Making Sense of Microposts 2013 – Concept Extraction [fixme]
- Making Sense of Microposts 2016 – Named Entity rEcognition and Linking
- Multi-Domain Sentiment Dataset (version 2.0)
- Noisy speech database for training speech enhancement algorithms and TTS […] [fixme]
- Open Multilingual Wordnet
- POS/NER/Chunk annotated data
- Personae Corpus [fixme]
- SMS Spam Collection in English
- SaudiNewsNet Collection of Saudi Newspaper Articles (Arabic, 30K articles)
- Stanford Question Answering Dataset (SQuAD)
- USENET postings corpus of 2005~2011
- Universal Dependencies
- Webhose – News/Blogs in multiple languages
- Wikidata – Wikipedia databases
- Wikipedia Links data – 40 Million Entities in Context
- WordNet databases and tools
- WorldTree Corpus of Explanation Graphs for Elementary Science Questions – […]
- Allen Institute Datasets
- Brain Catalogue
- CodeNeuro Datasets [fixme]
- Collaborative Research in Computational Neuroscience (CRCNS)
- Human Connectome Project
- NIMH Data Archive
- NeuroMorpho – NeuroMorpho.Org is a centrally curated inventory of […]
- Study Forrest
- CERN Open Data Portal
- Crystallography Open Database
- IceCube – South Pole Neutrino Observatory
- Ligo Open Science Center (LOSC) – Gravitational wave data from the LIGO […]
- NASA Exoplanet Archive
- NSSDC (NASA) data of 550 space spacecraft
- Sloan Digital Sky Survey (SDSS) – Mapping the Universe
- EOPC-DE-Early-Onset-Prostate-Cancer-Germany – Early Onset Prostate Cancer […]
- GENIE – Data from the Genomics Evidence Neoplasia Information Exchange […]
- Genomic-Hallmarks-Prostate-Adenocarcinoma-CPC-GENE – Comprehensive […]
- MSK-IMPACT-Clinical-Sequencing-Cohort-MSKCC-Prostate-Cancer – Targeted […]
- Metastatic-Prostate-Adenocarcinoma-MCTP – Comprehensive profiling of 61 […]
- Metastatic-Prostate-Cancer-SU2CPCF-Dream-Team – Comprehensive analysis of […]
- NPCR-2001-2015 – Database from CDC’s National Program of Cancer […]
- NPCR-2005-2015 – Database from CDC’s National Program of Cancer […]
- NaF-Prostate – NaF Prostate is a collection of F-18 NaF positron emission […]
- Neuroendocrine-Prostate-Cancer – Whole exome and RNA Seq data of […]
- PLCO-Prostate-Diagnostic-Procedures – The Prostate Diagnostic Procedures […]
- PLCO-Prostate-Medical-Complications – The Prostate Medical Complications […]
- PLCO-Prostate-Screening-Abnormalities – The Prostate Screening […]
- PLCO-Prostate-Screening – The Prostate Screening dataset (177,315 […]
- PLCO-Prostate-Treatments – The Prostate Treatments dataset (13,409 […]
- PLCO-Prostate – The Prostate dataset is a comprehensive dataset that […]
- PRAD-CA-Prostate-Adenocarcinoma-Canada – Prostate Adenocarcinoma – […]
- PRAD-FR-Prostate-Adenocarcinoma-France – Prostate Adenocarcinoma – […]
- PRAD-UK-Prostate-Adenocarcinoma-United-Kingdom – Prostate Adenocarcinoma […]
- PROSTATEx-Challenge – Retrospective set of prostate MR studies. All […]
- Prostate-3T – The Prostate-3T project provided imaging data to TCIA as […]
- Prostate-Adenocarcinoma-Broad-Cornell-2012 – Comprehensive profiling of […]
- Prostate-Adenocarcinoma-Broad-Cornell-2013 – Comprehensive profiling of […]
- Prostate-Adenocarcinoma-CNA-study-MSKCC – Copy-number profiling of 103 […]
- Prostate-Adenocarcinoma-Fred-Hutchinson-CRC – Comprehensive profiling of […]
- Prostate Adenocarcinoma (MSKCC/DFCI) – Whole Exome Sequencing of 1013 […]
- Prostate-Adenocarcinoma-MSKCC – MSKCC Prostate Oncogenome Project. 181 […]
- Prostate-Adenocarcinoma-Organoids-MSKCC – Exome profiling of prostate […]
- Prostate-Adenocarcinoma-Sun-Lab – Whole-genome and Transcriptome […]
- Prostate-Adenocarcinoma-TCGA-PanCancer-Atlas – Comprehensive TCGA […]
- Prostate-Adenocarcinoma-TCGA – Integrated profiling of 333 primary […]
- Prostate-Diagnosis – PCa T1- and T2-weighted magnetic resonance images […]
- Prostate-Fused-MRI-Pathology – The Prostate Fused-MRI-Pathology […]
- Prostate-MRI – The Prostate-MRI collection of prostate Magnetic Resonance […]
- Prostate-R – The R package ‘ElemStatLearn’ contains a prostate cancer […]
- QIN-PROSTATE-Repeatability – The QIN-PROSTATE-Repeatability dataset is a […]
- QIN-PROSTATE – The QIN PROSTATE collection of the Quantitative Imaging […]
- SEER-YR1973_2015.SEER9 – The SEER November 2017 Research Data files from […]
- SEER-YR1992_2015.SJ_LA_RG_AK – The SEER November 2017 Research Data files […]
- SEER-YR2000_2015.CA_KY_LO_NJ_GA – The SEER November 2017 Research Data […]
- SEER-YR2000_2015.CA_KY_LO_NJ_GA – The July – December 2005 diagnoses for […]
- TCGA-PRAD-US – TCGA Prostate Adenocarcinoma (499 samples).
- Ably Open Realtime Data
- Archive.org Datasets
- Archive-it from Internet Archive
- CMU JASA data archive
- CMU StatLab collections
- Data360 [fixme]
- Enigma Public
- Grand Comics Database – The Grand Comics Database (GCD) is a nonprofit, […]
- Infochimps [fixme]
- KDNuggets Data Collections
- Microsoft Azure Data Market Free DataSets [fixme]
- Microsoft Data Science for Research
- Microsoft Research Open Data
- Open Library Data Dumps
- Reddit Datasets
- RevolutionAnalytics Collection [fixme]
- Sample R data sets
- Stats4Stem R data sets (archived)
- The Washington Post List
- UCLA SOCR data collection
- UFO Reports
- Wikileaks 911 pager intercepts
- Yahoo Webscope
- Academic Torrents of data sharing from UMB
- Domains Project – Sorted list of Internet domains
- Harvard Dataverse Network of scientific data
- ICPSR (UMICH)
- Institute of Education Sciences
- National Technical Reports Library
- Open Data Certificates (beta)
- OpenDataNetwork – A search engine of all Socrata powered data portals
- Statista.com – statistics and Studies
- Zenodo – An open dependable home for the long-tail of science
- 2021 Portuguese Elections Twitter Dataset – 57M+ tweets, 1M+ users – This […]
- 72 hours #gamergate Twitter Scrape
- CMU Enron Email of 150 users
- Cheng-Caverlee-Lee September 2009 – January 2010 Twitter Scrape
- China Biographical Database – The China Biographical Database is a freely […]
- A Twitter Dataset of 40+ million tweets related to COVID-19 – Due to the […]
- 43k+ Donald Trump Twitter Screenshots – This archive contains screenshots […]
- EDRM Enron EMail of 151 users, hosted on S3
- Facebook Data Scrape (2005)
- Facebook Social Connectedness Index – We use an anonymized snapshot of […]
- Facebook Social Networks from LAW (since 2007)
- Foursquare from UMN/Sarwat (2013)
- GitHub Collaboration Archive
- Google Scholar citation relations
- High-Resolution Contact Networks from Wearable Sensors
- Indie Map: social graph and crawl of top IndieWeb sites
- Mobile Social Networks from UMASS
- Network Twitter Data
- Reddit Comments
- Skytrax’ Air Travel Reviews Dataset
- Social Twitter Data
- SourceForge.net Research Data
- Twitch Top Streamer’s Data
- Twitter Data for Online Reputation Management
- Twitter Data for Sentiment Analysis
- Twitter Graph of entire Twitter site
- Twitter Scrape Calufa May 2011 [fixme]
- UNIMI/LAW Social Network Datasets
- United States Congress Twitter Data – Daily datasets with tweets of 1100+ […]
- Yahoo! Graph and Social Data
- Youtube Video Social Graph in 2007,2008
- ACLED (Armed Conflict Location & Event Data Project)
- Authoritarian Ruling Elites Database – The Authoritarian Ruling Elites […]
- Canadian Legal Information Institute
- Center for Systemic Peace Datasets – Conflict Trends, Polities, State Fragility, etc [fixme]
- Correlates of War Project
- Cryptome Conspiracy Theory Items
- Datacards [fixme]
- European Social Survey
- FBI Hate Crime 2013 – aggregated data
- Fragile States Index [fixme]
- GDELT Global Events Database
- General Social Survey (GSS) since 1972
- German Social Survey
- Global Religious Futures Project
- Gun Violence Data – A comprehensive, accessible database that contains […]
- Humanitarian Data Exchange
- INFORM Index for Risk Management
- Institute for Demographic Studies
- International Networks Archive
- International Social Survey Program ISSP
- International Studies Compendium Project
- James McGuire Cross National Data
- MIT Reality Mining Dataset
- MacroData Guide by Norsk samfunnsvitenskapelig datatjeneste
- Mass Mobilization Data Project – The Mass Mobilization (MM) data are an […]
- Microsoft Academic Knowledge Graph – The Microsoft Academic Knowledge […]
- Minnesota Population Center
- Notre Dame Global Adaptation Index (ND-GAIN)
- Open Crime and Policing Data in England, Wales and Northern Ireland
- OpenSanctions – A global database of persons and companies of political, […]
- Paul Hensel General International Data Page
- PewResearch Internet Survey Project
- PewResearch Society Data Collection
- Political Polarity Data [fixme]
- StackExchange Data Explorer
- Terrorism Research and Analysis Consortium
- Texas Inmates Executed Since 1984
- Titanic Survival Data Set
- UCB’s Archive of Social Science Data (D-Lab) [fixme]
- UCLA Social Sciences Data Archive
- UN Civil Society Database
- UPJOHN for Labor Employment Research
- Universities Worldwide
- Uppsala Conflict Data Program
- World Bank Open Data
- World Inequality Database – The World Inequality Database (WID.world) […]
- WorldPop project – Worldwide human population distributions
- FLOSSmole data about free, libre, and open source software development
- GHTorrent – Scalable, queryable, offline mirror of data offered through […]
- Libraries.io Open Source Repository and Dependency Metadata
- Public Git Archive – a Big Code dataset for all – dataset of 182,014 top- […]
- Code duplicates – 2k Java file and 600 Java function pairs labeled as […]
- Commit messages – 1.3 billion GitHub commit messages till March 2019
- Pull Request review comments – 25.3 million GitHub PR review comments […]
- Source Code Identifiers – 41.7 million distinct splittable identifiers […]
- American Ninja Warrior Obstacles – Contains every obstacle in the history […]
- Betfair Historical Exchange Data
- Cricsheet Matches (cricket)
- Equity in Athletics – The Equity in Athletics Data Analysis Cutting Tool […]
- Ergast Formula 1, from 1950 up to date (API)
- Football/Soccer resources (data and APIs)
- Lahman’s Baseball Database
- NFL play-by-play data – NFL play-by-play data sourced from: […]
- Pinhooker: Thoroughbred Bloodstock Sale Data
- Pro Kabadi season 1 to 7 – Pro Kabadi League is a professional-level […]
- Retrosheet Baseball Statistics
- Tennis database of rankings, results, and stats for ATP
- Tennis database of rankings, results, and stats for WTA
- USA Soccer Teams and Locations – USA soccer teams and locations. MLS, […]
- 3W dataset – To the best of its authors’ knowledge, this is the first […]
- Databanks International Cross National Time Series Data Archive
- Hard Drive Failure Rates
- Heart Rate Time Series from MIT
- Time Series Data Library (TSDL) from MU
- Turing Change Point Dataset – Contains 42 annotated time series collected […]
- UC Riverside Time Series Dataset
- Airlines OD Data 1987-2008
- Ford GoBike Data (formerly Bay Area Bike Share Data) [fixme]
- Bike Share Systems (BSS) collection
- Dutch Traffic Information
- GeoLife GPS Trajectory from Microsoft Research
- German train system by Deutsche Bahn
- Hubway Million Rides in MA [fixme]
- Montreal BIXI Bike Share
- NYC Taxi Trip Data 2009-
- NYC Taxi Trip Data 2013 (FOIA/FOILed)
- NYC Uber trip data April 2014 to September 2014
- Open Traffic collection
- OpenFlights – airport, airline and route data
- Philadelphia Bike Share Stations (JSON)
- Plane Crash Database, since 1920
- RITA Airline On-Time Performance data [fixme]
- RITA/BTS transport data collection (TranStat) [fixme]
- Renfe (Spanish National Railway Network) dataset
- Toronto Bike Share Stations (JSON and GBFS files)
- Transport for London (TFL)
- Travel Tracker Survey (TTS) for Chicago [fixme]
- U.S. Bureau of Transportation Statistics (BTS)
- U.S. Domestic Flights 1990 to 2009
- U.S. Freight Analysis Framework since 2007
- U.S. National Highway Traffic Safety Administration – Fatalities since […]
- CS:GO Competitive Matchmaking Data – In this data set we have data about […]
- FIFA-2021 Complete Player Dataset
- OpenDota data dump
- Data Packaged Core Datasets
- Database of Scientific Code Contributions
- A growing collection of public datasets: CoolDatasets.
- DataWrangling: Some Datasets Available on the Web
- Inside-r: Finding Data on the Internet
- OpenDataMonitor: An overview of available open data resources in Europe
- Quora: Where can I find large datasets open to the public?
- RS.io: 100+ Interesting Data Sets for Statistics
- StaTrek: Leveraging open data to understand urban lives
- CV Papers: CV Datasets on the web
- CVonline: Image Databases
- Cross-Platform – Writing cross-platform code on Node.js.
- Frontend Development
- iOS – Mobile operating system for Apple phones and tablets.
- Android – Mobile operating system developed by Google.
- IoT & Hybrid Apps
- Xamarin – Mobile app development IDE, testing, and distribution.
- macOS – Operating system for Apple’s Mac computers.
- watchOS – Operating system for the Apple Watch.
- Amazon Web Services
- IPFS – P2P hypermedia protocol.
- Fuse – Mobile development tools.
- Heroku – Cloud platform as a service.
- Raspberry Pi – Credit card-sized computer aimed at teaching kids programming, but capable of a lot more.
- Qt – Cross-platform GUI app framework.
- WebExtensions – Cross-browser extension system.
- RubyMotion – Write cross-platform native apps for iOS, Android, macOS, tvOS, and watchOS in Ruby.
- Smart TV – Create apps for different TV platforms.
- GNOME – Simple and distraction-free desktop environment for Linux.
- KDE – A free software community dedicated to creating an open and user-friendly computing experience.
- Amazon Alexa – Virtual home assistant.
- DigitalOcean – Cloud computing platform designed for developers.
- Flutter – Google’s mobile SDK for building native iOS and Android apps from a single codebase written in Dart.
- Home Assistant – Open source home automation that puts local control and privacy first.
- IBM Cloud – Cloud platform for developers and companies.
- Firebase – App development platform built on Google Cloud Platform.
- Robot Operating System 2.0 – Set of software libraries and tools that help you build robot apps.
- Adafruit IO – Visualize and store data from any device.
- Cloudflare – CDN, DNS, DDoS protection, and security for your site.
- Actions on Google – Developer platform for Google Assistant.
- ESP – Low-cost microcontrollers with WiFi and broad IoT applications.
- DOS – Operating system for x86-based personal computers that was popular during the 1980s and early 1990s.
- Nix – Package manager for Linux and other Unix systems that makes package management reliable and reproducible.
- Standard Style – Style guide and linter.
- Must Watch Talks
- Network Layer
- Micro npm Packages
- Mad Science npm Packages – Impossible sounding projects that exist.
- Maintenance Modules – For npm packages.
- npm – Package manager.
- AVA – Test runner.
- ESLint – Linter.
- Functional Programming
- npm scripts – Task runner.
- 30 Seconds of Code – Code snippets you can understand in 30 seconds.
- Ponyfills – Like polyfills but without overriding native APIs.
- Swift – Apple’s compiled programming language that is secure, modern, programmer-friendly, and fast.
- Python – General-purpose programming language designed for readability.
- Asyncio – Asynchronous I/O in Python 3.
- Scientific Audio – Scientific research in audio/music.
- CircuitPython – A version of Python for microcontrollers.
- Data Science – Data analysis and machine learning.
- Typing – Optional static typing for Python.
- MicroPython – A lean and efficient implementation of Python 3 for microcontrollers.
- Scala Native – Optimizing ahead-of-time compiler for Scala based on LLVM.
- Julia – High-level dynamic programming language designed to address the needs of high-performance numerical analysis and computational science.
- C/C++ – General-purpose language with a bias toward system programming and embedded, resource-constrained software.
- R – Functional programming language and environment for statistical computing and graphics.
- Common Lisp – Powerful dynamic multiparadigm language that facilitates iterative and interactive development.
- Java – Popular secure object-oriented language designed for flexibility to “write once, run anywhere”.
- PHP – Server-side scripting language.
- Composer – Package manager.
- Frege – Haskell for the JVM.
- CMake – Build, test, and package software.
- ActionScript 3 – Object-oriented language targeting Adobe AIR.
- Eta – Functional programming language for the JVM.
- Idris – General purpose pure functional programming language with dependent types influenced by Haskell and ML.
- Ada/SPARK – Modern programming language designed for large, long-lived apps where reliability and efficiency are essential.
- Q# – Domain-specific programming language used for expressing quantum algorithms.
- Vala – Programming language designed to take full advantage of the GLib and GNOME ecosystems, while preserving the speed of C code.
- Coq – Formal language and environment for programming and specification which facilitates interactive development of machine-checked proofs.
- V – Simple, fast, safe, compiled language for developing maintainable software.
- ES6 Tools
- Web Performance Optimization
- Web Tools
- CSS – Style sheet language that specifies how HTML elements are displayed on screen.
- React – App framework.
- Web Components
- Angular – App framework.
- Backbone – App framework.
- HTML5 – Markup language used for websites & web apps.
- SVG – XML-based vector image format.
- Ember – App framework.
- Android UI
- iOS UI
- Web Typography
- Web Accessibility
- Material Design
- D3 – Library for producing dynamic, interactive data visualizations.
- Web Audio
- Static Website Services
- Text Editing
- Motion UI Design
- Vue.js – App framework.
- Marionette.js – App framework.
- Aurelia – App framework.
- Ionic Framework 2
- Chrome DevTools
- PostCSS – CSS tool.
- Draft.js – Rich text editor framework for React.
- Service Workers
- Progressive Web Apps
- choo – App framework.
- webpack – Module bundler.
- Browserify – Module bundler.
- Sass – CSS preprocessor.
- Ant Design – Enterprise-class UI design language.
- Less – CSS preprocessor.
- Preact – App framework.
- Progressive Enhancement
- Next.js – Framework for server-rendered React apps.
- WordPress-Gatsby – Web development technology stack with WordPress as a back end and Gatsby as a front end.
- Mobile Web Development – Creating a great mobile web experience.
- Storybook – Development environment for UI components.
- Blazor – .NET web framework using C#/Razor and HTML that runs in the browser with WebAssembly.
- PageSpeed Metrics – Metrics to help understand page speed and user experience.
- Tailwind CSS – Utility-first CSS framework for rapid UI development.
- Seed – Rust framework for creating web apps running in WebAssembly.
- Web Performance Budget – Techniques to ensure certain performance metrics for a website.
- Yew – Rust framework inspired by Elm and React for creating multi-threaded frontend web apps with WebAssembly.
- Material-UI – Material Design React components for faster and easier web development.
- Building Blocks for Web Apps – Standalone features to be integrated into web apps.
- Svelte – App framework.
- Design systems – Collection of reusable components, guided by rules that ensure consistency and speed.
- Flask – Python framework.
- Vagrant – Automation virtual machine environment.
- Pyramid – Python framework.
- Play1 Framework
- CakePHP – PHP framework.
- Symfony – PHP framework.
- Laravel – PHP framework.
- Rails – Web app framework for Ruby.
- Gems – Packages.
- Phalcon – PHP framework.
- nginx – Web server.
- Dropwizard – Java framework.
- Kubernetes – Open-source platform that automates Linux container operations.
- Lumen – PHP micro-framework.
- Serverless Framework – Serverless computing and serverless architectures.
- Apache Wicket – Java web app framework.
- Vert.x – Toolkit for building reactive apps on the JVM.
- Terraform – Tool for building, changing, and versioning infrastructure.
- Vapor – Server-side development in Swift.
- Dash – Python web app framework.
- FastAPI – Python web app framework.
- CDK – Open-source software development framework for defining cloud infrastructure in code.
- IAM – User accounts, authentication and authorization.
- Chalice – Python framework for serverless app development on AWS Lambda.
- University Courses
- Data Science
- Machine Learning
- ML with Ruby – Learning, implementing, and applying Machine Learning using Ruby.
- Core ML Models – Models for Apple’s machine learning framework.
- H2O – Open source distributed machine learning platform written in Java with APIs in R, Python, and Scala.
- Software Engineering for Machine Learning – From experiment to production-level machine learning.
- AI in Finance – Solving problems in finance with machine learning.
- JAX – Automatic differentiation and XLA compilation brought together for high-performance machine learning research.
- Speech and Natural Language Processing
- Papers – Theory basics for using cryptography by non-cryptographers.
- Computer Vision
- Deep Learning – Neural networks.
- TensorFlow – Library for machine intelligence.
- TensorFlow Lite – Framework that optimizes TensorFlow models for on-device machine learning.
- Papers – The most cited deep learning papers.
- Deep Vision
- Open Source Society University
- Functional Programming
- Empirical Software Engineering – Evidence-based research on software systems.
- Static Analysis & Code Quality
- Information Retrieval – Learn to develop your own search engine.
- Quantum Computing – Computing which utilizes quantum mechanics and qubits on quantum computers.
- Big Data
- Public Datasets
- Hadoop – Framework for distributed storage and processing of very large data sets.
- Data Engineering
- Apache Spark – Unified engine for large-scale data processing.
- Qlik – Business intelligence platform for data visualization, analytics, and reporting apps.
- Splunk – Platform for searching, monitoring, and analyzing structured and unstructured machine-generated big data in real-time.
- Papers We Love
- Education – Learning and practicing.
- Algorithm Visualizations
- Artificial Intelligence
- Search Engine Optimization
- Competitive Programming
- Recursion Schemes – Traversing nested data structures.
- Sublime Text
- Atom – Open-source and hackable text editor.
- Visual Studio Code – Cross-platform open-source text editor.
- Game Development
- Game Talks
- Godot – Game engine.
- Open Source Games
- Unity – Game engine.
- LÖVE – Game engine.
- PICO-8 – Fantasy console.
- Game Boy Development
- Construct 2 – Game engine.
- Gideros – Game engine.
- Minecraft – Sandbox video game.
- Game Datasets – Materials and datasets for Artificial Intelligence in games.
- Haxe Game Development – A high-level strongly typed programming language used to produce cross-platform native code.
- libGDX – Java game framework.
- PlayCanvas – Game engine.
- Game Remakes – Actively maintained open-source game remakes.
- Flame – Game engine for Flutter.
- Discord Communities – Chat with friends and communities.
- CHIP-8 – Virtual computer game machine from the 70s.
- Games of Coding – Learn a programming language by making games.
- Quick Look Plugins – For macOS.
- Dev Env
- Fish – User-friendly shell.
- Command-Line Apps
- ZSH Plugins
- GitHub – Hosting service for Git repositories.
- Git Cheat Sheet & Git Flow
- Git Tips
- Git Add-ons – Enhance the
- Git Hooks – Scripts for automating tasks during
- FOSS for Developers
- Hyper – Cross-platform terminal app built on web technologies.
- PowerShell – Cross-platform object-oriented shell.
- Alfred Workflows – Productivity app for macOS.
- Terminals Are Sexy
- GitHub Actions – Create tasks to automate your workflow and share them with others on GitHub.
- MongoDB – NoSQL database.
- TinkerPop – Graph computing framework.
- PostgreSQL – Object-relational database.
- CouchDB – Document-oriented NoSQL database.
- HBase – Distributed, scalable, big data store.
- NoSQL Guides – Help on using non-relational, distributed, open-source, and horizontally scalable databases.
- Contexture – Abstracts queries/filters and results/aggregations from different backing data stores like ElasticSearch and MongoDB.
- Database Tools – Everything that makes working with databases easier.
- Grakn – Logical database to organize large and complex networks of data as one body of knowledge.
- Creative Commons Media
- Codeface – Text editor fonts.
- Stock Resources
- GIF – Image format known for animated images.
- Open Source Documents
- Audio Visualization
- Pixel Art – Pixel-level digital art.
- FFmpeg – Cross-platform solution to record, convert and stream audio and video.
- Icons – Downloadable SVG/PNG/font icon projects.
- Audiovisual – Lighting, audio and video in professional environments.
- CLI Workshoppers – Interactive tutorials.
- Learn to Program
- Tech Videos
- Dive into Machine Learning
- Computer History
- Programming for Kids
- Educational Games – Learn while playing.
- CSS Learning – Mainly about CSS – the language and the modules.
- Product Management – Learn how to be a better product manager.
- Roadmaps – Gives you a clear route to improve your knowledge and skills.
- YouTubers – Watch video tutorials from YouTubers that teach you about technology.
- Application Security
- CTF – Capture The Flag.
- Malware Analysis
- Android Security
- Honeypots – Deception trap, designed to entice an attacker into attempting to compromise the information systems in an organization.
- Incident Response
- Vehicle Security and Car Hacking
- Web Security – Security of web apps & services.
- Lockpicking – The art of unlocking a lock by manipulating its components without the key.
- Cybersecurity Blue Team – Groups of individuals who identify security flaws in information technology systems.
- Fuzzing – Automated software testing technique that involves feeding pseudo-randomly generated input data.
- Embedded and IoT Security
- GDPR – Regulation on data protection and privacy for all individuals within EU.
- DevSecOps – Integration of security practices into DevOps.
- Refinery CMS – Ruby on Rails CMS.
- Wagtail – Django CMS focused on flexibility and user experience.
- Textpattern – Lightweight PHP-based CMS.
- Drupal – Extensible PHP-based CMS.
- Craft CMS – Content-first CMS.
- Sitecore – .NET digital marketing platform that combines CMS with tools for managing multiple websites.
- Silverstripe CMS – PHP MVC framework that serves as a classic or headless CMS.
- Internet of Things
- Electronics – For electronic engineers and hobbyists.
- Bluetooth Beacons
- Electric Guitar Specifications – Checklist for building your own electric guitar.
- Plotters – Computer-controlled drawing machines and other visual art robots.
- Robotic Tooling – Free and open tools for professional robotic development.
- LIDAR – Sensor for measuring distances by illuminating the target with laser light.
- Open Companies
- Places to Post Your Startup
- OKR Methodology – Goal setting & communication best practices.
- Leading and Managing – Leading people and being a manager in a technology company/environment.
- Indie – Independent developer businesses.
- Tools of the Trade – Tools used by companies on Hacker News.
- Clean Tech – Fighting climate change with technology.
- Wardley Maps – Provides high situational awareness to help improve strategic planning and decision making.
- Social Enterprise – Building an organization primarily focused on social impact that is at least partially self-funded.
- Engineering Team Management – How to transition from software development to engineering management.
- Developer-First Products – Products that target developers as the user.
- Slack – Team collaboration.
- Remote Jobs
- Niche Job Boards
- Programming Interviews
- Code Review – Reviewing code.
- Creative Technology – Businesses & groups that specialize in combining computing, design, art, and user experience.
- Software-Defined Networking
- Network Analysis
- Real-Time Communications – Network protocols for near simultaneous exchange of media and data.
- Bitcoin – Bitcoin services and tools for software developers.
- Ripple – Open source distributed settlement network.
- Non-Financial Blockchain – Non-financial blockchain applications.
- Mastodon – Open source decentralized microblogging network.
- Ethereum – Distributed computing platform for smart contract development.
- Blockchain AI – Blockchain projects for artificial intelligence and machine learning.
- EOSIO – A decentralized operating system supporting industrial-scale apps.
- Corda – Open source blockchain platform designed for business.
- Waves – Open source blockchain platform and development toolset for Web 3.0 apps and decentralized solutions.
- Substrate – Framework for writing scalable, upgradeable blockchains in Rust.
- Computational Neuroscience – A multidisciplinary science which uses computational approaches to study the nervous system.
- Digital History – Computer-aided scientific investigation of history.
- Scientific Writing – Distraction-free scientific writing with Markdown, reStructuredText and Jupyter notebooks.
- Creative Tech Events – Events around the globe for creative coding, tech, design, music, arts and cool stuff.
- Events in Italy – Tech-related events in Italy.
- Events in the Netherlands – Tech-related events in the Netherlands.
- Testing – Software testing.
- Visual Regression Testing – Ensures changes did not break the functionality or style.
- Selenium – Open-source browser automation framework and ecosystem.
- Appium – Test automation tool for apps.
- TAP – Test Anything Protocol.
- JMeter – Load testing and performance measurement tool.
- k6 – Open-source, developer-centric performance monitoring and load testing solution.
- Playwright – Node.js library to automate Chromium, Firefox and WebKit with a single API.
- Quality Assurance Roadmap – How to start & build a career in software testing.
- JSON – Text based data interchange format.
- CSV – A text file format that stores tabular data and uses a comma to separate values.
- Discounts for Student Developers
- Awesome – Recursion illustrated.
- Continuous Integration and Continuous Delivery
- Services Engineering
- Free for Developers
- Answers – Stack Overflow, Quora, etc.
- Sketch – Design app for macOS.
- Boilerplate Projects
- Design and Development Guides
- Software Engineering Blogs
- Self Hosted
- FOSS Production Apps
- Gulp – Task runner.
- AMA – Ask Me Anything.
- Open Source Photography
- OpenGL – Cross-platform API for rendering 2D and 3D graphics.
- Research Tools
- Data Visualization
- Social Media Share Links
- Unicode – Unicode standards, quirks, packages and resources.
- Beginner-Friendly Projects
- Tools for Activism
- Citizen Science – For community-based and non-institutional scientists.
- MQTT – “Internet of Things” connectivity protocol.
- Hacking Spots
- For Girls
- Vorpal – Node.js CLI framework.
- Vulkan – Low-overhead, cross-platform 3D graphics and compute API.
- LaTeX – Typesetting language.
- Economics – An economist’s starter kit.
- Funny Markov Chains
- Cheminformatics – Informatics techniques applied to problems in chemistry.
- Colorful – Choose your next color scheme.
- Steam – Digital distribution platform.
- Bots – Building bots.
- Site Reliability Engineering
- Empathy in Engineering – Building and promoting more compassionate engineering cultures.
- DTrace – Dynamic tracing framework.
- Userscripts – Enhance your browsing experience.
- Pokémon – Pokémon and Pokémon GO.
- ChatOps – Managing technical and business operations through a chat.
- Falsehood – Falsehoods programmers believe in.
- Domain-Driven Design – Software development approach for complex needs by connecting the implementation to an evolving model.
- Quantified Self – Self-tracking through technology.
- SaltStack – Python-based config management system.
- Web Design – For digital designers.
- Creative Coding – Programming something expressive instead of something functional.
- No-Login Web Apps – Web apps that work without login.
- Free Software – Free as in freedom.
- Framer – Prototyping interactive UI designs.
- Markdown – Markup language.
- Dev Fun – Funny developer projects.
- Healthcare – Open source healthcare software for facilities, providers, developers, policy experts, and researchers.
- Magento 2 – Open Source eCommerce built with PHP.
- TikZ – Graph drawing packages for TeX/LaTeX/ConTeXt.
- Neuroscience – Study of the nervous system and brain.
- Ad-Free – Ad-free alternatives.
- Esolangs – Programming languages designed for experimentation or as jokes rather than actual use.
- Prometheus – Open-source monitoring system.
- Homematic – Smart home devices.
- Ledger – Double-entry accounting on the command-line.
- Web Monetization – A free open web standard service that allows you to send money directly in your browser.
- Uncopyright – Public domain works.
- Crypto Currency Tools & Algorithms – Digital currency where encryption is used to regulate the generation of units and verify transfers.
- Diversity – Creating a more inclusive and diverse tech community.
- Open Source Supporters – Companies that offer their tools and services for free to open source projects.
- Design Principles – Create better and more consistent designs and experiences.
- Theravada – Teachings from the Theravada Buddhist tradition.
- inspectIT – Open source Java app performance management tool.
- Open Source Maintainers – The experience of being an open source maintainer.
- Calculators – Calculators for every platform.
- Captcha – A type of challenge–response test used in computing to determine whether or not the user is human.
- Jupyter – Create and share documents that contain code, equations, visualizations and narrative text.
- FIRST Robotics Competition – International high school robotics championship.
- Humane Technology – Open source projects that help improve society.
- Speakers – Conference and meetup speakers in the programming and design community.
- Board Games – Table-top gaming fun for all.
- Software Patreons – Fund individual programmers or the development of open source projects.
- Parasite – Parasites and host-pathogen interactions.
- Food – Food-related projects on GitHub.
- Mental Health – Mental health awareness and self-care in the software industry.
- Bitcoin Payment Processors – Start accepting Bitcoin.
- Scientific Computing – Solving complex scientific problems using computers.
- Amazon Sellers
- Agriculture – Open source technology for farming and gardening.
- Product Design – Design a product from the initial concept to production.
- Prisma – Turn your database into a GraphQL API.
- Software Architecture – The discipline of designing and building software.
- Connectivity Data and Reports – Better understand who has access to telecommunication and internet infrastructure and on what terms.
- Stacks – Tech stacks for building different apps and features.
- Cytodata – Image-based profiling of biological phenotypes for computational biologists.
- IRC – Open source messaging protocol.
- Advertising – Advertising and programmatic media for websites.
- Earth – Find ways to resolve the climate crisis.
- Naming – Naming things in computer science done right.
- Biomedical Information Extraction – How to extract information from unstructured biomedical data and text.
- Web Archiving – An effort to preserve the Web for future generations.
- WP-CLI – Command-line interface for WordPress.
- Credit Modeling – Methods for classifying credit applicants into risk classes.
- Ansible – A Python-based, open source IT configuration management and automation platform.
- Biological Visualizations – Interactive visualization of biological data on the web.
- QR Code – A type of matrix barcode that can be used to store and share a small amount of information.
- Veganism – Making the plant-based lifestyle easy and accessible.
- Translations – The transfer of the meaning of a text from one language to another.
- All Awesome Lists – All the Awesome lists on GitHub.
- Awesome Indexed – Search the Awesome dataset.
- Awesome Search – Quick search for Awesome lists.
- StumbleUponAwesome – Discover random pages from the Awesome dataset using a browser extension.
- Awesome CLI – A simple command-line tool to dive into Awesome lists.
- Awesome Viewer – A visualizer for all of the above Awesome lists.
US Department of Education CRDC Dataset
The US Department of Ed has a dataset called the CRDC that collects data from all the public schools in the US and has demographic, academic, financial and all sorts of other fun data points. They also have corollary datasets that use the same identifier—an expansion pack if you may. It comes out every 2-3 years. Access it here
Nasa Dataset: sequencing data from bacteria before and after being taken to space
NASA has some sequencing data from bacteria before and after being taken to space, to look at genetic differences caused by lack of gravity, radiation and others. Very fun if you want to try your hand at some bio data science. Access it here.
Extracted from the NYT story: here
Data is plural
Data is Plural is a really good newsletter published by Jeremy Singer-Vine. The datasets are very random, but super interesting. Access it here.
Global terrorism database
Huge list of terrorism incidents from inside the US and abroad. Each entry has date and location of the incident, motivations, whether people or property were lost, the size of the attack, type of attack, etc. Access it here
Terrorist Attacks Dataset: This dataset consists of 1293 terrorist attacks each assigned one of 6 labels indicating the type of the attack. Each attack is described by a 0/1-valued vector of attributes whose entries indicate the absence/presence of a feature. There are a total of 106 distinct features. The files in the dataset can be used to create two distinct graphs. The README file in the dataset provides more details. Download Link:
Terrorists: This dataset contains information about terrorists and their relationships. This dataset was designed for classification experiments aimed at classifying the relationships among terrorists. The dataset contains 851 relationships, each described by a 0/1-valued vector of attributes where each entry indicates the absence/presence of a feature. There are a total of 1224 distinct features. Each relationship can be assigned one or more labels out of a maximum of four labels making this dataset suitable for multi-label classification tasks. The README file provides more details. Download Link
The dolphin social network
This network dataset is in the category of Social Networks. A social network of bottlenose dolphins. The dataset contains a list of all of links, where a link represents frequent associations between dolphins. Access it here
Dataset of 200,000 jokes
There are about 208 000 jokes in this database scraped from three sources.
The Million Song Dataset
The Million Song Dataset is a freely-available collection of audio features and metadata for a million contemporary popular music tracks.
Its purposes are:
- To encourage research on algorithms that scale to commercial sizes
- To provide a reference dataset for evaluating research
- As a shortcut alternative to creating a large dataset with APIs (e.g. The Echo Nest’s)
- To help new researchers get started in the MIR field
Cornell University’s eBird dataset
Decades of observations of birds all around the world, truly an impressive way to leverage citizen science. Access it here.
UFO Report Dataset
NUFORC geolocated and time standardized ufo reports for close to a century of data. 80,000 plus reports. Access it here
CDC’s Trend Drug Data
The CDC has a public database called NAMCS/NHAMCS that allows you to trend drug data. It has a lot of other data points so it can be used for a variety of other reasons. Access it here.
Health and Retirement study: Public Survey data
A listing of publicly available biennial, off-year, and cross-year data products.
Example: COVID-19 Data
|2020||2020 HRS COVID-19 Project|
HRS data products produced by the RAND Center for the Study of Aging.
HRS data products produced by the USC Program on Global Aging, Health, and Policy.
Data products (unsupported by the HRS) provided by researchers sharing their work.
The Quick Draw Dataset
The Quick Draw Dataset is a collection of 50 million drawings across 345 categories, contributed by players of the game Quick, Draw!. The drawings were captured as timestamped vectors, tagged with metadata including what the player was asked to draw and in which country the player was located. Access it here.
The AirNow API replaces the previous AirNow Gateway web services. It includes file outputs and RSS data feeds. AirNow Gateway users can use their existing login information to access the new AirNow API web pages and web services. Access to the AirNow API is generally available to the public, and new accounts can be acquired via the Log In page
Search and extract the measurements from 600 Wastewater Treatment Sites owned and operated by UK Water Companies and part of the Chemical Investigations Programme (CIP2).
The M4 competition which is a continuation of the Makridakis Competitions for forecasting and was conducted in 2018. This competion includes the prediction of both Point Forecasts and Prediction Intervals.
Used by Google’s deep-learning program for determining the 3D shapes of proteins stands to transform biology, say scientists. Access it here.
In computer science, Artificial Intelligence (AI) is intelligence demonstrated by machines. Its definition, AI research as the study of “intelligent agents”: any device that perceives its environment and takes actions that achieving its goals Russell et. al (2016).
Withal, Data Mining (DM) is the process of discovering patterns in data sets (or datasets) involving methods of machine learning, statistics, and database systems; DM focus on extract the information of datasets Han (2011).
This repository serves as a guide for anyone who wants to work with Artificial Intelligence or Data Mining applied in digital games! Here you will find a series of datasets, tools and materials available to build your application or dataset. Access it here.
Help Predict whether teachers’ project proposals are accepted
The Squirrel Census is a multimedia science, design, and storytelling project focusing on the Eastern gray (Sciurus carolinensis). They count squirrels and present their findings to the public.
BigQuery public datasets are made available without any restrictions to all Google Cloud users. Google pays for the storage of these datasets. You can use them to learn how to work with BigQuery or even build your application on top of them, exactly as we’re going to do.
IMDb dataset importer – loads into a Marten DB document store. It imports the public datasets into a database, and provides repositories for querying. The total imported size is about 40 million rows, and 14 gigabytes on disk!
PHOnA: A Public Dataset of Measured Headphone Transfer Functions
A dataset of measured headphone transfer functions (HpTFs), the Princeton Headphone Open Archive (PHOnA), is presented. Extensive studies of HpTFs have been conducted for the past twenty years, each requiring a separate set of measurements, but this data has not yet been publicly shared. PHOnA aggregates HpTFs from different laboratories, including measurements for multiple different headphones, subjects, and repositionings of headphones for each subject. The dataset uses the spatially oriented format for acoustics (SOFA), and SOFA conventions are proposed for efficiently storing HpTFs. PHOnA is intended to provide a foundation for machine learning techniques applied to HpTF equalization. This shared data will allow optimization of equalization algorithms to provide more universal solutions to perceptually transparent headphone reproduction. Access it here.
Provide both basic and sabermetric statistics and resources for sports fans everywhere. Access here
Explore, analyze, and share quality data here
Spreadsheets and Datasets:
John Hopkins University Github confirmed case numbers.
Google Sheets From DXY.cn (Contains some patient information [age,gender,etc] )
Strain Data repo
Other Good sources:
BNO Seems to have latest number w/ sources. (scrape)
DXY.cn Chinese online community for Medical Professionals *translate page.
Early Transmission Dynamics Provides statistics on the early cases, median age, gender etc.
- COVID-19 Mobility Data Aggregator
The NYT provides cases on county level and mask usage by county level on their github. https://github.com/nytimes/covid-19-data
covid19.richdataservices.com – RDS COVID-19 Data Center
covid19.richdataservices.com/rds-explorer – view the available datasets and explore the data
covid19.richdataservices.com/rds-tabengine – tabulate datasets
https://github.com/mtna – Open-source tools
https://documenter.getpostman.com/view/2220438/SzYevv9u – Postman documentation with data visualizations and example API queries for each dataset
- Kaggle COVID Dataset
This is an example 3D model of an antibody neutralizing SARS-CoV-2. Their dataset is essentially simulating this interaction between thousands of different antibodies and antigens.
In order to make all this data more accessible, they have converted everything into about 50 GB of CSV files with rows corresponding to “contact points” between the antibody and SARS-CoV-2 (or another antigen). Here’s a pastebin example of the contacts predicted between Matuzumab and SARS-CoV-2.
If you want to contribute to finding antibody treatments for COVID-19, these simulations can be used in data mining similar to the approach described in this paper.
- The World Mortality Dataset is a new repository that contains country-level data on all-cause mortality in 2015–2021 collected from various sources. It is maintained on a monthly basis and provides data for 79 countries. This is useful for tracking excess mortality across countries during the COVID-19 pandemic.
Natural History Museum in London
The Natural History Museum in London has 80 million items (and counting!) in its collections, from the tiniest specks of stardust to the largest animal that ever lived – the blue whale.
The Digital Collections Programme is a project to digitise these specimens and give the global scientific community access to unrivalled historical, geographic and taxonomic specimen data gathered in the last 250 years. Mobilising this data can facilitate research into some of the most pressing scientific and societal challenges.
Digitising involves creating a digital record of a specimen which can consist of all types of information such as images, and geographical and historical information about where and when a specimen was collected. The possibilities for digitisation are quite literally limitless – as technology evolves, so do possible uses and analyses of the collections. We are currently exploring how machine learning and automation can help us capture information from specimen images and their labels.
With such a wide variety of specimens, digitising looks different for every single collection. How we digitise a fly specimen on a microscope slide is very different to how we might digitise a bat in a spirit jar! We develop new workflows in response to the type of specimens we are dealing with. Sometimes we have to get really creative, and have even published on workflows which have involved using pieces of LEGO to hold specimens in place while we are imaging them.
Mobilising this data and making it open access is at the heart of the project. All of the specimen data is released on our Data Portal, and we also feed the data into international databases such as GBIF.
TSA Throughput Dataset (alternate source)
The TSA has is publishing more and more data via it’s Freedom of Information Act (FOIA) Reading Room. This project on github https://github.com/mikelor/tsathroughput contains the source for extracting the information from the .PDF files and converts them to JSON and CSV files.
The /data folder contains the source .PDFs going back to 2018 while the /data/raw/tsa/throughput folder contains .json files.
ML Dataset to practice methods of regression
- The dataset is gathered on Sep. 17th 2020 from GitHub.
- It has more than 5.2K Python repositories and 4.2M type annotations.
- Use it to train ML-based type inference model for Python
- Access it here
Measurements of the normal (i.e. non-superconducting) state magnetoresistance (change in resistance with magnetic field) in several single crystalline samples of copper-oxide high-temperature superconductors. The measurements were performed predominantly at the High Field Magnet Laboratory (HFML) in Nijmegen, the Netherlands, and the Pulsed Magnetic Field Facility (LNCMI-T) in Toulouse, France. Complete Zip Download
The UMA-SAR Dataset: Multimodal data collection from a ground vehicle during outdoor disaster response training exercises
Collection of multimodal raw data captured from a manned all-terrain vehicle in the course of two realistic outdoor search and rescue (SAR) exercises for actual emergency responders conducted in Málaga (Spain) in 2018 and 2019: the UMA-SAR dataset. Full Dataset.
Child mortality numbers caused by malaria by country
Number of deaths of infants, neonatal, and children up to 4 years old caused by malaria by country from 2000 to 2015. Originator: World Health Organization
Quora Question Pairs at Data.world
The dataset will give anyone the opportunity to train and test models of semantic equivalence, based on actual Quora data. 400,000 lines of potential question duplicate pairs. Each line contains IDs for each question in the pair, the full text for each question, and a binary value that indicates whether the line truly contains a duplicate pair. Access it here.
MIMIC Critical Care Database
MIMIC is an openly available dataset developed by the MIT Lab for Computational Physiology, comprising deidentified health data associated with ~60,000 intensive care unit admissions. It includes demographics, vital signs, laboratory tests, medications, and more. Access it here.
Data.Gov: The home of the U.S. Government’s open data
US Census Bureau: QuickFacts Dataset
QuickFacts provides statistics for all states and counties, and for cities and towns with a population of 5,000 or more.
Art that does not attempt to represent an accurate depiction of a visual reality but instead use shapes, colours, forms and gestural marks to achieve its effect
Interactive map of indigenous people around the world
Native-Land.ca is a website run by the nonprofit organization Native Land Digital. Access it here.
Data Visualization: A Wordcloud for each of the Six Largest Religions and their Religious Texts (Judaism, Christianity, and Islam; Hinduism, Buddhism, and Sikhism)
Over 200+ public datasets, including COVID data. Access it here.
Ohio Data, Ohio Insights. The DataOhio catalog is a single source for the most critical and relevant datasets from state agencies and entities.
Conducted by the Federal Highway Administration (FHWA), the NHTS is the authoritative source on the travel behavior of the American public. It is the only source of national data that allows one to analyze trends in personal and household travel. It includes daily non-commercial travel by all modes, including characteristics of the people traveling, their household, and their vehicles. Access it here.
Statistics and data about the National Travel Survey, based on a household survey to monitor trends in personal travel.
The survey collects information on how, why, when and where people travel as well as factors affecting travel (e.g. car availability and driving license holding).
ENTUR: NeTEx or GTFS datasets [Norway]
NeTEx is the official format for public transport data in Norway and is the most complete in terms of available data. GTFS is a downstream format with only a limited subset of the total data, but we generate datasets for it anyway since GTFS can be easier to use and has a wider distribution among international public transport solutions. GTFS sets come in “extended” and “basic” versions. Access here.
A subset of the field data collected on temporary NFI plots can be downloaded in Excel format from this web site. The file includes a Read_me sheet and a sheet with field data from temporary plots on forest land1 collected from 2007 to 2019. Note that plots located on boundaries (for example boundaries between forest stands, or different land use classes) are not included in the dataset. The dataset is primarily intended to be used as reference data and validation data in remote sensing applications. It cannot be used to derive estimates of totals or mean values for a geographic area of any size. Download the dataset here
Large data sets from finance and economics applicable in related fields studying the human condition
Boards of Governors of the Federal Reserve: Data Download Program
CIA: The world Factbook provides basic intelligence on the history, people, government, economy, energy, geography, environment, communications, transportation, military, terrorism, and transnational issues for 266 world entities.
Consumer Price Index: The Consumer Price Index (CPI) is a measure of the average change over time in the prices paid by urban consumers for a market basket of consumer goods and services. Indexes are available for the U.S. and various geographic areas. Average price data for select utility, automotive fuel, and food items are also available.
Gapminder.org: Unveiling the beauty of statistics for a fact based world view Watch everyday life in hundreds of homes on all income levels across the world, to counteract the media’s skewed selection of images of other places.
Our world in Data: International Trade
Research and data to make progress against the world’s largest problems: 3139 charts across 297 topics, All free: open access and open source.
World Input-Output Tables and underlying data. World Input-Output Tables and underlying data, covering 43 countries, and a model for the rest of the world for the period 2000-2014. Data for 56 sectors are classified according to the International Standard Industrial Classification revision 4 (ISIC Rev. 4).
- Data: Real and PPP-adjusted GDP in US millions of dollars, national accounts (household consumption, investment, government consumption, exports and imports), exchange rates and population figures.
- Geographical coverage: Countries around the world
- Time span: from 1950-2011 (version 8.1)
- Available at: Online
COW seeks to facilitate the collection, dissemination, and use of accurate and reliable quantitative data in international relations. Key principles of the project include a commitment to standard scientific principles of replication, data reliability, documentation, review, and the transparency of data collection procedures
- Data: Total national trade and bilateral trade flows between states. Total imports and exports of each country in current US millions of dollars and bilateral flows in current US millions of dollars
- Geographical coverage: Single countries around the world
- Time span: from 1870-2009
- Available at: Online here
- This data set is hosted by Katherine Barbieri, University of South Carolina, and Omar Keshk, Ohio State University.
Free and open access to global development data. Access it here.
The WTO provides quantitative information in relation to economic and trade policy issues. Its data-bases and publications provide access to data on trade flows, tariffs, non-tariff measures (NTMs) and trade in value added.
- Data: Many series on tariffs and trade flows
- Geographical coverage: Countries around the world
- Time span: Since 1948 for some series
- Available at: Online here
SMOKA Science Archive
The Subaru-Mitaka-Okayama-Kiso Archive, holds about 15 TB of astronomical data from facilities run by the National Astronomical Observatory of Japan. All data becomes publicly available after an embargo period of 12-24 months (to give the original observers time to publish their papers).
- Web crawl graph with 3.5 billion web pages and 128 billion hyperlinks
- Diverse graphs (Stanford) with up to 1.8 billion edges
- Twitter follower graph (Uni Koblence) with 1.4 billion edges
- Divers graph data sets (Yahoo) including bipartite graphs with 2.2 million edges
- Many web and social graphs with up to 95 billion edges. While this data collection seems to be very comprehensive, it is not trivially accessible without external tool.
- Over 3000 social, biological, web graph data sets with small to large scale (dozens to billions of edges).
- Github project with dozens of graph data sets.
- Brain graphs (among other biological networks) with up to tens of millions of edges.
- Heterogeneous graph data from wolve interactions to co-authorships to social network data.
The Multi-Domain Sentiment Dataset contains product reviews taken from Amazon.com from many product types (domains). Some domains (books and dvds) have hundreds of thousands of reviews. Others (musical instruments) have only a few hundred. Reviews contain star ratings (1 to 5 stars) that can be converted into binary labels if needed. Access it here.
Supported by Google Jigsaw, the GDELT Project monitors the world’s broadcast, print, and web news from nearly every corner of every country in over 100 languages and identifies the people, locations, organizations, themes, sources, emotions, counts, quotes, images and events driving our global society every second of every day, creating a free open platform for computing on the entire world.
The Yahoo News Feed: Ratings and Classification Data
Dataset is 1.5 TB compressed, 13.5 TB uncompressed
More than 1 TB
- The 1000 Genomes project makes 260 TB of human genome data available
- The Internet Archive is making an 80 TB web crawl available for research
- The TREC conference made the ClueWeb09  dataset available a few years back. You’ll have to sign an agreement and pay a nontrivial fee (up to $610) to cover the sneakernet data transfer. The data is about 5 TB compressed.
- ClueWeb12 is now available, as are the Freebase annotations, FACC1
- CNetS at Indiana University makes a 2.5 TB click dataset available
- ICWSM made a large corpus of blog posts available for their 2011 conference. You’ll have to register (an actual form, not an online form), but it’s free. It’s about 2.1 TB compressed. The dataset consists of over 386 million blog posts, news articles, classifieds, forum posts and social media content between January 13th and February 14th. It spans events such as the Tunisian revolution and the Egyptian protests (see http://en.wikipedia.org/wiki/January_2011 for a more detailed list of events spanning the dataset’s time period). Access it here
- The Yahoo News Feed dataset is 1.5 TB compressed, 13.5 TB uncompressed
- The Proteome Commons makes several large datasets available. The largest, the Personal Genome Project , is 1.1 TB in size. There are several others over 100 GB in size.
More than 1 GB
- The Reference Energy Disaggregation Data Set has data on home energy use; it’s about 500 GB compressed.
- The Tiny Images dataset has 227 GB of image data and 57 GB of metadata.
- The ImageNet dataset is pretty big.
- The MOBIO dataset is about 135 GB of video and audio data
- The Yahoo! Webscope program makes several 1 GB+ datasets available to academic researchers, including an 83 GB data set of Flickr image features and the dataset used for the 2020 KDD Cup , from Yahoo! Music, which is a bit over 1 GB.
- Freebase makes regular data dumps available. The largest is their Quad dump , which is about 3.6 GB compressed.
- Wikipedia made a dataset containing information about edits available for a recent Kaggle competition . The training dataset is about 2.0 GB uncompressed.
- The Research and Innovative Technology Administration (RITA) has made available a dataset about the on-time performance of domestic flights operated by large carriers. The ASA compressed this dataset and makes it available for download.
- The wiki-links data made available by Google is about 1.75 GB total.
- Google Research released a large 24GB n-gram data set back in 2006 based on processing 10^12 words of text and published counts of all sequences up to 5 words in length.
These data are intended to be used by researchers and other professionals working in power and energy related areas and requiring data for design, development, test, and validation purposes. These data should not be used for commercial purposes.
- Electric Vehicles
- Power Quality
- PV Generation
- Weather Data
- Wind Based Generation
- General Energy Data
- Monthly data on average electricity prices (US)
- Monthly data on average electricity prices (Mexico)
- Monthly data on average electricity prices (Brazil)
- Monthly data on average electricity prices (Europe)
- Monthly data on average electricity prices (Australia)
- Monthly data on average electricity prices (UK)
A dataset and open-ended challenge for music recommendation research ( RecSys Challenge 2018). Sampled from the over 4 billion public playlists on Spotify, this dataset of 1 million playlists consist of over 2 million unique tracks by nearly 300,000 artists, and represents the largest public dataset of music playlists in the world. Access it here
20k+ Hotel Reviews from Yelp for 5 Star Hotels in Las Vegas.
This dataset can be used for the following applications and more:
Analyzing trends, Sentiment Analysis / Opinion Mining, Sentiment Analysis / Opinion Mining, Competitor Analysis. Access it here.
A truncated version with 500 reviews is also available on Kaggle here
1- WorldoMeter: Countries in the world by population (2021)
2- Snowflake Data Marketplace: Snowflake Data Marketplace gives data scientists, business intelligence and analytics professionals, and everyone who desires data-driven decision-making, access to more than 375 live and ready-to-query data sets from more than 125 third-party data providers and data service providers
3- Quandl: The premier source for financial, economic and alternative datasets, serving investment professionals.
The National Health and Nutrition Examination Survey (NHANES) is conducted every two years by the National Center for Health Statistics and funded by the Centers for Disease Control and Prevention. The survey measures obesity rates among people ages 2 and older. Find the latest national data and trends over time, including by age group, sex, and race. Data are available through 2017-2018, with the exception of obesity rates for children by race, which are available through 2015-2016. Access here
Where do the world’s CO2 emissions come from? This map shows emissions during 2019. Darker areas indicate areas with higher emissions
NCEI first developed the Global Historical Climatology Network-Monthly (GHCN-M) temperature dataset in the early 1990s. Subsequent iterations include version 2 in 1997, version 3 in May 2011, and version 4 in October 2018.
The World’s Most Eco-Friendly Countries
Alternate Source from Wikipedia : List of countries by carbon dioxide emissions per capita
% change in life expectancy from 2020 to 2021 across the globe
Data Source Here: Note that these values can change with time based on the discovery of new reserves, and changes in annual production.
The Human Development Index (HDI) is a statistic composite index of life expectancy, education (mean years of schooling completed and expected years of schooling upon entering the education system), and per capita income indicators, which are used to rank countries into four tiers of human development.
Data source – subnational human development index website
Football/Soccer Leagues with the fairest distributions of money have seen the most growth in long-term global interest.
Results from survey on how to best reduce your personal carbon footprint
Data from IpsosMori
The data comes from the Global Power Plant Database. The Global Power Plant Database is a comprehensive, open source database of power plants around the world. It centralizes power plant data to make it easier to navigate, compare and draw insights for one’s own analysis. The database covers approximately 30,000 power plants from 164 countries and includes thermal plants (e.g. coal, gas, oil, nuclear, biomass, waste, geothermal) and renewables (e.g. hydro, wind, solar). Each power plant is geolocated and entries contain information on plant capacity, generation, ownership, and fuel type. It will be continuously updated as data becomes available.
Facebook Monthly Active Users
Facebook data is based on the end of year from 2004 to 2020
Heat map of the past 50,000 earthquakes pulled from USGS sorted by magnitude
Source: USGS website
Where do the world’s methane (CH4)emissions come from?
Darker areas indicate areas with higher emissions.
Source: Data comes from EDGARv5.0 website and Crippa et al. (2019)
Data Source: ECMWF ERA5
Wealth of Forbes’ Top 100 Billionaires vs All Households in Africa
20 years of Apple sales in a minute
Suppose your state is 60% orc, 30% undead, and 10% tauren. You chance in a random selection of two being of the same race is as follows:
36% chance ((60%)2) of two orcs
9% chance ((30%)2) of two undead
1% chance ((10%)2) of two tauren
For a total of 46%. The diversity index would be 100% minus that, or 54%.
The CIFAR-10 dataset consists of 60000 32×32 colour images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images.
The ImageNet dataset contains 14,197,122 annotated images according to the WordNet hierarchy. Since 2010 the dataset is used in the ImageNet Large Scale Visual Recognition Challenge (ILSVRC), a benchmark in image classification and object detection. The publicly released dataset contains a set of manually annotated training images.
Machine Learning: The MNIST Database of Handwritten Digits
The MNIST database of handwritten digits, available from this page, has a training set of 60,000 examples, and a test set of 10,000 examples. It is a subset of a larger set available from NIST. The digits have been size-normalized and centered in a fixed-size image.
It is a good database for people who want to try learning techniques and pattern recognition methods on real-world data while spending minimal efforts on preprocessing and formatting. Access it here.
MMID is a large-scale, massively multilingual dataset of images paired with the words they represent collected at the University of Pennsylvania. The dataset is doubly parallel: for each language, words are stored parallel to images that represent the word, and parallel to the word’s translation into English (and corresponding images.) . Dcumentation.
AWS CLI Access (No AWS account required)
aws s3 ls s3://mmid-pds/ --no-sign-request
Data Source: Downloaded performance data on these cryptocurrencies from Investing.com which provides free historic data
Source: US Federal Reserve
Data Source from https://coinmarketcap.com/
HDI is calculated by the UN every year to measure a country’s development using average life expectancy, education level, and gross national income per capita (PPP). The EU has a collective HDI of 0.911.
Data Source: Here
Data source: Human Development Report 2020
Child marriage by country, by gender
Data on the percentage of children married before reaching adulthood (18 years).
Data source The State of the World’s Children 2019
Data Source : Wikipedia
Population Projection for China and India till 2050
Data Source: Here
Dat Source: https://ourworldindata.org
Data Source : https://www.f1-fansite.com/f1-results/
Countries with the most nuclear warheads. A couple of days ago I posted this with a logarithmic scale.
Data source: Wikipedia
Data collected from a series of rushing and passing statistics for NFL Quarterbacks from 2015-2020 and performed a machine learning algorithm called clustering, which automatically sorts observations into groups based on shared common characteristics using a mathematical “distance metric.”
The idea was to use machine learning to determine NFL Quarterback Archetype to agnostically determine which quarterbacks were truly “mobile” quarterbacks, and which were “pocket passers” that relied more on passing. I used a number of metrics in my actual clustering analysis, but they can be effectively summarized across two dimensions: passing and rushing, which can be further roughly summarized across two metrics: passer rating and rushing yards per year. Plotting the quarterbacks along these dimensions and plotting the groups chosen by the clustering methodology shows how cleanly the methodology selected the groups.
Data: Collected from the ESPN API
Intraday Stock Data (1 min) – S&P 500 – 2008-21: 12 years of 1 minute bars for data science / machine learning.
Granular stock bar data for research is difficult to find and expensive to buy. The author has compiled this library from a variety of sources and is making it available for free.
One compressed CSV file with 9 columns and 2.07 million rows worth of 1 minute SPY bars. Access it here
Industrial ML Datasets: curated list of datasets, publicly available for machine learning researches in the area of manufacturing.
|Name||Year||Feature Type||Feature Count||Target Variable||Instances||Official Train/Test Split||Data Source||Format|
|Diesel Engine Faults Features||2020||Signal||84||C (4)||3.500||Synthetic||MAT||Link|
|Name||Year||Feature Type||Feature Count||Target Variable||Instances||Official Train/Test Split||Data Source||Format|
|High Storage System Anomaly Detection||2018||Signal||20||C (2)||91.000||Synthetic||CSV||Link|
|Name||Year||Feature Type||Feature Count||Target Variable||Instances||Official Train/Test Split||Data Source||Format|
|Casting Product Quality Inspection||2020||Image||300×300
|Name||Year||Feature Type||Feature Count||Instances||Official Train/Test Split||Data Source||Format|
Here is a collection of datasets with images of leaves
and more generic image datasets that include plant leaves
English alphabet organized by each letter’s note in ABC
Discover datasets hosted in thousands of repositories across the Web using datasetsearch.research.google.com
Create, maintain, and contribute to a long-living dataset that will update itself automatically across projects.
Datasets should behave like git repositories.
Learn how to create, maintain, and contribute to a long-living dataset that will update itself automatically across projects, using git and DVC as versioning systems, and DAGsHub as a host for the datasets.
Data source: https://ourworldindata.org/energy
Courtesy of Google’s Project Sunroof: This dataset essentially describes the rooftop solar potential for different regions, based on Google’s analysis of Google Maps data to find rooftops where solar would work, and aggregate those into region-wide statistics.
It comes in a couple of aggregation flavors – by census tract , where the region name is the census tract id, and by postal code , where the name is the postal code. Each also contains latitude/longitude bounding boxes and averages, so that you can download based on that, and you should be able to do custom larger aggregations using those, if you’d like.
Carbon emission arithmetic + hard v. soft science
A large dataset aimed at teaching AI to code, it consists of some 14M code samples and about 500M lines of code in more than 55 different programming languages, from modern ones like C++, Java, Python, and Go to legacy languages like COBOL, Pascal, and FORTRAN.
Download instructions are here
Data source: https://databank.worldbank.org/home.aspx
Data source: w3school
Data Source: DataBayou
Data Source from The Guardian:
Some context here
Data scraped from StatsCan
Data source: https://ourworldindata.org/covid-vaccinations
What does per 100 mean?
When the whole country is double vaccinated, the value will be 200 doses per 100 population. At the moment the UK is like 85, which is because ~70% of the population has had at least one dose and ~15% of the population (which is a subset of that 70%) have had two. Hence ~30% are currently unprotected – myself included until Sunday.
DataSource: SIPRI Arms Transfer Database
Aggregated disease comparison dataset – Ensemble de données agrégées de comparaison des maladies
According to the author of the source data: “For the 1918 Spanish Flu, the data was collected by knowing that the total counts were 500M cases and 50M deaths, and then taking a fraction of that per day based on the area of this graph image:” – the graph is used is here:
Visualización y conjunto de datos de comparación de enfermedades agregadas
Data source: https://trends.google.com Trending topics from 2010 to 2019 were taken from Google’s annual Year in Search summary 2010-2029
The full, ~11 minute video covering the whole 2010s decade is available here at https://youtu.be/xm91jBeN4oo
Google Trends provides weekly relative search interest for every search term, along with the interest by state. Using these two datasets for each term, we’re able to calculate the relative search interest for every state for a particular week. Linear interpolation was used to calculate the daily search interest.
Data source: CoinMarket from end of 2013 until present
Capitalisation boursière en milliards de dollars des 20 principales crypto-monnaies en 2021-05-20
Data source: https://ratings.fide.com/
The y-axis is the world elo ratings (called FIDE ratings).
Data sources: Here
Source article: Here
Data is from the Global Power Plant Database (World Resources Institute)
Source: r/all on Reddit
Tool used: https://www.meta-chart.com
Las rutas más rápidas en tierra (y, a veces, en barco) entre los 990 pares de capitales europeas
Les itinéraires les plus rapides sur terre (et parfois en bateau) entre les 990 paires de capitales européennes
From the author: I started with data on roads from naturalearth.com, which also includes some ferry lines. I then calculated the fastest routes (assuming a speed of 90 km/h on roads, and 35 km/h on boat) between each pair of 45 European capitals. The animation visualizes these routes, with brighter lines for roads that are more frequently “traveled”.
In reality these are of course not the most traveled roads, since people don’t go from all capitals to all other capitals in equal measure. But I thought it would be fun to visualize all the possible connections.
The model is also very simple, and does not take into account varying speed limits, road conditions, congestion, border checks and so on. It is just for fun!
In order to keep the file size manageable, the animation only shows every tenth frame.
Is Russia, Turkey or country X really part of Europe? That of course depends on the definition, but it was more fun to include them than to exclude them! The Vatican is however not included since it would just be the same as the Rome routes. And, unfortunately, Nicosia on Cyprus is not included to due an error on my behalf. It should be!
- Dataset of all 825 Pokemon (this includes Alolan Forms). It would be preferable if there are at least 100 images of each individual Pokemon.
2) This dataset comprises of more than 800 pokemons belonging up to 8 generations.
Using this dataset have been fun for me. I used it to create a mosaic of pokemons taking image as reference. You can find it here and it’s free to use: Couple Mosaic (powered by Pokemons)
Here is the data type information in the file:
- Name: Pokemon Name
- Type: Type of Pokemon like Grass / Fire / Water etc..,.
- HP: Hit Points
- Attack: Attack Points
- Defense: Defence Points
- Sp. Atk: Special Attack Points
- Sp. Def: Special Defence Points
- Speed: Speed Points
- Total: Total Points
- url: Pokemon web-page
- icon: Pokemon Image
Data File: Pokemon-Data.csv
ETL pipeline for Facebook’s research project to provide detailed large-scale demographics data. It’s broken down in roughly 30×30 m grid cells and provides info on groups by age and gender.
Rasterized GDP dataset – basically a heat map of global economic activity.
Gap-filled multiannual datasets in gridded form for Gross Domestic Product (GDP) and Human Development Index (HDI)
Countries of the world sorted by those that have warmed the most in the last 10 years, showing temperatures from 1890 to 2020
Data source: Gistemp temperature data
The GISS Surface Temperature Analysis ver. 4 (GISTEMP v4) is an estimate of global surface temperature change. Graphs and tables are updated around the middle of every month using current data files from NOAA GHCN v4 (meteorological stations) and ERSST v5 (ocean areas), combined as described in our publications Hansen et al. (2010) and Lenssen et al. (2019).
Data Source: Competitive Enterprise Institute (PDF)
Buying a chocolate bar? There are seemingly hundreds to choose from, but its just the illusion of choice. They pretty much all come from Mars, Nestlé, or Mondelēz (which owns Cadbury).
Source: Visual Capitalist
Some context for these numbers :
- PS4 holds the record for being the console to have sold the most games in video game history (> 1.622B units)
- Previous record holder was PS2 at 1.537B games sold
- PS4 holds the record for having sold the most games in a single year (> 300M units in FY20)
- FY20 marks the biggest yearly software sales in PlayStation ecosystem with more than 338M units
- Since PS5 release, Sony starts combining PS4/PS5 software sales
- In FY12, Sony combined PS2/PS3 and PSP/VITA software sales
- Sony stopped disclosing software sales in FY13/14
- Source: Sony’s financial results
Sony combined PS2/PS3 hardware sales in FY12 and combined PSP/VITA sales in FY12/13/14
- Source: Sony’s financial results
The City with 32 million is Chongqing, Shan is Shanghai, Beijin is Beijing, and Guangzho is Guangzhou
Data source: Builds on data from the 2021 European Quality of Government Index. You can read more about the survey and download the data here
Dataset: Visual Capitalist
Dataset (according to author): Dictionaries are scattered on the internet and had to be borrowed from several sources: the Scrabble3d project, and Linux spellcheck dictionaries. The data can be found in the folder “Avec_diacritiques”.
Criteria for choosing a dictionary:
– No proper nouns
– “Official” source if available
– Inclusion of inflected forms
– Among two lists, the largest was fancied
– No or very rare abbreviations if possible- but hard to detect in unknown languages and across hundreds of thousands of words.
Dataset: Visual Capitalist
The author found this dataset in a more accessible format upon searching for the keyword “CDPB” (Carcinogenic Potency Database) in the National Library of Medicine Catalog. Check out this parent website for the data source and dataset description. The dataset referenced in OP’s post concerns liver specific carcinogens, which are marked by the “liv” keyword as described in the dataset description’s Tissue Codes section.
The SMS Spam Collection is a public set of SMS labeled messages that have been collected for mobile phone spam research
- Looking for Dataset on the outcomes of abstinence-only sex education.
- Looking for Data set of horse race results / lottery results any results related to gambling [1, 2, 3]
- Looking for Football (Soccer) Penalties Dataset [1, 2]
- Looking for public datasets on baseball [1, 2, 3]
- Looking for Datasets on edge computing for AI bandwidth usage, latency, memory, CPU/GPU resource usage? [1 ,2 ]
- Dataset of employee attrition or turnover rate? [1, 2]
- Is there a Dataset for homophobic tweets? [1 ,2, 3, 4, ]
- Looking for a Machine condition Monitoring Dataset [1,2]
- Where to find data for credit risk analysis? [1, 2]
- Datasets on homicides anywhere in the world [1, 2]
- Looking for a dataset containing coronavirus self-test (if this is a thing globally) pictures for ML use
- Looking for Beam alignment 5G vehicular networks dataset
- Looking for tidy dataset for multivariate analysis (PCA, FA, canonical correlations, clustering)
- Indian all types of Fuel location datasets
- Curated social network datasets with summary statistics and background info
- Looking for textile crop disease datasets such as jute, flax, hemp
- Shopify App Store and Chrome Webstore Datasets
- Looking for dataset for university chatbot
- Collecting real life (dirty/ugly) datasets for data analysis
- In Need of Food Additive/Ingredient Definition Database
- Recent smart phone sensor Dataset – Android
- Cracked Mobile Screen Image Dataset for Detection
- Looking for Chiller fault data in a chiller plant
- Looking for dataset that contains the genetic sequences of native plasmids?
- Looking for a dataset containing fetus size measurements at various gestational ages.
- Looking for datasets about mental health since 2021
- Do you know where to find a dataset with Graphical User Interfaces defects of web applications? [1, 2, 3 ]
- Looking for most popular accounts on social medias like Twitter, Tik Tok, instagram, [1, 2, 3]
- GPS dataset of grocery stores
- What is the easiest way to bulk download all of the data from this epidemiology website? (~20,000 files)
- Looking for Dataset on Percentage of death by US state and Canadian province grouped by cause of death?
- Looking for Social engineering attack dataset in social media
- Steam Store Games (Clean dataset) by Nik Davis
- Dataset that lists all US major hospitals by county
- Another Data that list all US major hospitals by county
- Looking for open source data relating privacy behavior or related marketing sets about the trustworthiness of responders?
- Looking for a dataset that tracks median household income by country and year
- Dataset on the number of specific surgical procedures performed in the US (yearly)
- Looking for a dataset from reddit or twitter on top posts or tweets related to crypto currency
- Looking for Image and flora Dataset of All Known Plants, Trees and Shrubs
- US total fertility rates data one the state level
- Dataset of Net Worth of *World* Politicians
- Looking for water wells and borehole datasets
- Looking for Crop growth conditions dataset
- Dataset for translate machine JA-EG
- Looking for Electronic Health Record (EHR) record prices
- Looking for tax data for different countries
- Musicians Birthday Datasets and Associated groups
- Searching for dataset related to car dealerships 
- Looking for Credit Score Approval dataset
- Cyberbullying Dataset by demographics
- Datasets on financial trends for minors
- Data where I can find out about reading habits? [1, 2]
- Data sets for global technology adoption rates
- Looking for any and all cat / feline cancer datasets, for both detection and treatment
- ITSM dictionary/taxonomy datasets for topic modeling purposes
- Multistage Reliability Dataset
- Looking for dataset of ingredients for food
- Looking for datasets with responses to psychological questionnaires[1,2,3]
- Data source for OEM automotive parts
- Looking for dataset about gene regulation
- Customer Segmentation Datasets (For LTV Models)
- Automobile dataset, years of ownership and repairs
- Historic Housing Prices Dataset for Individual Houses
- Looking for the data for all the tokens on the Uniswap graph
- Job applications emails datasets, either rejection, applications or interviews
- E-learning datasets for impact on e learning on school/university students
- Food delivery dataset (Uber Eats, Just Eat, …)
- Data Sets for NFL Quarterbacks since 1995
- Medicare Beneficiary Population Data
- Covid 19 infected Cancer Patients datasets
- Looking for EV charging behavior dataset
- State park budget or expansionary spending dataset
- Autonomous car driving deaths dataset
- FMCG Spending habits over the pandemic
- Looking for a Question Type Classification dataset
- 20 years of Manufacturer/Retail price of Men’s footwear
- Dataset of Global Technology Adoption Rates
- Looking For Real Meeting Transcripts Dataset
- Dataset For A Large Archive Of Lyrics [1,2,3]
- Audio dataset with swearing words
- A global, georeferenced event dataset on electoral violence with lethal outcomes from 1989 to 2017. [1,]
- Looking for Jaundice Dataset for ML model
- Looking for social engineering attack detection dataset?
- Wound image datasets to train ML model 
- Seeking for resume and job post dataset
- Labelled dataset (sets of images or videos) of human emotions [1,2]
- Dataset of specialized phone call transcripts
- Looking for Emergency Response Plan Dataset for family Homes, condo buildings and Companies
- Looking for Birthday wishes datasets
- Desperately in need of national data for real estate [1,2,]
- NFL playoffs games stadium attendance dataset
- Datasets with original publication dates of novels [1,2]
- Annotated Documents with Images Data Dump
- Looking for dataset for “Face Presentation Attack Detection”
- Electric vehicle range & performance dataset [1, 2]
- Dataset or API with valid postal codes for US, Mexico, and Canada with country, state/province, and city/town [1, 2, 3, 4, 5, 6]
- Looking for Data sources regarding Online courses dropout rate, preferably by countries [1,2 ]
- Are there dataset for language learning [1, 2]
- Corporate Real Estate Data [1,2, 3]
- Looking for simple clinical trials datasets [1, 2]
- CO2 Emissions By Aircraft (or Aircraft Type) – Climate Analysis Dataset [1,2, 3, 4]
- Player Session/playtime dataset from games [1,2]
- Data sets that support Data Science (Technology, AI etc) being beneficial to sustainability [1,2]
- Datasets of a grocery store [1,2]
- Looking for mri breast cancer annotation datasets [1,2]
- Looking for free exportable data sets of companies by industry [1,2]
- Datasets on Coffee Production/Consumption [1,2]
- Video gaming industry datasets – release year, genre, games, titles, global data [1,2]
- Looking for mobile speaker recognition dataset [1,2]
- Public DMV vehicle registration data [1,2]
- Looking for historical news articles based on industry sector [1,2]
- Looking for Historical state wide Divorce dataset [1,2]
- Public Big Datasets – with In-Database Analytics [1,2]
- Dataset for detecting Apple products (object detection) [1,2]
- Help needed to get the American Hospital Association (AHA) datasets (AHA Annual Survey, AHA Financial Database, and AHA IT Survey datasets) [1, 2]
- Looking for help Getting College Football Betting Data [1,2]
- 2012-2020 US presidential election results by state/city dataset [1,2, 3]
- Looking for datasets of models and images captured using iphone’s LIDAR? [1,2]
- Finding Datasets to Age Texts (Newspapers, Books, Anything works) [1, 2, 3]
- Looking for cost of living index of some type for US [1,2]
- Looking for dataset that recorded historical NFT prices and their price increases, as well as timestamps. [1,2]
- Looking for datasets on park boundaries across the country [1, 2, 3]
- Looking for medical multimodal datasets [1, 2, 3]
- Looking for Scraped Parler Data [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
- Looking for Silicon Wafer Demand dataset [1, 2]
- Looking for a dataset with the values [Gender – Weight – Height – Health] [1, 2]
- Exam questions (mcqs and short answer) datasets? [1, 2]
- Canada Botanical Plants API/Database [1, 2, 3]
- Looking for a geospatial dataset of birds Migration path [1, 2, 3]
- WhatsApp messages dataset/archives [1, 2]
- Dataset of GOOD probiotic microorganisms in the HUMAN gut [1, 2]
- Twitter competition to reduce bias in its image cropping [1,2]
- Dataset: US overseas military deployments, 1950–2020 [1,2]
- Dataset on human clicking on desktop [1,2]
- Covid-19 Cough Audio Classification Dataset [1, 2]
- 12,000+ known superconductors database [1, 2, 3]
- Looking for good dataset related to cyber security for prediction [1, 2]
- Where can I find face datasets to classify whether it is a real person or a picture of that person. For authentication purposes? [1,2]
- DataSet of Tokyo 2020 (2021) Olympics ( details about the Athletes, the countries they representing, details about events, coaches, genders participating in each event, etc.) [1, 2]
- What is your workflow for budget compute on datasets larger than 100GB? [1, 2, 3]
- Looking for chocolate consumer demographic data [1,2, 3]
- Looking for thorough dataset of housing price/tax history [1, 2, 3]
- Wallstreetbets data scraping from 01/01/2020 to 01/06/2021 [1, 2]
- Retinal Disease Classification Dataset [1, 2]
- 400,000 years of CO2 and global temperature data [1, 2, 3]
- Looking for datasets on neurodegenerative diseases [1, 2, 3]
- Dataset for Job Interviews (either Phone, Online, or Physical) [1,2 ,3]
- Firm Cyber Breach Dataset with Firm Identifiers [1, 2, 3]
- Wondering how Stock market and Crypto website get the Data from [1, 2, 3, 4, 5]
- Looking for a dataset with US tourist injuries, attacks, and/or fatalities when traveling abroad [1, 2, 3]
- Looking for Wildfires Database for all countries by year and month? The quantity of wildfires happening, the acreage, things like that, etc.. [1, 2, 3, ]
- Looking for a pill vs fake pill image dataset [1, 2, 3, 4, 5, 6, 7]
Dataset scraped from AutoScout24 with information about new and used cars.
The data was obtained from the UK government website here , so unfortunately there are some things I’m unaware of regarding data and methodology.
All the passes: A visualization of ~1 million passes from 890 matches played in major football/soccer leagues/cups
- Champion League 1999
- FA Women’s Super League 2018
- FIFA World Cup 2018, La Liga 2004 – 2020
- NWSL 2018
- Premier League 2003 – 2004
- Women’s World Cup 2019
Data Source: StatsBomb
In this project, the authors have designed a spatial model which is able to classify urbanity levels globally and with high granularity. As the target geographic support for our model we selected the quadkey grid in level 15, which has cells of approximately 1x1km at the equator.
The author obtained the data from the UK Government website, so unfortunately don’t know the methodology or how they collected the data etc.
The comparison to the general public is a great idea – according to the Government site, 6% of children, 16% of working-age adults and 45% of Pension-age adults are disabled.
Data source: https://www.sports-reference.com/
According to the author (https://www.reddit.com/user/newpua_bie/) , this animation depicts adult cognitive skills, as measured by the PIAAC study by OECD. Here, the numeracy and literacy skills have been combined into one. Each frame of the animation shows the xth percentile skill level of each individual country. Thus, you can see which countries have the highest and lowest scores among their bottom performers, median performers, and top performers. So for example, you can see that when the bottom 1st percentile of each country is ranked, Japan is at the top, Russia is second, etc. Looking at the 50th percentile (median) of each country, Japan is top, then Finland, etc.
Programme for the International Assessment of Adult Competencies (PIAAC) is a study by OECD to measure measured literacy, numeracy, and “problem-solving in technology-rich environments” skills for people ages 16 and up. For those of you who are familiar with the school-age children PISA study, this is essentially an adult version of it.
Dataset: Tax Foundation
Data Source: UEFA qualifying match data
The model was built in Stan and was inspired by Andrew Gelman’s World Cup model shown here. These plots show posterior probabilities that the team on the y axis will score more goals than the team on the x axis. There is some redundancy of information here (because if I know P(England beats Scotland) then I know P(Scotland beats England) )
Data source: Here (go to the “Babies per woman,” “Income,” and “Population” links on that page).
Data Source: Here
Data source: Here
DataSet: Gathered from https://www.worldpop.org/project/
The greater the length of each spike correlates to greater population density.
The portion of a country’s population that is fully vaccinated for COVID (as of June 2021) scales with GDP per capita.
4- Chemistry datasets
SEDE (Stack Exchange Data Explorer) is a dataset comprised of 12,023 complex and diverse SQL queries and their natural language titles and descriptions, written by real users of the Stack Exchange Data Explorer out of a natural interaction. These pairs contain a variety of real-world challenges which were rarely reflected so far in any other semantic parsing dataset. Access it here
Each map size is proportional to population, so China takes up about 18-19% of the map space.
Countries with very far-flung territories, such as France (or the USA) will have their maps shrunk to fit all territories. So it is the size of the map rectangle that is proportional to population, not the colored area. Made in R, using data from naturalearthdata.com. Maps drawn with the tmap package, and placed in the image with the gridExtra package. Map colors from the wesanderson package.
Data source: The Economist
What businesses in different countries search for when they look for a marketing agency – “creative” or “SEO”?
Data source: Google Trends
More maps, charts and written analysis on this topic here
Data source: Eurostat
- Beneath adds some useful features for shared data, like the ability to run SQL queries, sync changes in real-time, a Python integration, and monitoring. The monitoring is really useful as it lets you check out the write activity of the scraper (no surprise, WSB is most active when markets are open
- The scraper (which uses Async PRAW) is open source here
Data Source: SILAM
Data source: Marketplace Apps
Data source: Marketplace Apps
Recorded CDC deaths (2014 – June 16, 2021) from Symptoms, signs and abnormal clinical and laboratory findings, not elsewhere classified (R00-R99)
The chart shows the average daily gain in $ if $100 were invested at a date on x-axis. Total gain was divided by the number of days between the day of investing and June 13, 2021. Gains were calculated on average 30-day prices.
Time range: from March 28, 2013, till June 13, 2021
Data source: Here
Google Playstore dataset is now available with double the data (2.3 Million) android application data and a new attribute stating the scraped date time in Kaggle.
Dataset: Get it here or here
We have 3000 tribes or more in Africa and in that 3000 we have sub tribes.
Daily Temperature of Major Cities Dataset
Daily average temperature values recorded in major cities of the world.
The dataset is available as separate txt files for each city here. The data is available for research and non-commercial purposes only
According to the author: Looking at non-suicide firearms deaths by state (2019), and then grouping by the Guns to Carry rating (1-5 stars), it seems that stricter gun laws are correlated with fewer firearms homicides. Guns to Carry rates states based on “Gun friendliness” with 1 star being least friendly (California, for example), and 5 stars being most friendly (Wyoming, for example). The ratings aren’t perfect but they include considerations like: Permit required, Registration, Open carry, and Background checks to come up with a rating.
The numbers at the bottom are the average non-suicide deaths calculated within the rating group. Each bar shows the number for the individual state.
Interesting that DC is through the roof despite having strict laws. On the flip side, Maine is very friendly towards gun owners and has a very low homicide rate, despite having the highest ratio of suicides to homicides.
Obviously, lots of things to consider and this is merely a correlation at a basic level. This is a topic that interested me so I figured I’d share my findings. Not attempting to make a policy statement or anything.
Relative frequency of words in economics textbooks vs their frequency in mainstream English (the Google Books corpus)
Data for word frequency in econ textbooks was compiled by myself by scraping words from 43 undergraduate economics textbooks. For details see Deconstructing Econospeak.
Data Source: from eMarketer, as quoted byJon Erlichman
Purpose according to the author: raw textual numbers (like in the original tweet) are hard to compare, particularly the acceleration or deceleration of a trend. Did for myself, but maybe is useful to somebody.
More according to the author:
Measurements and calculations of NG and Electricity used to heat four cups of distilled water by Coffee Medley (6/14/2021)
Average coffee bag and pod weight by Coffee Medley (6/14/2021)
Data source: NPR
New Harvard Data (Accidentally) Reveal How Lockdowns Crushed the Working Class While Leaving Elites Unscathed
Data source: Harvard
Data source: PEW
Data source: SSA Actuarial Data,
Processing: Yearly probability of death is converted to the daily probability and expressed in micromorts. Plotted versus age in years.
According to the author,
A few things to notice: It’s dangerous to be a newborn. The same mortality rates are reached again only in the fifties. However, mortality drops after birth very quickly and the safest age is about ten years old. After experiencing mortality jump in puberty – especially high for boys, mortality increases mostly exponentially with age. Every thirty years of life increase chances of dying about ten times. At 80, chance of dying in a year is about 5.8% for males and 4.3% for females. This mortality difference holds for all ages. The largest disparity is at about twenty three years old when males die at a rate about 2.7 times higher than females.
This data is from before COVID.
Data Source: Copernicus Atmosphere Monitoring Service (CAMS)
IPO Returns 2000-2020
IPO Returns 2000-2020
IPO Returns 2000-2020
For every artist that appears in the Top 200, I add up their total streams (for all songs) and the total number of songs in the dataset.
For a commentary on the data, see The Half Life of a Spotify Hit.
Data Source: Wikipedia
Data Source: Federation of American Scientists – https://fas.org/issues/nuclear-weapons/status-world-nuclear-forces/
Tools: Excel, Datawrapper, https://coolors.co/
Check out the FAS site for notes and caveats about their estimates. Governments don’t just print this stuff on their websites. These are evidence-based estimates of tightly-guarded national secrets.
Of particular note – Here’s what the FAS says about North Korea: “After six nuclear tests, including two of 10-20 kilotons and one of more than 150 kilotons, we estimate that North Korea might have produced sufficient fissile material for roughly 40-50 warheads. The number of assembled warheads is unknown, but lower. While we estimate North Korea might have a small number of assembled warheads for medium-range missiles, we have not yet seen evidence that it has developed a functioning warhead that can be delivered at ICBM range.”
Data Source: Wikipedia
Data Source: The wikipediatrend package in R
Glacial Inter-glacial cycles over the past 450000 years
Source and links
The author used several sources for this video and article. The first, for the video, is GitHub Archive & CodersRank. For the analysis of the OSCI index data, the author used https://opensourceindex.io/.
Data source: Here
2021 is straight projections, must be taken with a grain of salt. However, the assumption of continuous rise of murder rate is not a bad one based on recent news reports, such as: here
In a property crime, a victim’s property is stolen or destroyed, without the use or threat of force against the victim. Property crimes include burglary and theft as well as vandalism and arson.
This image was generated for my research mapping the privacy research field. The visual is a combination of network visualisation and manual adding of the labels.
The data was gathered from Scopus, a high-quality academic publication database, and the visualisation was created with Gephi. The initial dataset held ~120k publications and over 3 million references, from which we selected only the papers and references in the field.
The labels were assigned by manually identifying clusters and two independent raters assigning names from a random sample of publications, with a 94% match between raters.
The scripts used are available on Github:
The full paper can be found on the author’s website:
Data source: IMF
Phone Call Anxiety dataset for Millennials and Gen Z
This is a randomized experiment the author conducted with 450 people on Amazon MTurk. Each person was randomly assigned to one of three writing activities in which they either (a) described their phone, (b) described what they’d do if they received a call from someone they know, or (c) describe what they’d do if they received a call from an unknown number. Pictures of an iPhone with a corresponding call screen were displayed above the text box (blank, “Incoming Call,” or “Unknown”). Participants then rated their anxiety on a 1-4 scale.
Data source: Reports – Hate Crimes – NYPD – NYC.gov
Data source – Click here
A meta-database of links to known face image databases. If you need faces images to train/test your machine learning algorithms, or stimuli for research on faces, you will probably find this useful.
A pre-print describing this, alongside a related resource (the Chatlab Facial Anomaly Database), is available here: https://psyarxiv.com/54utr/
u/specific is looking for dataset of books by the year in which they were written (or first published, even if it was in non-printed form).
Trash/Garbage dataset which can be used for waste detection and sustainibility based projects.
Dataset: On Kaggle
If you like it, you can give upvotes into our kaggle platform. The authors are trying to make custom dataset and open-sourcing on Kaggle to make AI models more robust.
SSA data is about people – their wages, their identifying information, their employers, their addresses, and much more.
The RSDD (Reddit Self-reported Depression Diagnosis) dataset consists of Reddit posts for approximately 9,000 users who have claimed to have been diagnosed with depression (“diagnosed users”) and approximately 107,000 matched control users. All posts made to mental health-related subreddits or containing keywords related to depression were removed from the diagnosed users’ data; control users’ data do not contain such posts due to the selection process. Access it here
Anything along the lines of
And body measures (height weight) here
Then you have to join them by their id num, SEQN.
According to the author jcceagle:
We consume China’s products in the G7, so we are partly responsible. China is the workshop of the world and we have outsourced our carbon emissions to them. If only I had per capita consumption data – from the factory to the consumer – this picture would look really different. This is probably what I will try to create for my next post.
Where do Turkeys come from:
According to the Author: Udzu
A few etymological notes
The word turkey originally referred to guinea fowl, an African bird imported from Madagascar via Turkey (and later called guinea fowl when it was brought by Portuguese traders from West Africa). It later started referring to the North American bird, either because it was viewed as a species of guinea fowl, or because it too was brought by way of the Ottoman Empire.
The French dinde (a contraction of poulet d’inde) and its various derivatives is based on the misconception that the New World was Eastern Asia. The Greek γαλοπούλα looks like it derives from Γαλλία “France” + πουλί “bird”, but actually the prefix is a contraction of the Venetian galo d’India “Indian cock”.
The Dutch (and Scandinavian) kalkoen refers to Calicut, which is modern day Kozhikode in Kerala. No idea why.
10. Reddit Source 10
Data Source: ESRI
It contains transcript data for 5,850 complete conversations. It is a paid for dataset however many universities that have a membership already can get it for free.
We are excited to present SPGISpeech (rhymes with “squeegee-speech”), a large-scale transcription dataset, freely available for academic research. SPGISpeech is a corpus of 5,000 hours of professionally-transcribed financial audio. In contrast to previous transcription datasets, SPGISpeech contains a broad cross-section of L1 and L2 English accents, strongly varying audio quality, and both spontaneous and narrated speech. The transcripts have each been cross-checked by multiple professional editors for high accuracy and are fully formatted, including capitalization, punctuation, and denormalization of non-standard words. You can read more about SPGISpeech here.
From the author: u/fajim123
There are tools out there like snapcrawl. which takes snapshots of websites so I have clean UI images but the problem is generating distorted UI images. I have to manually distort them in a photo editor and it’s taking a lot of time to generate the distorted images. I’m looking for an already existing repository of clean and distorted UI images or even a tool which will automatically distort the UI.
Android App Permission dataset is now populated with 2.2 million data items.
88.5% of US children whose family’s income is $100,000 and over are living with two parents married to each other
— relationship.data (@relationshipda4) June 28, 2021
Data source: Census.gov
Sulphur Dioxide is a byproduct of burning fossil fuels, primarily by power plants and industrial facilities.
In 2017, in which nation, on average, do you work the most per year? The nation where people work the most is Cambodia. In fact, the average hours worked is 2455 hours per year. Next, with about 18 hours difference on average, is Myannmar. These two nations are the only ones in the world that exceed 2400 hours per year. In third place, on the other hand, is Mexico, with 2255, followed by a series of countries, including Malaysia and South Africa, which have a value of 2200 hours and 2250.
Tamil and malayalam have some similarities, but are quite different. Malayalam diverged from old Tamil over a thousand years ago
Then there’s Telugu and Kannada which diverged relatively recently from each other. Telugu is sometimes called “sweet Kannada”. There’s a really cool historical chart of how the scripts diverged in the hampi museum. (Managed to find it: https://www.alamy.com/board-displaying-development-of-indian-scripts-at-archaeological-museum-hampi-kamalapura-karnataka-india-image242928367.html)
Besides those 4 big ones, you have other minor languages typically sharing one of the scripts from the big 4 or devanagari. Tulu for example.
Then there’s Konkani spoken on the west coast which is Sanskrit/Marathi descended.
So I’d say Telugu and Kannada are enough similar to be compared to romance languages. But they’re quite different from Tamil which is quite different from malayalam. And then those 4 are completely different from Hindi and other Sanskrit derivatives, and I know most Hindi/urdu speakers can somewhat pass in Punjabi, but I don’t think the same is true for Bengali, or Marathi
The author used data from the article ” Production, use, and fate of all plastics ever made” by Geyer et al. 2017, to point out the significance of plastic waste generation in the packaging industry.
Packaging Generates Almost 50% of Plastic Waste
The only way we can make a meaningful change with our plastic pollution is if large corporations find an alternative to single use plastic packaging. Plastic is an incredibly useful material and has proved to be cheaper and greener to produce. But even recycling some plastics cause more waste and unfavorable byproducts than the original production and most plastics simply just don’t break down in a landfill. Vicious loop indeed. Production of plastic is good, but using it is bad. Relatively speaking of course, ideally not using plastics at all would be the solution. But how would we package stuff, really? That’s one hell of a problem to solve.
In this dataset you can find real and nominal silver & gold prices since 1791 to 2020. The explanation of the differences between real and nominal prices are:
· Nominal values are the current monetary values.
· Real values are adjusted for inflation and show prices/wages at constant prices.
· Real values give a better guide to what you can actually buy and the opportunity costs you face.
Example of real vs nominal:
· If you receive an 8% increase in your wages from £100 to £108, this is the nominal increase.
· However, if inflation is 2%, then the real increase in wages is (8-2%) 6%.
· The real wage is a better guide to how your living standards changes. It shows what you are actually able to buy with the extra increase in wages.
· If wages increased 80%, but inflation was also 80%, the real increase in wages would be 0% – in effect, despite the monetary increase in wages of 80%, the
amount of goods and services you could buy would be the same.
Here is an analysis-ready version of the United States Consumer Product Safety Commission’s National Electronic Injury Surveillance System (NEISS) data from 2016-2020.
The raw data can be found here
The bit.io repository also links to the R script used for cleaning the data. The major data cleaning steps involved merging multiple years of data (originally need to download year-by-year as excel files) and translating numerical codes to more descriptive values (e.g. injury type 67 to “Electric shock”). This involved quite a bit of careful alignment across years of data.
Some key data characteristics:
More than 1.5M records of product-related injuries
Five complete years of data
Categorical columns indicating (1) which product(s) were involved in the injury; (2) which body part(s) were harmed; and (3) what the diagnosis was
Weights to extrapolate from individual records to nationally-representative estimates
Narrative summaries of each incident (I think there’s a lot of potential for some kind of NLP project with these summaries).
Projects the author has done with the data so far (more self-promotion, and hopefully some inspiration):
Local Outlier Factor Analysis with Scikit-Learn: includes a section applying outlier analysis to the NEISS data and concludes that holidays are outlier in terms of patterns of injuries (July 4 Fireworks).
Independence Day is the Most Dangerous Holiday: A holiday-themed analysis of injury data comparing holidays to determine which is the most dangerous and why (hint: fireworks).
Question: I have been researching some science problems that could be answered with queries and analysis of large or big datasets from public sources in climate, environmental data, energy, utilities, infrastructure, astronomy, economics, labor and industry, education, sociology, and health.
Probably the largest obstacle in many of these public datasets is the inability to conveniently run ad hoc analyses on just the data you need. Often the data lives in massive stores of file archives instead of databases.
Most convenient would be datasets already stored in a database that has some in-database processing and analytics available to aggregate or filter the data being queried before a data transfer.
Are there any public large datasets with such a convenient interface or API? The best I know of is perhaps:
-Socrata SODA API for mixed/misc. gov data
-CIA Factbook API
-FRED API and web viz for economic data
-Google Finance API
-Weather.gov API (forecasts/alerts only)
-USGS Earthquakes API
-WHO GHO OData API
-John Hopkins COVID-19 API
-Google ngram API
-Kaggle (usually deprecated/non-updated sets)
-AWS Open Data (no free basic processing at all)
-Bigquery Public Datasets (1TB of free queries, 1TB scans is quite limiting)
but most of these are extremely basic =match filters. I’m looking for other better examples before investing time to transfer large amounts of data just to filter other datasets down.
You can do this on bit.io; we saw this same problem and built a platform to let you query across real databases using SQL. So, for example you can take the NYTimes COVID data: https://bit.io/bitdotio/nytimes_covid/ And the JHU COVID data: https://bit.io/bitdotio/csse_covid_19_data/
And you can write SQL that joins them by FIPS code:
SELECT state, county, date, filename, cases, "bitdotio/nytimes_covid"."us_counties".deaths AS nytimes_deaths, "bitdotio/csse_covid_19_data"."csse_covid_19_daily_reports_us".deaths AS csse_deaths FROM "bitdotio/nytimes_covid"."us_counties", "bitdotio/csse_covid_19_data"."csse_covid_19_daily_reports_us" WHERE "bitdotio/nytimes_covid"."us_counties".fips=6059 AND "bitdotio/csse_covid_19_data"."csse_covid_19_daily_reports_us".fips=6059 AND date=last_update::date
According to the author, They have a real time map available. Some things I was thinking would be interesting..
Historic trend of fire containment. Ex, as the number of fires increase, I would expect the time to fully contain it would also increase.
Rate of spread. Are fires spreading more quickly.
Quantity and size distribution.
The author uploaded a dataset of MRI Scans for brain tumor segmentation. It is the training set for the BraTS competition for the years 2018, 2019 and 2020. The data contains MRI scans and expert segmentations for HGG and LGG (high grade and low grade gliomas), as well as survival data.
It can be used for tumor type classification, tumor segmentation and survival analysis.
All Digitized Texas Appeals Court Cases Since 1900 – 12GB – 696,036 cases
Scope of Data
All electronically-available Texas Appeals Court cases filed since 1900 (as of 2021-08-01).
Courts included: Texas Supreme Court, Court of Criminal Appeals, 14 Appeals Courts (regional)
Total cases: 696,036.
Full dataset approx 12GB.
Sample dataset approx 10MB.
Download 500 sample rows or full dataset — in either SQL or JSON format.
Aggregated by: https://www.judyrecords.com
Parsed fields include caseId, createdAt (Unix timestamp), siteCaseNum, courtKey, and httpReq.
Detailed case information is not parsed within each case currently, but can be parsed from standardized HTML structure.
Sample case from data source: https://search.txcourts.gov/Case.aspx?cn=01-20-00103-CV&coa=coa01
Court structure of Texas: https://www.txcourts.gov/media/1452084/court-structure-chart-february-2021.pdf
Case Count by Court
|Court||Case Count||Court Key|
|Texas Supreme Court||65,945||cossup|
|Texas Court of Criminal Appeals||242,915||coscca|
|Texas Court of Appeals #1||46,663||coa01|
|Texas Court of Appeals #2||34,458||coa02|
|Texas Court of Appeals #3||24,629||coa03|
|Texas Court of Appeals #4||33,469||coa04|
|Texas Court of Appeals #5||69,112||coa05|
|Texas Court of Appeals #6||12,206||coa06|
|Texas Court of Appeals #7||17,136||coa07|
|Texas Court of Appeals #8||16,180||coa08|
|Texas Court of Appeals #9||18,710||coa09|
|Texas Court of Appeals #10||14,550||coa10|
|Texas Court of Appeals #11||13,058||coa11|
|Texas Court of Appeals #12||14,366||coa12|
|Texas Court of Appeals #13||26,440||coa13|
|Texas Court of Appeals #14||46,199||coa14|
From the authors: We are sharing an open OSDG Community Dataset (OSDG-CD) on our GitHub. The dataset contains thousands of text excerpts labelled by citizen scientists from around the world with respect to the UN Sustainable Development Goals (SDGs).
The data can be used to derive insights into the nature of SDGs using either ontology-based or machine learning approaches.
OSDG-CD is a direct contribution of hundreds of volunteers who have already taken part in the OSDG Community platform citizen science exercise. The OSDG Community Platform is an ambitious attempt to bring together volunteers and subject matter experts from all around the world to create a large and accurate source of textual information on SDGs.
How does it work? We use publicly available texts such as publications, reports and other written data sources. Each text is broken down into smaller pieces of paragraph length, and these smaller pieces are then labelled by the Community volunteers.
We are making this data open to help researchers discover new insights into and meaningful connections among Sustainable Development Goals. We would like to know what you discover in the data. So do not hesitate to share with us your outputs, be it a research paper, a machine learning model, a blog post, or just an interesting observation. If you are using the dataset in a research paper, you can attribute the dataset as OSDG Community Dataset v2021.07.
This article explains what’s Z-score and how it makes a difference to our datasets.
#dataanalytics #machinelearning #analytics #visualization #datasets
The author scripted Blender to generate a synthetic dataset for 600 unique lego parts with multiple parts per image resulting in 900,000 labeled class instances!
What’s cool about this dataset:
It’s the largest publicly available LEGO dataset for object detection
Uses SoTA domain randomization techniques to bridge the sim-to-real gap
Cheap to generate more data with DreamFactor
- You can train an image detection ML model that can detect different lego parts
How did you script blender to make you Lego pieces? And where can I learn this magic? Answer
Posting this here to be more visible to a Google search on the off chance someone else could use it. It was used to generate Pokemon names for an AI hobby project I worked on some months ago:
The source for chosen words was Bulbapedia. The dataset had to be compiled manually as the English name origins didn’t lend themselves well to being scraped.
The author is looking for a data set with these fields: [‘author’, ‘text’, ‘label’ (fake or real)] like this one
The U.S. Gender Pay Gap: Visualized by Professions and its pay.
Data Source – Women in the labor force: a databook (table 18)
Program: Google Data Studio (scatter plot); Figma (design) See data (with plot labels).
Tile cartogram of annual CO2 emissions by country (2019 data)
This map rescales each country by its CO2 emissions, rounded to the nearest 10 megatons. I did my best to preserve country shape and relative locations. Each square is 10 Mt of CO2 emitted in 2019. Countries that did not reach 5 Mt were lumped together with other countries as black squares.
Data is from here. This was made in a combination of GIMP, Python, and Inkscape.
From the author: This chart was created for the Policy chapter of the Renewables 2021 Global Status Report and is based on data from the World Bank, Energy Climate Intelligence Unit, IEA Global Electric Vehicle Outlook and the REN21 Policy Database. For more information read Chapter 02 (Policy Landscape) of the report.
All Time NBA Team Win %’s (Playoffs vs Regular Season)
The data is from wikipedia and the graphic was made with R
Histomap: Visualizing the 4,000 Year History of Global Power
Life expectancy at birth across the US, the EU, India, and China. Data for 2019.
For comparison, in ancient Greece times life expectancy was 25 years, in medieval Europe it was 35 years, in early 19th century England it was 40 years, and in 1950 world average life expectancy was 45 years
Kings and Queens of England and Great Britain (by name, dynasty, gender)
Ronaldo vs Messi when they were Trending on Twitter Worldwide
Author: The VisualizED
Data Source: Twitter API.
Visualization generated by my Application thevisualized.com.
Every time Cristiano Ronaldo and Lionel Messi were trending on Twitter in the Year 2020 📅
Full HD Video https://youtu.be/uggWXz0IREY 📹
🟢 Cristiano Ronaldo was trending 168+ Times in the Year 2020
#Ronaldo Trends with an average of 70.8K (Thousands) Tweets
Find more on The Visualized Twitter Timeline of Cristiano Ronaldo
93.5M Followers https://thevisualized.com/twitter/timeline/Cristiano 📊
🔴 Lionel Messi was trending 245+ Times in the Year 2020
#Messi Trends with an average of 97.3K (Thousands) Tweets
Find more on The Visualized Twitter Timeline of Lionel Mess
3M Followers https://thevisualized.com/twitter/timeline/TeamMessi 📊
Find what’s currently Trending Worldwide https://thevisualized.com/trending
Tool – Python, Altair
Carbon Free Energy Generation in the United States (1990-2018)
This visualization showcases the proportion of energy generation in each state by carbon and carbon-free energy sources. A greener shade correlates to a higher proportion of green energy generated in that state. Let me know any suggestions or insights you might have! Also, would energy consumption data be more interesting than energy generation? Let me know!
The Pacific Northwest is a bright light of non-carbon energy in the form of nuclear.
Also, S/O to that midwestern corn and its role in the biofuels industry.
100 biggest companies by market capitalization
Source: PwC, 2021 Map made with QGIS and Adobe Illustrator
r:Number of people killed in Afghanistan over the years in terrorist attacks
Source of the data – Global Terrorism Data, https://gtd.terrorismdata.com/files/gtd-1970-2019-4/
Tool – Python, Altair
Most notorious extremist groups in Afghanistan in terms of deaths caused
Source of the data – Global Terrorism Data, https://gtd.terrorismdata.com/files/gtd-1970-2019-4/
Tool – Python, Altair
The author used Olympic medals listed and used GGanimate in R to make this animation.
Japan electricity production 1914-2019
Legality of cannabis (marijuana) across the US and the EU. August 2021 data 🇺🇸🇪🇺
Tools: MS Office
Same-sex marriage public support across the US and the EU. 2017-2019 data 🇺🇸🇪🇺
UK dark green (73%): (2012)
Norway dark green (78%): Here
Andorra dark green (70%): Here
Serbia red (26%): Here
Bosnia red (13%): Here
As per this EU report, which should be where the other EU data came from, it’s 85%
World Index of Moral Freedom 2020
Tools: 1 2
Wikipedia’s description of the index:
The World Index of Moral Freedom is sponsored and published by the Foundation for the Advancement of Liberty, a libertarian think tank based in Madrid, Spain. The Index is an international index ranking one hundred and sixty countries on their performance on five categories of indicators:
religious freedom (taking into account both the freedom to practice any religion or none, and the situation of religious control on the state);
bioethical freedom (including the legal status of abortion, euthanasia and other practices pertaining to bioethics, like surrogacy or stem cell research);
drugs freedom (including the legal status of cannabis and the country’s general policy on hard drugs);
sexual freedom (including the legal status of pornography and sex services among consenting adults, and the country’s age of sexual consent), and
family and gender freedom (including women’s freedom of movement, the legal status of cohabitation of unmarried couples, same sex marriage and the situation of transgender people).
The US score dropped significantly more between 2018->2020 than other country in the top 25% of the list. Anyone know what changed?
The absolute score decreased from 79.15 to 73.68
The religion freedom indicator remained almost the same at 97.13 vs 97.12
The bioethical freedom indicator decreased slightly from 89.38 to 88.13
The drugs freedom indicator increased significantly from 45.75 to 65.18
Sexual freedom decreased dramatically from 73.50 to 30.00
Gender & family freedom decreased slightly from 90.00 to 88.00
The 2020 report attributes some amount of the loss to methodological changes (most severely impacting Cambodia’s ranking), but the decline seems to be driven primarily by the sexual freedom indicator. Here’s what the 2020 report has to say on that category:
How free are sexual intercourse, pornography and the provision of sex services
As the sexual revolution keeps spreading to reach all places, the amount of government interference provides useful information on a country’s individual freedom on moral decisions. In this category, indicator weights are more distributed: 40% is allocated to the free consumption of pornographic content. This is significant because censorship still plays a role in many countries, while technology makes it increasingly harder for states. 35% is reserved to the legal status of prostitution, and 25% to the legal age of sexual consent.
It passed in 2018 and made it a lot harder to advertise prostitution online
The source doesn’t even break down the scores. I don’t understand how there could be such a discrepancy between France and Spain for example, anyone got a clue?
Edit: nvm, it does, it’s just weirdly formatted. The gist is France heavily criminalizes drugs, Spain does not. All of the other differences between the two countries are mostly ignored by this study. Besides French draconian drug laws, bioethical freedom may account for the discrepancy too. Euthanasia in Spain is legal and publicly funded, framed within the public health system. In France, not so much.
Deaths from all causes in the United States: year-to-year comparison 2015-2021 (through week 30)
Average alcohol consumption by country
2021 World Happiness Index
Finns: Often pessimistic by nature and reserved about their emotions, drink too much, it’s dark, the winters are cold and hard psychologically. Also Finns: We are the happiest!
The discrepancy comes from the fact that the happiness study is, paradoxically, not actually about happiness. The World Happiness Report is not an emotional study at all, it is rather a look at the quality of life (GDP, education, health, security, freedom,…) around the world.
It should be labeled as potential happiness, not actual happiness, because actual happiness is impossible to measure. But I wouldn’t say measuring smiles in the street is a good way either, in many cultures smiling is customary, not necessarily a indication of happiness.
Happiness can be measured by self reporting in a survey. How happy are you with life right now 1-10? It’s a subjective data point but happiness is also subjective after all.
Measuring smiles in the street as a way to measure happiness seems insanely ridiculous.
Self-reporting is also a weird thing. People tend to measure their life against their surroundings and are affected by small-scale personal events.
If I lived my entire life in safety and comfort, I am most likely to take those things for granted and not consider them as contribution to my happiness. My personal problems on the other hand can affect my emotional state quite a lot. If my mom is ill or I’ve had a bad fight with my best friend, I would be far from happy no matter where I live.
World Broadband Internet Speed 2021
Tools: 1 2
Data originates from 1
Internet Speed is understood as average download speed in Mbps.
Yearly road deaths per million people across the US and the EU.
This calculation includes drivers, passengers, and pedestrians who died in car, motorcycle, bus, and bicycle accidents. 2018-2019 data 🇺🇸🇪🇺
Tools: MS Office
I actually found an interesting graphic here that breaks down where most of our crashes and fatalities come from! It doesn’t include winter conditions as a factor, so I can’t use that information, but according to this: single car alcohol related accidents are our #1 killer. We have a rampant drinking and driving problem here.
Classic Machine Learning Algorithms
Each chapter in this book corresponds to a single machine learning method or group of methods. Each method includes the elaboration of Concepts and the implementation of Python Codes (construct algorithm from scratch).
Credit: Danny Friedman
Deep Learning by Andrew Ng
One of the most comprehensive Deep Learning books in 2021
Amazon Machine Learning
Credit: Amazon Web Services
Learn SQL with Practical Exercises
SQL is definitely one of the most fundamental skills needed to be a data scientist.
This is a comprehensive handbook that can help you to learn SQL (Structured Query Language), which could be directly downloaded here
Credit: D Armstrong
How important in life is family, work, friends, leisure, religion, and politics?
Answers from the World Values Survey. Results from each region of the world in separate images.
Data from the World Values Survey and European Values Survey, wave 7. All data (as well as from previous waves) can be accessed and analyzed online here.
In general about 1000-3000 people answered the survey in each country. Many countries, for instance India, did not take part in the survey this wave. Some have however been part of previous waves, and their answers can be analyzed online.
Made with R with the ggplot and ggflags packages.
Data Visualization: A comprehensive VIP Matplotlib Cheat sheet
Machine Learning Course by Microsoft
In this course you will learn:
Core concepts of machine learning
The history of ML
ML and fairness
Regression ML techniques
Classification ML techniques
Clustering ML techniques
Natural language processing ML techniques
Time series forecasting ML techniques
Real-world applications for ML
-Start with a pre-lecture quiz
-Read the lecture and complete the activities, pausing and reflecting at each knowledge check.
-Try to create the projects by comprehending the lessons rather than running the solution code; however that code is available in the /solution folders in each project-oriented lesson.
-Take the post-lecture quiz
-Complete the challenge
-Complete the assignment
It is a pretty comprehensive course with all the material you need to learn. Enjoy!
Check it out here:
Free, cleaned database of US Schools
At DoltHub, they have completed a US Schools data bounty. They built a free, cleaned database of US Schools.
Database to collect US Schools identifying information, K through 12 and Post Secondary.
The Rosenbrock dataset suite for benchmarking machine learning algorithms and platforms
This post introduces the Rosenbrock function to measure a machine learning platforms’ data capacity, training speed, model accuracy, and inference speed. Rosenbrock datasets have a strong consistency and do not have noise. For this reason, it is a powerful alternative to datasets from popular repositories for benchmarking.
Poverty headcount ratio at $1.90 a day (2011 PPP) (% of population)2>
- Prevalence of violent crime
- Cost of living. I want to know given $X USD, how far that gets you for rent, groceries, eating out, etc. There seems to be tons of stuff that tries to estimate this but what’s the best one? Is the Big Mac index a good reference or a meme?
- Average historical temperature, and precipitation (rain) in January
- Whether it’s land-locked or coastal (for beaches. I’ve been land-locked my whole life.)
- Native languages spoken
- Population density
Poverty headcount ratio at $1.90 a day is the percentage of the population living on less than $1.90 a day at 2011 international prices.
World Bank, Development Research Group. Data are based on primary household survey data obtained from government statistical agencies and World Bank country departments. Data for high-income economies are from the Luxembourg Income Study database. For more information and methodology, please see PovcalNet.
A Dataset of Cryptic Crossword Clues
A dataset of cryptic crossword clues, collected from various blogs and publicly available digital archives.
It’s a little over half a million clues from cryptic crosswords published in British newspapers over the past twelve years.
The Upworthy Research Archive, a time series of 32,487 experiments in U.S. media
The dataset is available under the Creative Commons Attribution 4.0 International License on the Upworthy Research Archive website at https://upworthy.natematias.com, with an archival copy on the Open Science Framework20. The website include a description of each column in the data, a list of resources and papers based on the dataset, and guidance for meta-analyzing the included experiments. The data are stored using a plain-text, ASCII-encoded, comma-delimited CSV file.
Power BI for Intermediates
Credit: Soheil Bakhshi and Bruce Anderson
Number of Open Missing Persons Cases per 100k People in Each US State
Sources: The National Missing and Unidentified Persons System (NamUs), US Census Bureau 2020 Population Data
Tools: QGIS, Excel
Notes from the author: This map depicts the number of open missing persons cases per 100k people in each US State as of September 23, 2021. NamUs collects data from law enforcement agencies and provides data and services to forensic investigators to locate and identify missing persons and unidentified bodies. It is important to note that while efforts are currently underway for more accurate counts at local, state, and national levels, the true number of open missing persons cases among indigenous persons is unknown due to systemic issues. Of the current number of open missing persons cases, approximately 3.6% are for indigenous persons but this number is estimated to be higher. The ongoing Gabby Petito investigation inspired me to look further into this topic and I was personally surprised to see the shocking numbers of missing persons in many of these states. I encourage people to look through the database provided to understand the issue further and to read about the efforts to ensure more accurate counts of the missing indigenous persons in this country. As always, I am open to constructive feedback and questions about this map. So please leave a comment or question and I will try my best to answer you soon. Thank you for reading and please be kind and look out for each other. Stay awesome Reddit.
U.S. Automobile Fatalities, 1920-2020
Data from National Safety Council. Created in Excel/Illustrator. Full description found here
Brazilian Racial Distribution by Region
Race in Brazil is set by self declaration.
Where and when were Lewis Hamilton’s 100 victories won
ggplot2), Affinity Publisher
Data wrangling: Postgres, Python
Data: Ergast API
More than thirty years fighting against the HIV/AIDS epidemic. Evolution and Milestones.
The HIV is one of the most severe epidemics in human history. It disproportionally affects demographic groups with limited access to steady and quality healthcare such as racial/ethnic minorities, people with alternative sexual orientations, people who inject drugs, and people living in poverty. Only a coordinated, global action, has made possible the improvements that now, after over 30 years of fighting the epidemic, we can see. Ending the HIV/AIDS epidemic requires innovative solutions to understand the healthcare access barrier in each setting and finally provide care (for diagnostics, preventions, and sustained antiretroviral therapy) to all people-at-risk and living with HIV.
For more details of the evolution of the epidemic and the actions deployed to contain it, see this very interesting timeline created by HIV.gov: https://www.hiv.gov/hiv-basics/overview/history/hiv-and-aids-timeline
The top goal scorers in 40 years of elite football (soccer)
This viz aggregates all passes on a grid with 1 meter step. It means, all distances and passes on a square meter of the football pitch represent by a line with average length and direction. So this viz is an ‘averaged’ picture.
The data includes:
– Champion League 1999 – 2019
– FA Women’s Super League 2018 – 2020
– FIFA World Cup 2018
– La Liga 2004 – 2020
– NWSL 2018
– Premier League 2003 – 2004
– Women’s World Cup 2019
The 186 players shown are those who have scored at least 20 league goals in any single season of one of Europe’s big five leagues since 1990, and have a career-average goal-scoring rate across senior matches in these five countries of at least 0.4 non-penalty goals per 90 minutes.
If you liked this, you might like The top scorers in Champions League history.
Economic prosperity and soccer prowess
The author made the visualization with ggplot2, with data from eloratings.net and the World Bank.
Breakdown of worldwide greenhouse gases emissions by source, 2019
Data from: IEA, Global Carbon Project, IPCC, FAO, World Resources Institute
Viz Made with Google slides
Some interesting comments:
1- I find it crazy that there’s about 30,000 planes or so around the world and about 1,500,000,000 personal cars, that’s 50,000x more, yet cars only produce 3.5x as much pollution. Even crazier is how cargo ships, who spew out some of the most foul crude oil emissions, produce the same amount as planes. I would have never thought lol
2- Utility-scale solar is increasingly cheaper than the operational expense of maintaining a coal plant, and grows ever more so. Economics is no longer the central problem for coal; the obstacle is entrenched fossil capital throwing all the political heft it can behind a losing hand (as well as, to a certain extent, pressure from military planners for autarky)
3- In Canada we have an issue where emissions from almost all sources and places going down is counter acted by emissions from Alberta and Saskatchewan going up, mostly due to their fossil fuel industries. Where in this graphic would the emissions created by the extraction, processing and distribution of fossil fuels go? In the other industrial usages categories?
4- Switching road vehicles to EVs will get rid of ~6 Gt.
(increased direct electricity offset by reductions in refining electricity, reductions in fugitive emissions, reduction in ocean freight emissions).
This is underway, but the more we push (as individuals and as voters) the sooner it can happen.
History of teen fertility rates in the U.S.
Data from CDC WONDER query. Visualization with Tableau.
Some notable comments:
1- Halved in a decade? What was it like before? Presumably on a decline as steep as it is now?
2- It’s fascinating to me as I don’t see this as being so different from 2010. I guess sometimes we’re just blind to the short term changes going on around us.
3- We are living in the lowest violence period in history. We have included lots of stuff that used to be accepted ( like child abuse and wife beating) and it’s still way down. Kids today are just so much better than old people. Old people ? They couldn’t wait to get knocked up!
4- It’s not really that they couldn’t wait to get knocked up. It’s generally either that they were actually capable of making an income necessary to support a family way younger than today if we are looking at 50s and 60s data when the average marriage age was at its lowest recorded in US history, or they didn’t want to get pregnant but didn’t have sex education or birth control. As far as violence goes unfortunately the pandemic era data is looking worse that 2019.
800 Data Science Questions & Answers doc by Steve Nouri
Q1: What is the Central Limit Theorem and why is it important?
Answer1: Suppose that we are interested in estimating the average height among all people. Collecting data for every person in the world is impractical, bordering on impossible. While we can’t obtain a height measurement from everyone in the population, we can still sample some people. The question now becomes, what can we say about the average height of the entire population given a single sample.
The Central Limit Theorem addresses this question exactly. Formally, it states that if we sample from a population using a sufficiently large sample size, the mean of the samples (also known as the sample population) will be normally distributed (assuming true random sampling), the mean tending to the mean of the population and variance equal to the variance of the population divided by the size of the sampling.
What’s especially important is that this will be true regardless of the distribution of the original population.
As we can see, the distribution is pretty ugly. It certainly isn’t normal, uniform, or any other commonly known distribution. In order to sample from the above distribution, we need to define a sample size, referred to as N. This is the number of observations that we will sample at a time. Suppose that we choose
N to be 3. This means that we will sample in groups of 3. So for the above population, we might sample groups such as [5, 20, 41], [60, 17, 82], [8, 13, 61], and so on.
Suppose that we gather 1,000 samples of 3 from the above population. For each sample, we can compute its average. If we do that, we will have 1,000 averages. This set of 1,000 averages is called a sampling distribution, and according to Central Limit Theorem, the sampling distribution will approach a normal distribution as the sample size N used to produce it increases. Here is what our sample distribution looks like for N = 3.
As we can see, it certainly looks uni-modal, though not necessarily normal. If we repeat the same process with a larger sample size, we should see the sampling distribution start to become more normal. Let’s repeat the same process again with N = 10. Here is the sampling distribution for that sample size.
Credit: Steve Nouri
Q2: What is sampling? How many sampling methods do you know?
Data sampling is a statistical analysis technique used to select, manipulate and analyze a representative subset of data points to identify patterns and trends in the larger data set being examined. It enables data scientists, predictive modelers and other data analysts to work with a small, manageable amount of data about a statistical population to build and run analytical models more quickly, while still producing accurate findings.
Sampling can be particularly useful with data sets that are too large to efficiently analyze in full – for example, in big data analytics applications or surveys. Identifying and analyzing a representative sample is more efficient and cost-effective than surveying the entirety of the data or population.
An important consideration, though, is the size of the required data sample and the possibility of introducing a sampling error. In some cases, a small sample can reveal the most important information about a data set. In others, using a larger sample can increase the likelihood of accurately representing the data as a whole, even though the increased size of the sample may impede ease of manipulation and interpretation.
There are many different methods for drawing samples from data; the ideal one depends on the data set and situation. Sampling can be based on probability, an approach that uses random numbers that correspond to points in the data set to ensure that there is no correlation between points chosen for the sample. Further variations in probability sampling include:
Simple random sampling: Software is used to randomly select subjects from the whole population.
• Stratified sampling: Subsets of the data sets or population are created based on a common factor,
and samples are randomly collected from each subgroup. A sample is drawn from each strata (using a random sampling method like simple random sampling or systematic sampling).
o EX: In the image below, let’s say you need a sample size of 6. Two members from each
group (yellow, red, and blue) are selected randomly. Make sure to sample proportionally:
In this simple example, 1/3 of each group (2/6 yellow, 2/6 red and 2/6 blue) has been
sampled. If you have one group that’s a different size, make sure to adjust your
proportions. For example, if you had 9 yellow, 3 red and 3 blue, a 5-item sample would
consist of 3/9 yellow (i.e. one third), 1/3 red and 1/3 blue.
• Cluster sampling: The larger data set is divided into subsets (clusters) based on a defined factor, then a random sampling of clusters is analyzed. The sampling unit is the whole cluster; Instead of sampling individuals from within each group, a researcher will study whole clusters.
o EX: In the image below, the strata are natural groupings by head color (yellow, red, blue).
A sample size of 6 is needed, so two of the complete strata are selected randomly (in this
example, groups 2 and 4 are chosen).
• Multistage sampling: A more complicated form of cluster sampling, this method also involves dividing the larger population into a number of clusters. Second-stage clusters are then broken out based on a secondary factor, and those clusters are then sampled and analyzed. This staging could continue as multiple subsets are identified, clustered and analyzed.
• Systematic sampling: A sample is created by setting an interval at which to extract data from the larger population – for example, selecting every 10th row in a spreadsheet of 200 items to create a sample size of 20 rows to analyze.
Sampling can also be based on non-probability, an approach in which a data sample is determined and extracted based on the judgment of the analyst. As inclusion is determined by the analyst, it can be more difficult to extrapolate whether the sample accurately represents the larger population than when probability sampling is used.
Non-probability data sampling methods include:
• Convenience sampling: Data is collected from an easily accessible and available group.
• Consecutive sampling: Data is collected from every subject that meets the criteria until the predetermined sample size is met.
• Purposive or judgmental sampling: The researcher selects the data to sample based on predefined criteria.
• Quota sampling: The researcher ensures equal representation within the sample for all subgroups in the data set or population (random sampling is not used).
Once generated, a sample can be used for predictive analytics. For example, a retail business might use data sampling to uncover patterns about customer behavior and predictive modeling to create more effective sales strategies.
Credit: Steve Nouri
Q3: What is linear regression? What do the terms p-value, coefficient, and r-squared value mean? What is the significance of each of these components?
Imagine you want to predict the price of a house. That will depend on some factors, called independent variables, such as location, size, year of construction… if we assume there is a linear relationship between these variables and the price (our dependent variable), then our price is predicted by the following function: Y = a + bX
The p-value in the table is the minimum I (the significance level) at which the coefficient is relevant. The lower the p-value, the more important is the variable in predicting the price. Usually we set a 5% level, so that we have a 95% confidentiality that our variable is relevant.
The p-value is used as an alternative to rejection points to provide the smallest level of significance at which the null hypothesis would be rejected. A smaller p-value means that there is stronger evidence in favor of the alternative hypothesis.
The coefficient value signifies how much the mean of the dependent variable changes given a one-unit shift in the independent variable while holding other variables in the model constant. This property of holding the other variables constant is crucial because it allows you to assess the effect of each variable in isolation from the others.
R squared (R2) is a statistical measure that represents the proportion of the variance for a dependent variable that’s explained by an independent variable or variables in a regression model.
Credit: Steve Nouri
Q5: What are the assumptions required for linear regression?
There are four major assumptions:
There is a linear relationship between the dependent variables and the regressors, meaning the model you are creating actually fits the data,
• The errors or residuals of the data are normally distributed and independent from each other,
• There is minimal multicollinearity between explanatory variables, and
• Homoscedasticity. This means the variance around the regression line is the same for all values of the predictor variable.
Q6: What is a statistical interaction?
Reference: Statistical Interaction
Basically, an interaction is when the effect of one factor (input variable) on the dependent variable (output variable) differs among levels of another factor. When two or more independent variables are involved in a research design, there is more to consider than simply the “main effect” of each of the independent variables (also termed “factors”). That is, the effect of one independent variable on the dependent variable of interest may not be the same at all levels of the other independent variable. Another way to put this is that the effect of one independent variable may depend on the level of the other independent
variable. In order to find an interaction, you must have a factorial design, in which the two (or more) independent variables are “crossed” with one another so that there are observations at every
combination of levels of the two independent variables. EX: stress level and practice to memorize words: together they may have a lower performance.
Q7: What is selection bias?
Selection (or ‘sampling’) bias occurs when the sample data that is gathered and prepared for modeling has characteristics that are not representative of the true, future population of cases the model will see.
That is, active selection bias occurs when a subset of the data is systematically (i.e., non-randomly) excluded from analysis.
Selection bias is a kind of error that occurs when the researcher decides what has to be studied. It is associated with research where the selection of participants is not random. Therefore, some conclusions of the study may not be accurate.
The types of selection bias include:
• Sampling bias: It is a systematic error due to a non-random sample of a population causing some members of the population to be less likely to be included than others resulting in a biased sample.
• Time interval: A trial may be terminated early at an extreme value (often for ethical reasons), but the extreme value is likely to be reached by the variable with the largest variance, even if all variables have a similar mean.
• Data: When specific subsets of data are chosen to support a conclusion or rejection of bad data on arbitrary grounds, instead of according to previously stated or generally agreed criteria.
• Attrition: Attrition bias is a kind of selection bias caused by attrition (loss of participants)
discounting trial subjects/tests that did not run to completion.
Q8: What is an example of a data set with a non-Gaussian distribution?
The Gaussian distribution is part of the Exponential family of distributions, but there are a lot more of them, with the same sort of ease of use, in many cases, and if the person doing the machine learning has a solid grounding in statistics, they can be utilized where appropriate.
Binomial: multiple toss of a coin Bin(n,p): the binomial distribution consists of the probabilities of each of the possible numbers of successes on n trials for independent events that each have a probability of p of
Bernoulli: Bin(1,p) = Be(p)
Q9: What is bias-variance trade-off?
Bias: Bias is an error introduced in the model due to the oversimplification of the algorithm used (does not fit the data properly). It can lead to under-fitting.
Low bias machine learning algorithms — Decision Trees, k-NN and SVM
High bias machine learning algorithms — Linear Regression, Logistic Regression
Variance: Variance is error introduced in the model due to a too complex algorithm, it performs very well in the training set but poorly in the test set. It can lead to high sensitivity and overfitting.
Possible high variance – polynomial regression
Normally, as you increase the complexity of your model, you will see a reduction in error due to lower bias in the model. However, this only happens until a particular point. As you continue to make your model more complex, you end up over-fitting your model and hence your model will start suffering from high variance.
Bias-Variance trade-off: The goal of any supervised machine learning algorithm is to have low bias and low variance to achieve good prediction performance.
1. The k-nearest neighbor algorithm has low bias and high variance, but the trade-off can be changed by increasing the value of k which increases the number of neighbors that contribute to the prediction and in turn increases the bias of the model.
2. The support vector machine algorithm has low bias and high variance, but the trade-off can be changed by increasing the C parameter that influences the number of violations of the margin allowed in the training data which increases the bias but decreases the variance.
3. The decision tree has low bias and high variance, you can decrease the depth of the tree or use fewer attributes.
4. The linear regression has low variance and high bias, you can increase the number of features or use another regression that better fits the data.
There is no escaping the relationship between bias and variance in machine learning. Increasing the bias will decrease the variance. Increasing the variance will decrease bias.
Q10: What is Data Science? List the differences between supervised and unsupervised learning.
Data Science is a blend of various tools, algorithms, and machine learning principles with the goal to discover hidden patterns from the raw data. How is this different from what statisticians have been doing for years? The answer lies in the difference between explaining and predicting: statisticians work a posteriori, explaining the results and designing a plan; data scientists use historical data to make predictions.
The differences between supervised and unsupervised learning are:
|Input data is labelled||Input data is unlabeled|
|Split in training/validation/test||No split|
|Used for prediction||Used for analysis|
|Classification and Regression||
Clustering, dimension reduction,
and density estimation
Q11: What is a confusion matrix?
The confusion matrix is a 2X2 table that contains 4 outputs provided by the binary classifier.
A data set used for performance evaluation is called a test data set. It should contain the correct labels and predicted labels. The predicted labels will exactly the same if the performance of a binary classifier is perfect. The predicted labels usually match with part of the observed labels in real-world scenarios.
A binary classifier predicts all data instances of a test data set as either positive or negative. This produces four outcomes: TP, FP, TN, FN. Basic measures derived from the confusion matrix:
Q12: What is the difference between “long” and “wide” format data?
In the wide-format, a subject’s repeated responses will be in a single row, and each response is in a separate column. In the long-format, each row is a one-time point per subject. You can recognize data in wide format by the fact that columns generally represent groups (variables).
Q13: What do you understand by the term Normal Distribution?
Data is usually distributed in different ways with a bias to the left or to the right or it can all be jumbled up. However, there are chances that data is distributed around a central value without any bias to the left or right and reaches normal distribution in the form of a bell-shaped curve.
The random variables are distributed in the form of a symmetrical, bell-shaped curve. Properties of Normal Distribution are as follows:
1. Unimodal (Only one mode)
2. Symmetrical (left and right halves are mirror images)
3. Bell-shaped (maximum height (mode) at the mean)
4. Mean, Mode, and Median are all located in the center
Q14: What is correlation and covariance in statistics?
Correlation is considered or described as the best technique for measuring and also for estimating the quantitative relationship between two variables. Correlation measures how strongly two variables are related. Given two random variables, it is the covariance between both divided by the product of the two standard deviations of the single variables, hence always between -1 and 1.
Covariance is a measure that indicates the extent to which two random variables change in cycle. It explains the systematic relation between a pair of random variables, wherein changes in one variable reciprocal by a corresponding change in another variable.
Q15: What is the difference between Point Estimates and Confidence Interval?
Point Estimation gives us a particular value as an estimate of a population parameter. Method of Moments and Maximum Likelihood estimator methods are used to derive Point Estimators for population parameters.
A confidence interval gives us a range of values which is likely to contain the population parameter. The confidence interval is generally preferred, as it tells us how likely this interval is to contain the population parameter. This likeliness or probability is called Confidence Level or Confidence coefficient and represented by 1 − ∝, where ∝ is the level of significance.
Q16: What is the goal of A/B Testing?
It is a hypothesis testing for a randomized experiment with two variables A and B.
The goal of A/B Testing is to identify any changes to the web page to maximize or increase the outcome of interest. A/B testing is a fantastic method for figuring out the best online promotional and marketing strategies for your business. It can be used to test everything from website copy to sales emails to search ads. An example of this could be identifying the click-through rate for a banner ad.
Q17: What is p-value?
When you perform a hypothesis test in statistics, a p-value can help you determine the strength of your results. p-value is the minimum significance level at which you can reject the null hypothesis. The lower the p-value, the more likely you reject the null hypothesis.
Q18: What do you understand by statistical power of sensitivity and how do you calculate it?
Sensitivity is commonly used to validate the accuracy of a classifier (Logistic, SVM, Random Forest etc.). Sensitivity = [ TP / (TP +TN)]
Q19: Why is Re-sampling done?
- Sampling is an active process of gathering observations with the intent of estimating a population variable.
- Resampling is a methodology of economically using a data sample to improve the accuracy and quantify the uncertainty of a population parameter. Resampling methods, in fact, make use of a nested resampling method.
Once we have a data sample, it can be used to estimate the population parameter. The problem is that we only have a single estimate of the population parameter, with little idea of the variability or uncertainty in the estimate. One way to address this is by estimating the population parameter multiple times from our data sample. This is called resampling. Statistical resampling methods are procedures that describe how to economically use available data to estimate a population parameter. The result can be both a more accurate estimate of the parameter (such as taking the mean of the estimates) and a quantification of the uncertainty of the estimate (such as adding a confidence interval).
Resampling methods are very easy to use, requiring little mathematical knowledge. A downside of the methods is that they can be computationally very expensive, requiring tens, hundreds, or even thousands of resamples in order to develop a robust estimate of the population parameter.
The key idea is to resample from the original data — either directly or via a fitted model — to create replicate datasets, from which the variability of the quantiles of interest can be assessed without longwinded and error-prone analytical calculation. Because this approach involves repeating the original data analysis procedure with many replicate sets of data, these are sometimes called computer-intensive methods. Each new subsample from the original data sample is used to estimate the population parameter. The sample of estimated population parameters can then be considered with statistical tools in order to quantify the expected value and variance, providing measures of the uncertainty of the
estimate. Statistical sampling methods can be used in the selection of a subsample from the original sample.
A key difference is that process must be repeated multiple times. The problem with this is that there will be some relationship between the samples as observations that will be shared across multiple subsamples. This means that the subsamples and the estimated population parameters are not strictly identical and independently distributed. This has implications for statistical tests performed on the sample of estimated population parameters downstream, i.e. paired statistical tests may be required.
Two commonly used resampling methods that you may encounter are k-fold cross-validation and the bootstrap.
- Bootstrap. Samples are drawn from the dataset with replacement (allowing the same sample to appear more than once in the sample), where those instances not drawn into the data sample may be used for the test set.
- k-fold Cross-Validation. A dataset is partitioned into k groups, where each group is given the opportunity of being used as a held out test set leaving the remaining groups as the training set. The k-fold cross-validation method specifically lends itself to use in the evaluation of predictive models that are repeatedly trained on one subset of the data and evaluated on a second held-out subset of the data.
Resampling is done in any of these cases:
- Estimating the accuracy of sample statistics by using subsets of accessible data or drawing randomly with replacement from a set of data points
- Substituting labels on data points when performing significance tests
- Validating models by using random subsets (bootstrapping, cross-validation)
Q20: What are the differences between over-fitting and under-fitting?
In statistics and machine learning, one of the most common tasks is to fit a model to a set of training data, so as to be able to make reliable predictions on general untrained data.
In overfitting, a statistical model describes random error or noise instead of the underlying relationship.
Overfitting occurs when a model is excessively complex, such as having too many parameters relative to the number of observations. A model that has been overfitted, has poor predictive performance, as it overreacts to minor fluctuations in the training data.
Underfitting occurs when a statistical model or machine learning algorithm cannot capture the underlying trend of the data. Underfitting would occur, for example, when fitting a linear model to non-linear data.
Such a model too would have poor predictive performance.
Q21: How to combat Overfitting and Underfitting?
To combat overfitting:
1. Add noise
2. Feature selection
3. Increase training set
4. L2 (ridge) or L1 (lasso) regularization; L1 drops weights, L2 no
5. Use cross-validation techniques, such as k folds cross-validation
6. Boosting and bagging
7. Dropout technique
8. Perform early stopping
9. Remove inner layers
To combat underfitting:
1. Add features
2. Increase time of training
Q22: What is regularization? Why is it useful?
Regularization is the process of adding tuning parameter (penalty term) to a model to induce smoothness in order to prevent overfitting. This is most often done by adding a constant multiple to an existing weight vector. This constant is often the L1 (Lasso – |∝|) or L2 (Ridge – ∝2). The model predictions should then minimize the loss function calculated on the regularized training set.
Q23: What Is the Law of Large Numbers?
It is a theorem that describes the result of performing the same experiment a large number of times. This theorem forms the basis of frequency-style thinking. It says that the sample means, the sample variance and the sample standard deviation converge to what they are trying to estimate. According to the law, the average of the results obtained from a large number of trials should be close to the expected value and will tend to become closer to the expected value as more trials are performed.
Q24: What Are Confounding Variables?
In statistics, a confounder is a variable that influences both the dependent variable and independent variable.
If you are researching whether a lack of exercise leads to weight gain:
lack of exercise = independent variable
weight gain = dependent variable
A confounding variable here would be any other variable that affects both of these variables, such as the age of the subject.
Q25: What is Survivorship Bias?
It is the logical error of focusing aspects that support surviving some process and casually overlooking those that did not work because of their lack of prominence. This can lead to wrong conclusions in numerous different means. For example, during a recession you look just at the survived businesses, noting that they are performing poorly. However, they perform better than the rest, which is failed, thus being removed from the time series.
Q26: Explain how a ROC curve works?
The ROC curve is a graphical representation of the contrast between true positive rates and false positive rates at various thresholds. It is often used as a proxy for the trade-off between the sensitivity (true positive rate) and false positive rate.
Q27: What is TF/IDF vectorization?
TF-IDF is short for term frequency-inverse document frequency, is a numerical statistic that is intended to reflect how important a word is to a document in a collection or corpus. It is often used as a weighting factor in information retrieval and text mining.
The TF-IDF value increases proportionally to the number of times a word appears in the document but is offset by the frequency of the word in the corpus, which helps to adjust for the fact that some words appear more frequently in general.
Q28: Python or R – Which one would you prefer for text analytics?
We will prefer Python because of the following reasons:
• Python would be the best option because it has Pandas library that provides easy to use data structures and high-performance data analysis tools.
• R is more suitable for machine learning than just text analysis.
• Python performs faster for all types of text analytics.
Q29: How does data cleaning play a vital role in the analysis?
Data cleaning can help in analysis because:
- Cleaning data from multiple sources helps transform it into a format that data analysts or data scientists can work with.
- Data Cleaning helps increase the accuracy of the model in machine learning.
- It is a cumbersome process because as the number of data sources increases, the time taken to clean the data increases exponentially due to the number of sources and the volume of data generated by these sources.
- It might take up to 80% of the time for just cleaning data making it a critical part of the analysis task
Q30: Differentiate between univariate, bivariate and multivariate analysis.
Univariate analyses are descriptive statistical analysis techniques which can be differentiated based on one variable involved at a given point of time. For example, the pie charts of sales based on territory involve only one variable and can the analysis can be referred to as univariate analysis.
The bivariate analysis attempts to understand the difference between two variables at a time as in a scatterplot. For example, analyzing the volume of sale and spending can be considered as an example of bivariate analysis.
Multivariate analysis deals with the study of more than two variables to understand the effect of variables on the responses.
Q31: Explain Star Schema
It is a traditional database schema with a central table. Satellite tables map IDs to physical names or descriptions and can be connected to the central fact table using the ID fields; these tables are known as lookup tables and are principally useful in real-time applications, as they save a lot of memory. Sometimes star schemas involve several layers of summarization to recover information faster.
Q32: What is Cluster Sampling?
Cluster sampling is a technique used when it becomes difficult to study the target population spread across a wide area and simple random sampling cannot be applied. Cluster Sample is a probability sample where each sampling unit is a collection or cluster of elements.
For example, a researcher wants to survey the academic performance of high school students in Japan. He can divide the entire population of Japan into different clusters (cities). Then the researcher selects a number of clusters depending on his research through simple or systematic random sampling.
Q32: What is Systematic Sampling?
Systematic sampling is a statistical technique where elements are selected from an ordered sampling frame. In systematic sampling, the list is progressed in a circular manner so once you reach the end of the list, it is progressed from the top again. The best example of systematic sampling is equal probability method.
Q33: What are Eigenvectors and Eigenvalues?
Eigenvectors are used for understanding linear transformations. In data analysis, we usually calculate the eigenvectors for a correlation or covariance matrix. Eigenvectors are the directions along which a particular linear transformation acts by flipping, compressing or stretching.
Eigenvalue can be referred to as the strength of the transformation in the direction of eigenvector or the factor by which the compression occurs.
Q34: Examples where a false positive is important than a false negative?
Let us first understand what false positives and false negatives are:
- False Positives are the cases where you wrongly classified a non-event as an event a.k.a Type I error
- False Negatives are the cases where you wrongly classify events as non-events, a.k.a Type II error.
Example 1: In the medical field, assume you have to give chemotherapy to patients. Assume a patient comes to that hospital and he is tested positive for cancer, based on the lab prediction but he actually doesn’t have cancer. This is a case of false positive. Here it is of utmost danger to start chemotherapy on this patient when he actually does not have cancer. In the absence of cancerous cell, chemotherapy will do certain damage to his normal healthy cells and might lead to severe diseases, even cancer.
Example 2: Let’s say an e-commerce company decided to give $1000 Gift voucher to the customers whom they assume to purchase at least $10,000 worth of items. They send free voucher mail directly to 100 customers without any minimum purchase condition because they assume to make at least 20% profit on sold items above $10,000. Now the issue is if we send the $1000 gift vouchers to customers who have not actually purchased anything but are marked as having made $10,000 worth of purchase
Q35: Examples where a false negative important than a false positive? And vice versa?
Example 1 FN: What if Jury or judge decides to make a criminal go free?
Example 2 FN: Fraud detection.
Example 3 FP: customer voucher use promo evaluation: if many used it and actually if was not true, promo sucks
Q36: Examples where both false positive and false negatives are equally important?
In the Banking industry giving loans is the primary source of making money but at the same time if your repayment rate is not good you will not make any profit, rather you will risk huge losses.
Banks don’t want to lose good customers and at the same point in time, they don’t want to acquire bad customers. In this scenario, both the false positives and false negatives become very important to measure.
Q37: Difference between a Validation Set and a Test Set?
A Training Set:
• to fit the parameters i.e. weights
A Validation set:
• part of the training set
• for parameter selection
• to avoid overfitting
A Test set:
• for testing or evaluating the performance of a trained machine learning model, i.e. evaluating the
predictive power and generalization.
Q38: What is cross-validation?
Reference: k-fold cross validation
Cross-validation is a resampling procedure used to evaluate machine learning models on a limited data sample. The procedure has a single parameter called k that refers to the number of groups that a given data sample is to be split into. As such, the procedure is often called k-fold cross-validation. When a specific value for k is chosen, it may be used in place of k in the reference to the model, such as k=10 becoming 10-fold cross-validation. Mainly used in backgrounds where the objective is forecast, and one wants to estimate how accurately a model will accomplish in practice.
Cross-validation is primarily used in applied machine learning to estimate the skill of a machine learning model on unseen data. That is, to use a limited sample in order to estimate how the model is expected to perform in general when used to make predictions on data not used during the training of the model.
It is a popular method because it is simple to understand and because it generally results in a less biased or less optimistic estimate of the model skill than other methods, such as a simple train/test split.
The general procedure is as follows:
1. Shuffle the dataset randomly.
2. Split the dataset into k groups
3. For each unique group:
a. Take the group as a hold out or test data set
b. Take the remaining groups as a training data set
c. Fit a model on the training set and evaluate it on the test set
d. Retain the evaluation score and discard the model
4. Summarize the skill of the model using the sample of model evaluation scores
There is an alternative in Scikit-Learn called Stratified k fold, in which the split is shuffled to make it sure you have a representative sample of each class and a k fold in which you may not have the assurance of it (not good with a very unbalanced dataset).
Q39: What is Machine Learning?
Machine learning is the study of computer algorithms that improve automatically through experience. It is seen as a subset of artificial intelligence. Machine Learning explores the study and construction of algorithms that can learn from and make predictions on data. You select a model to train and then manually perform feature extraction. Used to devise complex models and algorithms that lend themselves to a prediction which in commercial use is known as predictive analytics.
Q40: What is Supervised Learning?
Supervised learning is the machine learning task of inferring a function from labeled training data. The training data consist of a set of training examples.
Algorithms: Support Vector Machines, Regression, Naive Bayes, Decision Trees, K-nearest Neighbor Algorithm and Neural Networks
Example: If you built a fruit classifier, the labels will be “this is an orange, this is an apple and this is a banana”, based on showing the classifier examples of apples, oranges and bananas.
Q41: What is Unsupervised learning?
Unsupervised learning is a type of machine learning algorithm used to draw inferences from datasets consisting of input data without labelled responses.
Algorithms: Clustering, Anomaly Detection, Neural Networks and Latent Variable Models
Example: In the same example, a fruit clustering will categorize as “fruits with soft skin and lots of dimples”, “fruits with shiny hard skin” and “elongated yellow fruits”.
Q42: What are the various Machine Learning algorithms?
Q43: What is “Naive” in a Naive Bayes?
Reference: Naive Bayes Classifier on Wikipedia
Naive Bayes methods are a set of supervised learning algorithms based on applying Bayes’ theorem with the “naive” assumption of conditional independence between every pair of features given the value of the class variable. Bayes’ theorem states the following relationship, given class variable y and dependent feature vector X1through Xn:
Q44: What is PCA (Principal Component Analysis)? When do you use it?
Reference: PCA on wikipedia
Principal component analysis (PCA) is a statistical method used in Machine Learning. It consists in projecting data in a higher dimensional space into a lower dimensional space by maximizing the variance of each dimension.
The process works as following. We define a matrix A with > rows (the single observations of a dataset – in a tabular format, each single row) and @ columns, our features. For this matrix we construct a variable space with as many dimensions as there are features. Each feature represents one coordinate axis. For each feature, the length has been standardized according to a scaling criterion, normally by scaling to unit variance. It is determinant to scale the features to a common scale, otherwise the features with a greater magnitude will weigh more in determining the principal components