Option strategies are the simultaneous, and often mixed, buying or selling of one or more options that differ in one or more of the options’ variables. Call options, simply known as calls, give the buyer a right to buy a particular stock at that option’s strike price. Conversely, put options, simply known as puts, give the buyer the right to sell a particular stock at the option’s strike price. This is often done to gain exposure to a specific type of opportunity or risk while eliminating other risks as part of a trading strategy. A very straightforward strategy might simply be the buying or selling of a single option; however, option strategies often refer to a combination of simultaneous buying and or selling of options.
Options strategies allow traders to profit from movements in the underlying assets based on market sentiment (i.e., bullish, bearish or neutral). In the case of neutral strategies, they can be further classified into those that are bullish on volatility, measured by the lowercase Greek letter sigma (σ), and those that are bearish on volatility. Traders can also profit off time decay, measured by the uppercase Greek letter theta (Θ), when the stock market has low volatility. The option positions used can be long and/or short positions in calls and puts.
Thou shall always take 100% daily gains or 200% all time gains.
Do not fall into temptation and buy during the first 30 minutes of market open. (Selling positions is still permitted)
Thou shall not buy calls on green days.
Thou shall not buy puts on red days.
Avoid greed and do not buy consecutive options on 1 company.
Give thyself at least 3 weeks time to play the option.
End your suffering and sell if down 50% all time on an option play.
Avoid gluttony and do not day trade options. (Swing trades allowed)
Be fruitful, multiply earnings and sell covered calls if holding any.
Celebrate and binge drink after big gains (or losses)
Off topic, but relevant – You absolutely need to be doing a 401k or IRA as well as investing in crypto: 401ks and IRAs offer fantastic tax advantages that straight investing does not. Also if you have an employer who matches you are leaving money on the table by not taking advantage of that. It’s foolish. Crypto is great and should definitely be in your portfolio but it should not be your whole portfolio. Sources: 1- WallStreetBets 2- Wikipedia
Banks gave out more paper money than they had gold in the vault. They ran “fractional reserves”. A real money maker. But every now and then, banks collapsed because of runs on the bank.
Central banking was invented. Central banks would be lenders of last resort. Runs on the bank were thus mitigated by banks guaranteeing each other’s deposits through a central bank. The risk of a bank run was not lowered. Its frequency was diminished and its impact was increased. After all, banks remained basically insolvent in this fractional reserve scheme.
Banks would still get in trouble. But now, if one bank got in sufficient trouble, they would all be in trouble at the same time. Governments would have to step in to save them.
All ties between the financial system and gold were severed in 1971 when Nixon decided that the USD would no longer be exchangeable for a fixed amount of gold. This exacerbated the problem, because there was now effectively no limit anymore on the amount of paper money that banks could create.
From this moment on, all money was created as credit. Money ceased to be supported by an asset. When you take out a loan, money is created and lent to you. Banks expect this freshly minted money to be returned to them with interest. Sure, banks need to keep adequate reserves. But these reserves basically consist of the same credit-based money. And reserves are much lower than the loans they make.
This led to an explosion in the money supply. The Federal Reserve stopped reporting M3 in 2006. But the ECB currently reports a yearly increase in the supply of the euro of about 5%.
This leads to a yearly increase in prices. The price increase is somewhat lower than the increase in the money supply. This is because of increased productivity. Society gets better at producing stuff cheaper all the time. So, in absence of money creation you would expect prices to drop every year. That they don’t is the effect of money creation.
What remains is an inflation rate in the 2% range.
Banks have discovered that they can siphon off all the productivity increase + 2% every year, without people complaining too much. They accomplish this currently by increasing the money supply by 5% per year, getting this money returned to them at an interest.
Apart from this insidious tax on society, banks take society hostage every couple of years. In case of a financial crisis, banks need bailouts or the system will collapse.
Apart from these problems, banks and governments are now striving to do away with cash. This would mean that no two free men would be able to exchange money without intermediation by a bank. If you believe that to transact with others is a fundamental right, this should scare you.
The absence of sound money was at the root of the problem. We were force-fed paper money because there were no good alternatives. Gold and silver remain difficult to use.
When it was tried to launch a private currency backed by precious metals (Liberty dollar), this initiative was shut down because it undermined the U.S. currency system. Apparently, a currency alternative could only thrive if “nobody” launched it and if they was no central point of failure.
What was needed was a peer-to-peer electronic cash system. This was what Satoshi Nakamoto described in 2008. It was a response to all the problems described above. That is why he labeled the genesis block with the text: “03/Jan/2009 Chancellor on brink of second bailout for banks.”. Bitcoin was meant to be an alternative to our current financial system.
So, if you find yourself religiously checking some cryptocurrency’s price, or bogged down in discussions about the “one true bitcoin”, or constantly asking what currency to buy, please at least remember that we have bigger fish to fry.
We are here to fix the financial system.
Given how early in the Rogers Adoption Curve for Crypto we are, I would like to take a moment so we can just imagine what this technological revolution, which I consider is the next huge step for human kind, could bring. I will emphasize some socioeconomic implications of descentralization, but I`m mostly interested in listening to, and debating your inputs.
Blockchain and Crypto Currency are here to change the world forever.
The implications of decentralization
As you may know one of the core proposals of blockchain is decentralization, and with it we can optimize so many processes that this alone could be the revolution we are talking about. By eliminating intermediaries, we can save on the cost they add to the supply chain ensuring those that create the value, keep it. Or we can simply save on fees.
To quote the man himself:
Whereas most technologies tend to automate workers on the periphery doing menial tasks, blockchains automate away the center. Instead of putting the taxi driver out of a job, blockchain puts Uber out of a job and lets the taxi drivers work with the customer directly. – Vitalik Buterin.
To put it simply, imagine that you replace Binance (a centralized company) with a robot. A robot that you have programed so well, whose code you publicly audit, and that is so safe you can trust it with billions of dollars in liquidity pools, so it proceeds to host and operate the trading platform by itself. In case you didn’t know, this is already a reality! Many people here trade on those platforms on a daily basis.
But this goes beyond replacing Centralized Exchanges with Automated Market Makers, Airbnb with a blockchain DApp that connects landlords and costumers, or even banks with complex smart contracts that allow you to borrow, save, tokenize physical assets, and so on. This goes way beyond.
Here is where I start to fantasize of the future. Think about replacing capital itself, think about getting rid of corporations. Lets dream of a world with DAOs massive adoption.
With DeFi, we may no longer need a company like Nestlé…
And specially not their investors. Of course, you will still need the people administrating, planning, monitoring, generating new ideas that adapt to their context, and creating innovative solutions for a complex world only humans can comprehend. But the figure of shareholders and CEOs that steal all the value that workers create and leave them with a tiny fraction of it, can disappear. This can be the basis of a once in a century transformation.
Just as an example: Nestlè’s coffee growers in Colombia keep less than 10% of the final sale price, and barely make a living on it, so are actually abandoning the rural areas.
With Blockchain, DeFi and Smart Contracts, people like you and me can collectively fund such an operation, and then agree upon specific terms like wages by direct democracy, voting with our crypto holdings. Then we would proceed to allocate funds, hire “developers” which would ultimately be regular office jobs that keep the organization functioning. Once in operation we would frequently vote on decisions and results, which would ultimately keep the highest level of accountability for people working in the organization. This is already happening by the way, this is how some blockchain projects work today. We just haven’t applied it to industrial and physical supply chains yet.
Let’s go back to our project to replace Nestle. Imagine that an organization’s main goal is not to maximize profits for shareholders and bonuses for CEOs anymore. Instead, it’s the interest of regular people and the company’s collaborators that drive its actions.
Most likely, you and I will want to consolidate an efficient and effective supply chain, that is sustainable and keeps the dignity and wellbeing of its collaborators as a guiding principle. We are not longer at their mercy on issues like climate change, we can now take immediate action against it, or stop endangering and hoarding water supplies in classic Nestle fashion.
Also, we are making profits, so we are redistributing capital, and improving our quality of life, which will be most notorious in the most vulnerable communities, usually those that extract/harvest/mine raw materials.
This is what could happen with the blockchain descentralization of business. And you could apply it to pretty much anything, but maybe initially it could be for low labor and capital intensive businesses.
I’ll give you another example. I work for a solar power multinational company. If you don’t know it, solar energy is essentially a financial product, most people working in these companies don’t care about the world, its simply that solar is a very safe and lucrative hustle, and all investors care about is having a nice return of investment (ROI). As of now, my company works exclusively for large scale corporate clients or the state itself, given that’s where the nice ROIs are, since they give you the projects that allow you to place large capitals at once. This means, as of today, we blatantly ignore the regular people that seek for our help and funding to power their farms and/or houses with solar energy. They’re not that profitable my boss tells me. This is shitty, and I’ve thought of quitting several times.
But back to the point. Now, imagine once again, we get rid of the institutional investors. Now you and me create Reddit Solar Co, a DAO. Our only purpose is to facilitate access to electricity to those without it, and to advance in the urban implementation of renewable energy. We help the world, make dividends that are automatically distributed by the DAO, and also our own Crypto is rising in value.
And this is not the best.
Let’s not forget of synergies.
So, we just created a DAO that manufactures and distributes food globally right? Or maybe Reddit Solar Co. As an organization born on the blockchain, we won’t have to adapt to the state of the art innovations on the crypto world like an old steam locomotive attempting to adapt a warp drive on top of it. We were born in space.
From the beginning, our Ethereum based DAO could adopt VeChain’s solution for supply chains, Cardano will help us to give an integral solution to the unbanked communities that provide our raw material, they now have IDs, access to DeFi and education. The land deeds and legal documents that relate to our enterprise are certified by LTO Network, we move money internationally with XRP or Stellar, and don’t worry, we use Polkadot to ensure proper blockchain interoperability.
Too complex for you? Don’t worry, you don’t even have to know or care about this, leave that to others. You’re into finance. Maybe sales is your thing and there’s a little Michael Scott in you. Or you`re into social work and want to supervise our community engagement at the start of the supply chain. Just go do your thing! You don’t necessarily have to be involved in all of this.
All you know is you do your job and receive your crypto salary.
Just as computers and the internet changed the world forever, and not only had economic implications but also changed our culture, routines, work lives and ways to interact with each other, crypto will. We are just so early; that all we can do for now is dream.
You’re having too much hope in humanity dude…
Sure, I may be making some optimistic assumptions on the motivations of humans, I may be saying that we will use this technology for good, and that we care about each other, and that’s one way to look at it. But we could also argue in favor of this from a sceptic perspective: even if you don’t care about the collective wellbeing of your community, it’s in your interest to live in a safer environment right? Ergo you want to reduce poverty. Its also in your interest to stop global warming so organized human life can continue to exist, or to make sure you and your children will have water and food in 50 years, that’s why you will want to use technology for good even if you only care about yourself. Also lets not forget the powerful incentive of profits. Crypto has the clear potential to achieve all of this.
Most of the current generation of crypto projects will be ready and operating within the next 3 years, so all we will need by then is the will to use this technology for good, and the vision to change the world.
This is just the beginning, we will be killing industries but giving birth to others we could have never imagined before.
Cons of Crypto: A coin called “Chia” is gobbling up 1,125,000 TB storage per day. Just to farm this token that no one seems to use. This takes resource wastage to a whole new level.
Chia is a coin that works on a proof of time space consensus. I.e. to farm this coin, one must allot dedicated hard drives and allot the space (known as plots), and get rewarded for it. Sounds good on paper, and one could even be tempted to think they may put that spare 500 GB space left and earn some passive income on it.
Except, this one already requires industrial grade storage space, just to farm a token that has almost zero adoption anywhere.
As you can see from this coin’s explorer, the storage is growing by almost 1000 PiB per day, in the last few days.
So a growth of 1000 PiB per day => almost 1125000 TB of storage per day is added onto this network, just to mine these coins. This equates to 1.1 million 1 TB drives added per day just to support farming on this network!
Pros of Crypto: – People in Hong Kong Use The Crypto and Blockchain To Fight Against Media Censorship Reference
Network effect and staying power BTC is the first virtual currency to solve the double-spending issue. The Bitcoin Protocol offered a solution to the Byzantine Generals’ Problem with a blockchain network structure, a notion first created by Stuart Haber and W. Scott Stornetta in 1991.
Bitcoin undoubtedly has a ‘brand’. It has perhaps the most substantial name recognition of any existing crypto asset and is basically synonymous with ‘cryptocurrency’ to the lay public.
Despite near constant proclamations of its demise, Bitcoin has not died. One could argue that – as the progenitor of cryptocurrencies – its longevity and continued profitability is itself an investment thesis.
As the number of public addresses, daily active users (DAU), and large holders/long term holders continue to trend upwards, it becomes harder and harder to ‘put the genie back in the bottle’:
Bitcoin’s valuation is well described by the most fundamental factor intrinsic to its network: the number of addresses that hold BTC. Applying Metcalfe’s law, the total value of Bitcoin’s network is well explained, with an R squared of 93.8%, simply by the square of its user base, n.
Store of value to hedge inflation
Over its lifetime, narratives of Bitcoin’s value have gone through several shifts, from the original cypherpunk vision in the white paper of p2p ‘e-cash’ to today’s ‘digital gold’ narrative.
One theme underlying both of these points, however, is a reaction to or distrust in the current financial system. This was true during the financial crisis of 2008 (see the genesis block message) and is still relevant today with unprecedented levels of monetary and fiscal stimulus being pursued by governments worldwide. Government deficits and central bank money printing may lead to inflation and thus drive investors towards assets like gold or Bitcoin to preserve their wealth.
This notion that BTC is a store of value to hedge inflation has certainly caught on in the last few years – not just from institutional or hedge fund investors, but from companies like MicroStrategy, Square and Tesla adding BTC to their balance sheets.
Like gold, BTC is scarce – only 21M will ever exist. It is estimated that 3M-3.7M BTC have been lost forever/will never enter circulating supply again.. One estimate is that 14.5M BTC are essentially illiquid.
To take one example, Grayscale’s BTC trust – which has no redemption process and thus effectively takes BTC out of circulation – alone holds over 600k BTC.
Like gold, BTC is also divisible, interchangeable and durable. Unlike gold, however, BTC is a digital asset and is thus easier to purchase, move and store.
If the store of value narrative endures, Bitcoin may have significant upside in supplanting a share of gold’s use case (estimated to be a $10T asset class).
Development
One of the common counterarguments for Bitcoin is that it is a ‘dinosaur’ with little technological improvement or development (as compared to its more innovative successors).
Segregated Witness (SegWit): a protocol upgrade proposal that went live in August 2017. This protocol upgrade effectively increased the number of transactions that can be stored in a single block, enabling the network to handle more transactions per second (TPS)
Lightning Network: is a second-layer micropayment solution for scalability
While other blockchains boast enterprise development, some companies are indeed building on Bitcoin. For example, Microsoft recently launched a Decentralized Identifier (DID) network (ION) on the Bitcoin mainnet
Ideological foundation for a potentially new financial system, without the old, decrepit, and corrupt banks and middle men.
The Environmental Argument is almost pointless, as it is the most efficient way of transporting millions of dollars around the world in mere seconds. And I mean efficient in all ways, there us no other single asset in the world capable of transporting this amount of capital wealth with such a low environmental impact or financial cost. If not, try moving 4 millions dollars of gold. Also, as Btc increases in value, this gets more on more efficient.
Innovation of the technology and the first mover advantage in capturing this new market’s value/future value. Btc will always be at the top as mainstream adoption continues relating Crypto=Bitcoin.
Ability to be bankless, with proven liquidity (thanks to Tesla) and with the best performing asset creation-to-date.
Inability of third parties to do anything about your Btc holding without the seed phrase. Government’s can hardly tax it if, as Michael Saylor put it: “I had a boating accident and forgot my seedphrase, I don’t have acces to my crypto anymore so I can’t be taxed”. In a way, nobody but yourself can prove that you still have access to those funds, so, can they truly be taxable?
The S2F model and updated S2F XA model. So far they have been scarily precise. Otherwise, Metcalfe’s law assures anyone that bitcoin may never go to 0, as the network is already strong enough to provide a certain degree of value.
Bitcoin has been around way too long, and to the uneducated it is the face of the crypto world.
Bitcoin has no smart contracts.
Bitcoin is slow.
Bitcoin fees are expensive.
People see it as an investment, not a currency they can use and spend. In the end this is not defined as it’s supposed to be used, but only as store of value. It’s at the state of gold, not of a coin.
Bitcoin has become outdated, the only thing it’s useful for is investing, day to day transactions are useless.
Bitcoin’s largest advantage and in fact it’s greatest disadvantage is that it’s the oldest cryptocurrency. Since then technology has evolved so much to become more energy and time efficient.
Bitcoin is like the grandpa of crypto and we should look at it as such. Admire it for its wisdom because it has taught us so much, but also acknowledge that each of its children are trying to make their own marks on the world.
It’s huge environmental impact due to its proof-of-work concept. BTC has a carbon footprint like Singapore, uses as much electrical energy as the Netherlands, and produces as much electronic waste as Luxembourg. This is a huge problem and needs to be accepted more widely.
It’s slow. with an average transaction time of like 10 minutes, we are pretty far from instant transactions – this might not be a problem in all cases, but is one when one would like to use it like a currency, as it was planned originally
High transaction costs – not ETH-high, but too high
Bitcoin takes a lot of energy to mine and use. As of May 2021, a single Bitcoin transaction takes as much energy as 760,201 VISA credit card payments (source). To keep this in context, the world banking system uses about two times as much energy as the Bitcoin network (source)
Bitcoin is difficult to mine. GPUs and CPUs don’t have enough computing power to compete with other miners, meaning so-called Application-Specific Integrated Chips (ASICs) are required. These are expensive – generally in the range of $1000 to $6000, depending on how new the model is (source). This restricts Bitcoin’s mining pool to people and groups who have enough wealth to invest in ASICs, which threatens the goal of keeping cryptocurrency decentralized.
Bitcoin transactions can take a long time to be confirmed. The average time for a transaction to confirmed once is 10 minutes (source), but for a payment to be absolutely final, it needs to be included in multiple blocks to ensure consensus in the mining pool. This takes even longer, sometimes up to one hour (source, for 6 confirmations).
Bitcoin transactions require expensive mining fees. At the moment, the average fee for a single transaction is $14.35, making Bitcoin unsuitable for day to day use (source).
Bitcoin lacks many features available in other coins, including smart contracts (programs run on and enforced by the blockchain, see here), anonymity (source), and CPU mining (allowing anyone with a CPU to mine, thus making the network more democratic and less susceptible to being taken over by large groups).
Crypto is definitely a good way to make money. However, you might end up finding the tech interesting. I know that I sure did, and having a sound understanding of your investment will make a big difference in your ability to hodl. It doesn’t have to be much, just a few YouTube videos.
Strategies when it comes to cryptocurrencies The HODL’er: you buy and basically you never sell. It’s kind of the holy grail of strategies when it comes to crypto according to this sub. Buy and forget and check back 10 years later. You’re a millionaire, Harry! No stress and no maintenance. You can even buy more over time and continue stacking your fat holdings. Do this if you believe in crypto long term
The Goal Setter: set a goal and sell when you reach that goal. Maybe it’s 3x and I’m out. Or maybe it’s make enough for student loans and I’m out. Or maybe it’s $1MM and sell half. Can be anything. Stress depends on your goal.
The Active Trader: Buy high and sell low
The Swing Trader: Some people are good at trading – they usually wait for those days where the whole market bleeds 20-30% in a day then they buy and wait for the bounce and they sell. Rinse and repeat. But they also risk missing out on the rocket jumps. But they also minimize the risk of being in the market when there’s a crash. In the end they might be able to increase their total holdings but for most beginners they lose rather than win. High stress and high maintenance.
The Cycle Trader: you DCA in during the bear market when everything has lost 80-90% of its ATH (alternatively, a year before the Bitcoin halving). Then you slowly sell off everything approximately a year after crypto starts trending up and enters a bull market. So this method has worked well for many people – they don’t necessarily time the top right but they continue to increase their holdings over several cycles. This might be the smart move if you have discipline. The risk is that history no longer repeats itself. It has worked the past 2 cycles but it’s not guaranteed it’ll work again. Medium stress, low maintenance
The Arbitrager: usually they have algos do the trading for them. They minimize risk and just arbitrage the price differences between exchanges. They might not care about crypto and just want to make money. They miss out on the bull run but also miss out on the bear market. Low stress, medium maintenance.
The Moon Chaser: 1000x or bust. Forget $10K eth or $100K btc, they want the next shiba or safe moon. They buy coins with market caps in the millions and hope for the pump to sell. This is like the lottery ticket buyers of crypto. High stress, high maintenance, smooth brain
–HODL, dont sell with a loss if you believe in your Coin long term.
–Stake, staking is really important! I cant tell you enough, if we are in a bear market and you can stake for a few years you can easily get 20-30% more coins then you have right now.
–DCA, keep buying. The bear market is where you DCA, dont stop buying. Right now is where you can get coins cheap! Just dont stop DCAing cause you are scared! Pick projects you believe in long term and keep buying at low prices!
–Get rid of coins you dont believe in long term, shitcoins. Many wont survive the bear market.
–Research coins for the next bull run!
Crypto Currency Market Cap Visualized during the Pandemic
Data science is an interdisciplinary field that uses scientific methods, processes, algorithms and systems to extract knowledge and insights from structured and unstructured data, and apply knowledge and actionable insights from data across a broad range of application domains.
In this blog, we are going to provide popular open source and public data sets, data visualization, data analytics and data lakes.
Latest complete Netflix movie dataset
Created from 4 APIs. 11K+ rows and 30+ attributes of Netflix (Ratings, earnings, actors, language, availability, movie trailers, and many more)
A corpus of web crawl data composed of over 50 billion web pages. The Common Crawl corpus contains petabytes of data collected since 2008. It contains raw web page data, extracted metadata and text extractions.
The University of Chicago Project on Security and Threats presents the updated and expanded Database on Suicide Attacks (DSAT), which now links to Uppsala Conflict Data Program data on armed conflicts and includes a new dataset measuring the alliance and rivalry relationships among militant groups with connections to suicide attack groups. Access it here.
The HRRR is a NOAA real-time 3-km resolution, hourly updated, cloud-resolving, convection-allowing atmospheric model, initialized by 3km grids with 3km radar assimilation. Radar data is assimilated in the HRRR every 15 min over a 1-h period adding further detail to that provided by the hourly data assimilation from the 13km radar-enhanced Rapid Refresh.
The GDC Data Portal is a robust data-driven platform that allows cancer researchers and bioinformaticians to search and download cancer data for analysis.
The Cancer Genome Atlas (TCGA), a collaboration between the National Cancer Institute (NCI) and National Human Genome Research Institute (NHGRI), aims to generate comprehensive, multi-dimensional maps of the key genomic changes in major types and subtypes of cancer.
The Therapeutically Applicable Research to Generate Effective Treatments (TARGET) program applies a comprehensive genomic approach to determine molecular changes that drive childhood cancers. The goal of the program is to use data to guide the development of effective, less toxic therapies. TARGET is organized into a collaborative network of disease-specific project teams. TARGET projects provide comprehensive molecular characterization to determine the genetic changes that drive the initiation and progression of childhood cancers. The dataset contains open Clinical Supplement, Biospecimen Supplement, RNA-Seq Gene Expression Quantification, miRNA-Seq Isoform Expression Quantification, miRNA-Seq miRNA Expression Quantification data from Genomic Data Commons (GDC), and open data from GDC Legacy Archive. Access it here.
The Genome Aggregation Database (gnomAD) is a resource developed by an international coalition of investigators that aggregates and harmonizes both exome and genome data from a wide range of large-scale human sequencing projects. The summary data provided here are released for the benefit of the wider scientific community without restriction on use. Downloads
Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable. Access it here.
The Pubmed Diabetes dataset consists of 19717 scientific publications from PubMed database pertaining to diabetes classified into one of three classes. The citation network consists of 44338 links. Each publication in the dataset is described by a TF/IDF weighted word vector from a dictionary which consists of 500 unique words. The README file in the dataset provides more details.
This dataset contains interactions between drugs and targets collected from DrugBank, KEGG Drug, DCDB, and Matador. It was originally collected by Perlman et al. It contains 315 drugs, 250 targets, 1,306 drug-target interactions, 5 types of drug-drug similarities, and 3 types of target-target similarities. Drug-drug similarities include Chemical-based, Ligand-based, Expression-based, Side-effect-based, and Annotation-based similarities. Target-target similarities include Sequence-based, Protein-protein interaction network-based, and Gene Ontology-based similarities. The original task on the dataset is to predict new interactions between drugs and targets based on different types of similarities in the network. Download link
PharmGKB data and knowledge is available as downloads. It is often critical to check with their curators at feedback@pharmgkb.org before embarking on a large project using these data, to be sure that the files and data they make available are being interpreted correctly. PharmGKB generally does NOT need to be a co-author on such analyses; They just want to make sure that there is a correct understanding of our data before lots of resources are spent.
The dataset contains open RNA-Seq Gene Expression Quantification data and controlled WGS/WXS/RNA-Seq Aligned Reads, WXS Annotated Somatic Mutation, WXS Raw Somatic Mutation, and RNA-Seq Splice Junction Quantification. Documentation
This dataset contains soil infrared spectral data and paired soil property reference measurements for georeferenced soil samples that were collected through the Africa Soil Information Service (AfSIS) project, which lasted from 2009 through 2018. Documentation
DAiSEE is the first multi-label video classification dataset comprising of 9068 video snippets captured from 112 users for recognizing the user affective states of boredom, confusion, engagement, and frustration “in the wild”. The dataset has four levels of labels namely – very low, low, high, and very high for each of the affective states, which are crowd annotated and correlated with a gold standard annotation created using a team of expert psychologists. Download it here.
NatureServe Explorer provides conservation status, taxonomy, distribution, and life history information for more than 95,000 plants and animals in the United States and Canada, and more than 10,000 vegetation communities and ecological systems in the Western Hemisphere.
The data available through NatureServe Explorer represents data managed in the NatureServe Central Databases. These databases are dynamic, being continually enhanced and refined through the input of hundreds of natural heritage program scientists and other collaborators. NatureServe Explorer is updated from these central databases to reflect information from new field surveys, the latest taxonomic treatments and other scientific publications, and new conservation status assessments. Explore Data here
FlightAware.com has data but you need to pay for a full dataset.
The anyflights package supplies a set of functions to generate air travel data (and data packages!) similar to nycflights13. With a user-defined year and airport, the anyflights function will grab data on:
flights: all flights that departed a given airport in a given year and month
weather: hourly meterological data for a given airport in a given year and month
airports: airport names, FAA codes, and locations
airlines: translation between two letter carrier (airline) codes and names
planes: construction information about each plane found in flights
The U.S. Department of Transportation’s (DOT) Bureau of Transportation Statistics (BTS) tracks the on-time performance of domestic flights operated by large air carriers. Summary information on the number of on-time, delayed, canceled and diverted flights appears in DOT’s monthly Air Travel Consumer Report, published about 30 days after the month’s end, as well as in summary tables posted on this website. BTS began collecting details on the causes of flight delays in June 2003. Summary statistics and raw data are made available to the public at the time the Air Travel Consumer Report is released. Access it here
Flightera.net seems to have a lot of good data for free. It has in-depth data on flights and doesn’t seem limited by date. I can’t speak on the validity of the data though.
flightradar24.com has lots of data, also historically, they might be willing to help you get it in a nice format.
Yahoo is shutting down in 2021. This is Yahoo Answers datasets (300MB gzip) that is fairly extensive from 2015 with about 1.4m rows. This dataset has the best questions answers, I mean all the answers, including the most insane awful answers and the worst questions people put together. Download it here.
Another option here: According to the tracker, there are 77M done, 20M out(?), and 40M to go:
Air Quality Data Collected at Outdoor Monitors Across the US. This is a BigQuery Dataset. There are no files to download, but you can query it through Kernels using the BigQuery API. The AQS Data Mart is a database containing all of the information from AQS. It has every measured value the EPA has collected via the national ambient air monitoring program. It also includes the associated aggregate values calculated by EPA (8-hour, daily, annual, etc.). The AQS Data Mart is a copy of AQS made once per week and made accessible to the public through web-based applications. The intended users of the Data Mart are air quality data analysts in the regulatory, academic, and health research communities. It is intended for those who need to download large volumes of detailed technical data stored at EPA and does not provide any interactive analytical tools. It serves as the back-end database for several Agency interactive tools that could not fully function without it: AirData, AirCompare, The Remote Sensing Information Gateway, the Map Monitoring Sites KML page, etc.
This list of a topic-centric public data sources in high quality. They are collected and tidied from blogs, answers, and user responses. Most of the data sets listed below are free, however, some are not.
StumbleUponAwesome – Discover random pages from the Awesome dataset using a browser extension.
Awesome CLI – A simple command-line tool to dive into Awesome lists.
Awesome Viewer – A visualizer for all of the above Awesome lists.
US Department of Education CRDC Dataset
The US Department of Ed has a dataset called the CRDC that collects data from all the public schools in the US and has demographic, academic, financial and all sorts of other fun data points. They also have corollary datasets that use the same identifier—an expansion pack if you may. It comes out every 2-3 years. Access it here
Nasa Dataset: sequencing data from bacteria before and after being taken to space
NASA has some sequencing data from bacteria before and after being taken to space, to look at genetic differences caused by lack of gravity, radiation and others. Very fun if you want to try your hand at some bio data science. Access it here.
Data is Plural is a really good newsletter published by Jeremy Singer-Vine. The datasets are very random, but super interesting. Access it here.
Global terrorism database
Huge list of terrorism incidents from inside the US and abroad. Each entry has date and location of the incident, motivations, whether people or property were lost, the size of the attack, type of attack, etc. Access it here
Terrorist Attacks Dataset: This dataset consists of 1293 terrorist attacks each assigned one of 6 labels indicating the type of the attack. Each attack is described by a 0/1-valued vector of attributes whose entries indicate the absence/presence of a feature. There are a total of 106 distinct features. The files in the dataset can be used to create two distinct graphs. The README file in the dataset provides more details. Download Link:
Terrorists: This dataset contains information about terrorists and their relationships. This dataset was designed for classification experiments aimed at classifying the relationships among terrorists. The dataset contains 851 relationships, each described by a 0/1-valued vector of attributes where each entry indicates the absence/presence of a feature. There are a total of 1224 distinct features. Each relationship can be assigned one or more labels out of a maximum of four labels making this dataset suitable for multi-label classification tasks. The README file provides more details. Download Link
The dolphin social network
This network dataset is in the category of Social Networks. A social network of bottlenose dolphins. The dataset contains a list of all of links, where a link represents frequent associations between dolphins. Access it here
Dataset of 200,000 jokes
There are about 208 000 jokes in this database scraped from three sources.
Decades of observations of birds all around the world, truly an impressive way to leverage citizen science. Access it here.
UFO Report Dataset
NUFORC geolocated and time standardized ufo reports for close to a century of data. 80,000 plus reports. Access it here
CDC’s Trend Drug Data
The CDC has a public database called NAMCS/NHAMCS that allows you to trend drug data. It has a lot of other data points so it can be used for a variety of other reasons. Access it here.
The Quick Draw Dataset is a collection of 50 million drawings across 345 categories, contributed by players of the game Quick, Draw!. The drawings were captured as timestamped vectors, tagged with metadata including what the player was asked to draw and in which country the player was located. Access it here.
The AirNow API replaces the previous AirNow Gateway web services. It includes file outputs and RSS data feeds. AirNow Gateway users can use their existing login information to access the new AirNow API web pages and web services. Access to the AirNow API is generally available to the public, and new accounts can be acquired via the Log In page
In computer science, Artificial Intelligence (AI) is intelligence demonstrated by machines. Its definition, AI research as the study of “intelligent agents”: any device that perceives its environment and takes actions that achieving its goals Russell et. al (2016).
Withal, Data Mining (DM) is the process of discovering patterns in data sets (or datasets) involving methods of machine learning, statistics, and database systems; DM focus on extract the information of datasets Han (2011).
This repository serves as a guide for anyone who wants to work with Artificial Intelligence or Data Mining applied in digital games! Here you will find a series of datasets, tools and materials available to build your application or dataset. Access it here.
The Squirrel Census is a multimedia science, design, and storytelling project focusing on the Eastern gray (Sciurus carolinensis). They count squirrels and present their findings to the public.
BigQuery public datasets are made available without any restrictions to all Google Cloud users. Google pays for the storage of these datasets. You can use them to learn how to work with BigQuery or even build your application on top of them, exactly as we’re going to do.
IMDb dataset importer – loads into a Marten DB document store. It imports the public datasets into a database, and provides repositories for querying. The total imported size is about 40 million rows, and 14 gigabytes on disk!
PHOnA: A Public Dataset of Measured Headphone Transfer Functions
A dataset of measured headphone transfer functions (HpTFs), the Princeton Headphone Open Archive (PHOnA), is presented. Extensive studies of HpTFs have been conducted for the past twenty years, each requiring a separate set of measurements, but this data has not yet been publicly shared. PHOnA aggregates HpTFs from different laboratories, including measurements for multiple different headphones, subjects, and repositionings of headphones for each subject. The dataset uses the spatially oriented format for acoustics (SOFA), and SOFA conventions are proposed for efficiently storing HpTFs. PHOnA is intended to provide a foundation for machine learning techniques applied to HpTF equalization. This shared data will allow optimization of equalization algorithms to provide more universal solutions to perceptually transparent headphone reproduction. Access it here.
In order to make all this data more accessible, they have converted everything into about 50 GB of CSV files with rows corresponding to “contact points” between the antibody and SARS-CoV-2 (or another antigen). Here’s a pastebin example of the contacts predicted between Matuzumab and SARS-CoV-2.
If you want to contribute to finding antibody treatments for COVID-19, these simulations can be used in data mining similar to the approach described in this paper.
They also recently created a separate mAB Kaggle dataset and wrote an introductory article for those who are interested in learning more about this field of research.
The World Mortality Dataset is a new repository that contains country-level data on all-cause mortality in 2015–2021 collected from various sources. It is maintained on a monthly basis and provides data for 79 countries. This is useful for tracking excess mortality across countries during the COVID-19 pandemic.
The Natural History Museum in London has 80 million items (and counting!) in its collections, from the tiniest specks of stardust to the largest animal that ever lived – the blue whale.
The Digital Collections Programme is a project to digitise these specimens and give the global scientific community access to unrivalled historical, geographic and taxonomic specimen data gathered in the last 250 years. Mobilising this data can facilitate research into some of the most pressing scientific and societal challenges.
Digitising involves creating a digital record of a specimen which can consist of all types of information such as images, and geographical and historical information about where and when a specimen was collected. The possibilities for digitisation are quite literally limitless – as technology evolves, so do possible uses and analyses of the collections. We are currently exploring how machine learning and automation can help us capture information from specimen images and their labels.
With such a wide variety of specimens, digitising looks different for every single collection. How we digitise a fly specimen on a microscope slide is very different to how we might digitise a bat in a spirit jar! We develop new workflows in response to the type of specimens we are dealing with. Sometimes we have to get really creative, and have even published on workflows which have involved using pieces of LEGO to hold specimens in place while we are imaging them.
Mobilising this data and making it open access is at the heart of the project. All of the specimen data is released on our Data Portal, and we also feed the data into international databases such as GBIF.
TSA Throughput Dataset (alternate source)
The TSA has is publishing more and more data via it’s Freedom of Information Act (FOIA) Reading Room. This project on github https://github.com/mikelor/tsathroughput contains the source for extracting the information from the .PDF files and converts them to JSON and CSV files.
The /data folder contains the source .PDFs going back to 2018 while the /data/raw/tsa/throughput folder contains .json files.
Measurements of the normal (i.e. non-superconducting) state magnetoresistance (change in resistance with magnetic field) in several single crystalline samples of copper-oxide high-temperature superconductors. The measurements were performed predominantly at the High Field Magnet Laboratory (HFML) in Nijmegen, the Netherlands, and the Pulsed Magnetic Field Facility (LNCMI-T) in Toulouse, France. Complete Zip Download
Collection of multimodal raw data captured from a manned all-terrain vehicle in the course of two realistic outdoor search and rescue (SAR) exercises for actual emergency responders conducted in Málaga (Spain) in 2018 and 2019: the UMA-SAR dataset. Full Dataset.
Child mortality numbers caused by malaria by country
Number of deaths of infants, neonatal, and children up to 4 years old caused by malaria by country from 2000 to 2015. Originator: World Health Organization
The dataset will give anyone the opportunity to train and test models of semantic equivalence, based on actual Quora data. 400,000 lines of potential question duplicate pairs. Each line contains IDs for each question in the pair, the full text for each question, and a binary value that indicates whether the line truly contains a duplicate pair. Access it here.
MIMIC Critical Care Database
MIMIC is an openly available dataset developed by the MIT Lab for Computational Physiology, comprising deidentified health data associated with ~60,000 intensive care unit admissions. It includes demographics, vital signs, laboratory tests, medications, and more. Access it here.
Data.Gov: The home of the U.S. Government’s open data
Here you will find data, tools, and resources to conduct research, develop web and mobile applications, design data visualizations, and more. Search over 280000 Datasets.
Art that does not attempt to represent an accurate depiction of a visual reality but instead use shapes, colours, forms and gestural marks to achieve its effect
5000+ classical abstract art here, real artists with annotation. You can download them in very high resolution, however you would have to crawl them first with this scraper.
Interactive map of indigenous people around the world
Native-Land.ca is a website run by the nonprofit organization Native Land Digital. Access it here.
Conducted by the Federal Highway Administration (FHWA), the NHTS is the authoritative source on the travel behavior of the American public. It is the only source of national data that allows one to analyze trends in personal and household travel. It includes daily non-commercial travel by all modes, including characteristics of the people traveling, their household, and their vehicles. Access it here.
Statistics and data about the National Travel Survey, based on a household survey to monitor trends in personal travel.
The survey collects information on how, why, when and where people travel as well as factors affecting travel (e.g. car availability and driving license holding).
NeTEx is the official format for public transport data in Norway and is the most complete in terms of available data. GTFS is a downstream format with only a limited subset of the total data, but we generate datasets for it anyway since GTFS can be easier to use and has a wider distribution among international public transport solutions. GTFS sets come in “extended” and “basic” versions. Access here.
A subset of the field data collected on temporary NFI plots can be downloaded in Excel format from this web site. The file includes a Read_me sheet and a sheet with field data from temporary plots on forest land1 collected from 2007 to 2019. Note that plots located on boundaries (for example boundaries between forest stands, or different land use classes) are not included in the dataset. The dataset is primarily intended to be used as reference data and validation data in remote sensing applications. It cannot be used to derive estimates of totals or mean values for a geographic area of any size. Download the dataset here
Large data sets from finance and economics applicable in related fields studying the human condition
CIA: The world Factbook provides basic intelligence on the history, people, government, economy, energy, geography, environment, communications, transportation, military, terrorism, and transnational issues for 266 world entities.
Consumer Price Index: The Consumer Price Index (CPI) is a measure of the average change over time in the prices paid by urban consumers for a market basket of consumer goods and services. Indexes are available for the U.S. and various geographic areas. Average price data for select utility, automotive fuel, and food items are also available.
International Historical Statistics is a compendium of national and international socio-economic data from 1750 to 2010. Data are available in both Excel and PDF tabular formats. IHS is structured in three broad geographical divisions and ten themes: Africa / Asia / Oceania; The Americas and Europe. The database is structured in ten categories: Population and vital statistics; Labour force; Agriculture; Industry; External trade; Transport and communications; Finance; Commodity prices; Education and National accounts. Access here
World Input-Output Tables and underlying data. World Input-Output Tables and underlying data, covering 43 countries, and a model for the rest of the world for the period 2000-2014. Data for 56 sectors are classified according to the International Standard Industrial Classification revision 4 (ISIC Rev. 4).
Data: Real and PPP-adjusted GDP in US millions of dollars, national accounts (household consumption, investment, government consumption, exports and imports), exchange rates and population figures.
COW seeks to facilitate the collection, dissemination, and use of accurate and reliable quantitative data in international relations. Key principles of the project include a commitment to standard scientific principles of replication, data reliability, documentation, review, and the transparency of data collection procedures
Data: Total national trade and bilateral trade flows between states. Total imports and exports of each country in current US millions of dollars and bilateral flows in current US millions of dollars
Geographical coverage: Single countries around the world
The WTO provides quantitative information in relation to economic and trade policy issues. Its data-bases and publications provide access to data on trade flows, tariffs, non-tariff measures (NTMs) and trade in value added.
The Subaru-Mitaka-Okayama-Kiso Archive, holds about 15 TB of astronomical data from facilities run by the National Astronomical Observatory of Japan. All data becomes publicly available after an embargo period of 12-24 months (to give the original observers time to publish their papers).
Graph Datasets
Web crawl graph with 3.5 billion web pages and 128 billion hyperlinks
Many web and social graphs with up to 95 billion edges. While this data collection seems to be very comprehensive, it is not trivially accessible without external tool.
The Multi-Domain Sentiment Dataset contains product reviews taken from Amazon.com from many product types (domains). Some domains (books and dvds) have hundreds of thousands of reviews. Others (musical instruments) have only a few hundred. Reviews contain star ratings (1 to 5 stars) that can be converted into binary labels if needed. Access it here.
Supported by Google Jigsaw, the GDELT Project monitors the world’s broadcast, print, and web news from nearly every corner of every country in over 100 languages and identifies the people, locations, organizations, themes, sources, emotions, counts, quotes, images and events driving our global society every second of every day, creating a free open platform for computing on the entire world.
This dataset represents a snapshot of the Yahoo! Music community’s preferences for various musical artists. The dataset contains over ten million ratings of musical artists given by Yahoo! Music users over the course of a one month period sometime prior to March 2004. Users are represented as meaningless anonymous numbers so that no identifying information is revealed. The dataset may be used by researchers to validate recommender systems or collaborative filtering algorithms. The dataset may serve as a testbed for matrix and graph algorithms including PCA and clustering algorithms. The size of this dataset is 423 MB.
This dataset contains a small sample of the Yahoo! Movies community’s preferences for various movies, rated on a scale from A+ to F. Users are represented as meaningless anonymous numbers so that no identifying information is revealed. The dataset also contains a large amount of descriptive information about many movies released prior to November 2003, including cast, crew, synopsis, genre, average ratings, awards, etc. The dataset may be used by researchers to validate recommender systems or collaborative filtering algorithms, including hybrid content and collaborative filtering algorithms. The dataset may serve as a testbed for relational learning and data mining algorithms as well as matrix and graph algorithms including PCA and clustering algorithms. The size of this dataset is 23 MB.
The dataset is a collection of 964 hours (22K videos) of news broadcast videos that appeared on Yahoo news website’s properties, e.g., World News, US News, Sports, Finance, and a mobile application during August 2017. The videos were either part of an article or displayed standalone in a news property. Many of the videos served in this platform lack important metadata, such as an exhaustive list of topics associated with the video. We label each of the videos in the dataset using a collection of 336 tags based on a news taxonomy designed by in-house editors. In the taxonomy, the closer the tag is to the root, the more generic (topically) it is.
The Internet Archive is making an 80 TB web crawl available for research
The TREC conference made the ClueWeb09 [3] dataset available a few years back. You’ll have to sign an agreement and pay a nontrivial fee (up to $610) to cover the sneakernet data transfer. The data is about 5 TB compressed.
ClueWeb12 is now available, as are the Freebase annotations, FACC1
CNetS at Indiana University makes a 2.5 TB click dataset available
ICWSM made a large corpus of blog posts available for their 2011 conference. You’ll have to register (an actual form, not an online form), but it’s free. It’s about 2.1 TB compressed. The dataset consists of over 386 million blog posts, news articles, classifieds, forum posts and social media content between January 13th and February 14th. It spans events such as the Tunisian revolution and the Egyptian protests (see http://en.wikipedia.org/wiki/January_2011 for a more detailed list of events spanning the dataset’s time period). Access it here
The Yahoo News Feed dataset is 1.5 TB compressed, 13.5 TB uncompressed
The Proteome Commons makes several large datasets available. The largest, the Personal Genome Project , is 1.1 TB in size. There are several others over 100 GB in size.
The MOBIO dataset is about 135 GB of video and audio data
The Yahoo! Webscope program makes several 1 GB+ datasets available to academic researchers, including an 83 GB data set of Flickr image features and the dataset used for the 2020 KDD Cup , from Yahoo! Music, which is a bit over 1 GB.
Freebase makes regular data dumps available. The largest is their Quad dump , which is about 3.6 GB compressed.
The Research and Innovative Technology Administration (RITA) has made available a dataset about the on-time performance of domestic flights operated by large carriers. The ASA compressed this dataset and makes it available for download.
The wiki-links data made available by Google is about 1.75 GB total.
Google Research released a large 24GB n-gram data set back in 2006 based on processing 10^12 words of text and published counts of all sequences up to 5 words in length.
These data are intended to be used by researchers and other professionals working in power and energy related areas and requiring data for design, development, test, and validation purposes. These data should not be used for commercial purposes.
A dataset and open-ended challenge for music recommendation research ( RecSys Challenge 2018). Sampled from the over 4 billion public playlists on Spotify, this dataset of 1 million playlists consist of over 2 million unique tracks by nearly 300,000 artists, and represents the largest public dataset of music playlists in the world. Access it here
The National Health and Nutrition Examination Survey (NHANES) is conducted every two years by the National Center for Health Statistics and funded by the Centers for Disease Control and Prevention. The survey measures obesity rates among people ages 2 and older. Find the latest national data and trends over time, including by age group, sex, and race. Data are available through 2017-2018, with the exception of obesity rates for children by race, which are available through 2015-2016. Access here
NCEI first developed the Global Historical Climatology Network-Monthly (GHCN-M) temperature dataset in the early 1990s. Subsequent iterations include version 2 in 1997, version 3 in May 2011, and version 4 in October 2018.
Are there any places where the climate is recently getting colder?
Human development index (HDI) by world subdivisions
The Human Development Index (HDI) is a statistic composite index of life expectancy, education (mean years of schooling completed and expected years of schooling upon entering the education system), and per capita income indicators, which are used to rank countries into four tiers of human development.
The data comes from the Global Power Plant Database. The Global Power Plant Database is a comprehensive, open source database of power plants around the world. It centralizes power plant data to make it easier to navigate, compare and draw insights for one’s own analysis. The database covers approximately 30,000 power plants from 164 countries and includes thermal plants (e.g. coal, gas, oil, nuclear, biomass, waste, geothermal) and renewables (e.g. hydro, wind, solar). Each power plant is geolocated and entries contain information on plant capacity, generation, ownership, and fuel type. It will be continuously updated as data becomes available.
The ImageNet dataset contains 14,197,122 annotated images according to the WordNet hierarchy. Since 2010 the dataset is used in the ImageNet Large Scale Visual Recognition Challenge (ILSVRC), a benchmark in image classification and object detection. The publicly released dataset contains a set of manually annotated training images.
The MNIST database of handwritten digits, available from this page, has a training set of 60,000 examples, and a test set of 10,000 examples. It is a subset of a larger set available from NIST. The digits have been size-normalized and centered in a fixed-size image.
It is a good database for people who want to try learning techniques and pattern recognition methods on real-world data while spending minimal efforts on preprocessing and formatting. Access it here.
MMID is a large-scale, massively multilingual dataset of images paired with the words they represent collected at the University of Pennsylvania. The dataset is doubly parallel: for each language, words are stored parallel to images that represent the word, and parallel to the word’s translation into English (and corresponding images.) . Dcumentation.
HDI is calculated by the UN every year to measure a country’s development using average life expectancy, education level, and gross national income per capita (PPP). The EU has a collective HDI of 0.911.
Using machine learning methods to group NFL quarterbacks into archetypes
Data Source:
Data collected from a series of rushing and passing statistics for NFL Quarterbacks from 2015-2020 and performed a machine learning algorithm called clustering, which automatically sorts observations into groups based on shared common characteristics using a mathematical “distance metric.”
The idea was to use machine learning to determine NFL Quarterback Archetype to agnostically determine which quarterbacks were truly “mobile” quarterbacks, and which were “pocket passers” that relied more on passing. I used a number of metrics in my actual clustering analysis, but they can be effectively summarized across two dimensions: passing and rushing, which can be further roughly summarized across two metrics: passer rating and rushing yards per year. Plotting the quarterbacks along these dimensions and plotting the groups chosen by the clustering methodology shows how cleanly the methodology selected the groups.
Read this blog article on the process for more information if you’re interested, or just check out this blog in general if you found this interesting!
Intraday Stock Data (1 min) – S&P 500 – 2008-21: 12 years of 1 minute bars for data science / machine learning.
Granular stock bar data for research is difficult to find and expensive to buy. The author has compiled this library from a variety of sources and is making it available for free.
One compressed CSV file with 9 columns and 2.07 million rows worth of 1 minute SPY bars. Access it here
Datasets: A live version of the vaccination dataset and documentation are available in a public GitHub repository here. These data can be downloaded in CSV and JSON formats. PDF.
Learn how to create, maintain, and contribute to a long-living dataset that will update itself automatically across projects, using git and DVC as versioning systems, and DAGsHub as a host for the datasets.
Courtesy of Google’s Project Sunroof: This dataset essentially describes the rooftop solar potential for different regions, based on Google’s analysis of Google Maps data to find rooftops where solar would work, and aggregate those into region-wide statistics.
It comes in a couple of aggregation flavors – by census tract , where the region name is the census tract id, and by postal code , where the name is the postal code. Each also contains latitude/longitude bounding boxes and averages, so that you can download based on that, and you should be able to do custom larger aggregations using those, if you’d like.
A large dataset aimed at teaching AI to code, it consists of some 14M code samples and about 500M lines of code in more than 55 different programming languages, from modern ones like C++, Java, Python, and Go to legacy languages like COBOL, Pascal, and FORTRAN.
When the whole country is double vaccinated, the value will be 200 doses per 100 population. At the moment the UK is like 85, which is because ~70% of the population has had at least one dose and ~15% of the population (which is a subset of that 70%) have had two. Hence ~30% are currently unprotected – myself included until Sunday.
According to the author of the source data: “For the 1918 Spanish Flu, the data was collected by knowing that the total counts were 500M cases and 50M deaths, and then taking a fraction of that per day based on the area of this graph image:” – the graph is used is here:
Visualización y conjunto de datos de comparación de enfermedades agregadas
Data source: https://trends.google.com Trending topics from 2010 to 2019 were taken from Google’s annual Year in Search summary 2010-2029
The full, ~11 minute video covering the whole 2010s decade is available here at https://youtu.be/xm91jBeN4oo
Google Trends provides weekly relative search interest for every search term, along with the interest by state. Using these two datasets for each term, we’re able to calculate the relative search interest for every state for a particular week. Linear interpolation was used to calculate the daily search interest.
From the author: I started with data on roads from naturalearth.com, which also includes some ferry lines. I then calculated the fastest routes (assuming a speed of 90 km/h on roads, and 35 km/h on boat) between each pair of 45 European capitals. The animation visualizes these routes, with brighter lines for roads that are more frequently “traveled”.
In reality these are of course not the most traveled roads, since people don’t go from all capitals to all other capitals in equal measure. But I thought it would be fun to visualize all the possible connections.
The model is also very simple, and does not take into account varying speed limits, road conditions, congestion, border checks and so on. It is just for fun!
In order to keep the file size manageable, the animation only shows every tenth frame.
Is Russia, Turkey or country X really part of Europe? That of course depends on the definition, but it was more fun to include them than to exclude them! The Vatican is however not included since it would just be the same as the Rome routes. And, unfortunately, Nicosia on Cyprus is not included to due an error on my behalf. It should be!
Dataset of all 825 Pokemon (this includes Alolan Forms). It would be preferable if there are at least 100 images of each individual Pokemon.
https://github.com/veekun/pokedex: This is a Python library slash pile of data containing a whole lot of data scraped from Pokémon games. It’s the primary guts of veekun.
2) This dataset comprises of more than 800 pokemons belonging up to 8 generations.
Using this dataset have been fun for me. I used it to create a mosaic of pokemons taking image as reference. You can find it here and it’s free to use: Couple Mosaic (powered by Pokemons)
Here is the data type information in the file:
Name: Pokemon Name
Type: Type of Pokemon like Grass / Fire / Water etc..,.
ETL pipeline for Facebook’s research project to provide detailed large-scale demographics data. It’s broken down in roughly 30×30 m grid cells and provides info on groups by age and gender.
The GISS Surface Temperature Analysis ver. 4 (GISTEMP v4) is an estimate of global surface temperature change. Graphs and tables are updated around the middle of every month using current data files from NOAA GHCN v4 (meteorological stations) and ERSST v5 (ocean areas), combined as described in our publications Hansen et al. (2010) and Lenssen et al. (2019).
Buying a chocolate bar? There are seemingly hundreds to choose from, but its just the illusion of choice. They pretty much all come from Mars, Nestlé, or Mondelēz (which owns Cadbury).
Criteria for choosing a dictionary: – No proper nouns – “Official” source if available – Inclusion of inflected forms – Among two lists, the largest was fancied – No or very rare abbreviations if possible- but hard to detect in unknown languages and across hundreds of thousands of words.
The author found this dataset in a more accessible format upon searching for the keyword “CDPB” (Carcinogenic Potency Database) in the National Library of Medicine Catalog. Check out this parent website for the data source and dataset description. The dataset referenced in OP’s post concerns liver specific carcinogens, which are marked by the “liv” keyword as described in the dataset description’s Tissue Codes section.
In this project, the authors have designed a spatial model which is able to classify urbanity levels globally and with high granularity. As the target geographic support for our model we selected the quadkey grid in level 15, which has cells of approximately 1x1km at the equator.
The author obtained the data from the UK Government website, so unfortunately don’t know the methodology or how they collected the data etc.
The comparison to the general public is a great idea – according to the Government site, 6% of children, 16% of working-age adults and 45% of Pension-age adults are disabled.
According to the author (https://www.reddit.com/user/newpua_bie/) , this animation depicts adult cognitive skills, as measured by the PIAAC study by OECD. Here, the numeracy and literacy skills have been combined into one. Each frame of the animation shows the xth percentile skill level of each individual country. Thus, you can see which countries have the highest and lowest scores among their bottom performers, median performers, and top performers. So for example, you can see that when the bottom 1st percentile of each country is ranked, Japan is at the top, Russia is second, etc. Looking at the 50th percentile (median) of each country, Japan is top, then Finland, etc.
Programme for the International Assessment of Adult Competencies (PIAAC)is a study by OECD to measure measured literacy, numeracy, and “problem-solving in technology-rich environments” skills for people ages 16 and up. For those of you who are familiar with the school-age children PISA study, this is essentially an adult version of it.
The model was built in Stan and was inspired by Andrew Gelman’s World Cup model shown here. These plots show posterior probabilities that the team on the y axis will score more goals than the team on the x axis. There is some redundancy of information here (because if I know P(England beats Scotland) then I know P(Scotland beats England) )
SEDE (Stack Exchange Data Explorer) is a dataset comprised of 12,023 complex and diverse SQL queries and their natural language titles and descriptions, written by real users of the Stack Exchange Data Explorer out of a natural interaction. These pairs contain a variety of real-world challenges which were rarely reflected so far in any other semantic parsing dataset. Access it here
Each map size is proportional to population, so China takes up about 18-19% of the map space.
Countries with very far-flung territories, such as France (or the USA) will have their maps shrunk to fit all territories. So it is the size of the map rectangle that is proportional to population, not the colored area. Made in R, using data from naturalearthdata.com. Maps drawn with the tmap package, and placed in the image with the gridExtra package. Map colors from the wesanderson package.
Beneath adds some useful features for shared data, like the ability to run SQL queries, sync changes in real-time, a Python integration, and monitoring. The monitoring is really useful as it lets you check out the write activity of the scraper (no surprise, WSB is most active when markets are open
The scraper (which uses Async PRAW) is open source here
The chart shows the average daily gain in $ if $100 were invested at a date on x-axis. Total gain was divided by the number of days between the day of investing and June 13, 2021. Gains were calculated on average 30-day prices.
Time range: from March 28, 2013, till June 13, 2021
Google Playstore dataset is now available with double the data (2.3 Million) android application data and a new attribute stating the scraped date time in Kaggle.
According to the author: Looking at non-suicide firearms deaths by state (2019), and then grouping by the Guns to Carry rating (1-5 stars), it seems that stricter gun laws are correlated with fewer firearms homicides. Guns to Carry rates states based on “Gun friendliness” with 1 star being least friendly (California, for example), and 5 stars being most friendly (Wyoming, for example). The ratings aren’t perfect but they include considerations like: Permit required, Registration, Open carry, and Background checks to come up with a rating.
The numbers at the bottom are the average non-suicide deaths calculated within the rating group. Each bar shows the number for the individual state.
Interesting that DC is through the roof despite having strict laws. On the flip side, Maine is very friendly towards gun owners and has a very low homicide rate, despite having the highest ratio of suicides to homicides.
Obviously, lots of things to consider and this is merely a correlation at a basic level. This is a topic that interested me so I figured I’d share my findings. Not attempting to make a policy statement or anything.
Data for word frequency in econ textbooks was compiled by myself by scraping words from 43 undergraduate economics textbooks. For details see Deconstructing Econospeak.
Data Source: from eMarketer, as quoted byJon Erlichman
Purpose according to the author: raw textual numbers (like in the original tweet) are hard to compare, particularly the acceleration or deceleration of a trend. Did for myself, but maybe is useful to somebody.
A few things to notice: It’s dangerous to be a newborn. The same mortality rates are reached again only in the fifties. However, mortality drops after birth very quickly and the safest age is about ten years old. After experiencing mortality jump in puberty – especially high for boys, mortality increases mostly exponentially with age. Every thirty years of life increase chances of dying about ten times. At 80, chance of dying in a year is about 5.8% for males and 4.3% for females. This mortality difference holds for all ages. The largest disparity is at about twenty three years old when males die at a rate about 2.7 times higher than females.
Data from:iposcoop.com From the author u/nobjos: The full article on the above analysis can be found here I have sub r/market_sentiment where I do a comprehensive deep-dive on one investment strategy/topic every week! Some of the author popular articles are a. Performance of Jim Cramer’s stock picks b. Performance of buy and sell recommendations made by financial analysts in the last decade c. Benchmarking performance of Motely fool against SP500 Funko IPO is considered to have the worst first-day return for an IPO in the last two decades. Out of the top 10 list, only 3 Investment banks had below-average returns. On average, IPOs did make money for the investor. But the amount is significantly different if you got allocated the IPO at offer price vs you get the IPO at market price. Baidu.com made a whopping 354% on its listing day. Another interesting observation is 6 out of 10 companies in the list were listed in 200 (just before the dot com crash)
Check out the FAS site for notes and caveats about their estimates. Governments don’t just print this stuff on their websites. These are evidence-based estimates of tightly-guarded national secrets.
Of particular note – Here’s what the FAS says about North Korea: “After six nuclear tests, including two of 10-20 kilotons and one of more than 150 kilotons, we estimate that North Korea might have produced sufficient fissile material for roughly 40-50 warheads. The number of assembled warheads is unknown, but lower. While we estimate North Korea might have a small number of assembled warheads for medium-range missiles, we have not yet seen evidence that it has developed a functioning warhead that can be delivered at ICBM range.”
The author used several sources for this video and article. The first, for the video, is GitHub Archive & CodersRank. For the analysis of the OSCI index data, the author used https://opensourceindex.io/.
2021 is straight projections, must be taken with a grain of salt. However, the assumption of continuous rise of murder rate is not a bad one based on recent news reports, such as: here
This image was generated for my research mapping the privacy research field. The visual is a combination of network visualisation and manual adding of the labels.
The data was gathered from Scopus, a high-quality academic publication database, and the visualisation was created with Gephi. The initial dataset held ~120k publications and over 3 million references, from which we selected only the papers and references in the field.
The labels were assigned by manually identifying clusters and two independent raters assigning names from a random sample of publications, with a 94% match between raters.
This is a randomized experiment the author conducted with 450 people on Amazon MTurk. Each person was randomly assigned to one of three writing activities in which they either (a) described their phone, (b) described what they’d do if they received a call from someone they know, or (c) describe what they’d do if they received a call from an unknown number. Pictures of an iPhone with a corresponding call screen were displayed above the text box (blank, “Incoming Call,” or “Unknown”). Participants then rated their anxiety on a 1-4 scale.
The Cloud Education Certification App is an EduFlix App for AWS, Azure, Google Cloud Certification Prep[Android, iOS]
Technology is changing and is moving towards the cloud. The cloud will power most businesses in the coming years and is not taught in schools. How do we ensure that our kids and youth and ourselves are best prepared for this challenge?
Building mobile educational apps that work offline and on any device can help greatly in that sense.
The ability to tab on a button and learn the cloud fundamentals and take quizzes is a great opportunity to help our children and youth to boost their job prospects and be more productive at work.
The App covers the following certifications : AWS Cloud Practitioner Exam Prep CCP CLF-C01, Azure Fundamentals AZ 900 Exam Prep, AWS Certified Solution Architect Associate SAA-C02 Exam Prep, AWS Certified Developer Associate DVA-C01 Exam Prep, Azure Administrator AZ 104 Exam Prep, Google Associate Cloud Engineer Exam Prep, Data Analytics for AWS DAS-C01, Machine Learning for AWS and Google, AWS Certified Security – Specialty (SCS-C01), AWS Certified Machine Learning – Specialty (MLS-C01), Google Cloud Professional Machine Learning Engineer and more… [Android, iOS]
Features: – Practice exams – 1000+ Q&A updated frequently. – 3+ Practice exams per Certification – Scorecard / Scoreboard to track your progress – Quizzes with score tracking, progress bar, countdown timer. – Can only see scoreboard after completing the quiz. – FAQs for most popular Cloud services – Cheat Sheets – Flashcards – works offline
Note and disclaimer: We are not affiliated with AWS, Azure, Microsoft or Google. The questions are put together based on the certification study guide and materials available online. The questions in this app should help you pass the exam but it is not guaranteed. We are not responsible for any exam you did not pass.
Important: To succeed with the real exam, do not memorize the answers in this app. It is very important that you understand why a question is right or wrong and the concepts behind it by carefully reading the reference documents in the answers.
This blog is about Clever Questions, Answers, Resources, Links, Discussions, Tips about jobs and careers at FAANGM companies: Facebook, Apple, Amazon, AWS, Netflix, Google, Microsoft, Linkedin
How to prepare for FAANGM jobs interviews
You must be able to write code. It is as simple as that. Prepare for the interview by practicing coding exercises in different categories. You’ll solve one or more coding problems focused on CS fundamentals like algorithms, data structures, recursions, and binary trees.
Coding Interview Tips These tips from FAANGM engineers can help you do your best.
Make sure you understand the question. Read it back to your interviewer. Be sure to ask any clarifying questions.
An interview is a two-way conversation; feel free to be the one to ask questions, too.
Don’t rush. Take some time to consider your approach. For example, on a tree question, you’ll need to choose between an iterative or a recursive approach. It’s OK to first use a working, unoptimized solution that you can iterate on later.
Talk through your thinking and processes out loud. This can feel unnatural; be sure to practice it before the interview.
Test your code by running through your problem with a few test and edge cases. Again, talk through your logic out loud when you walk through your test cases.
Think of how your solution could be better, and try to improve it. When you’ve finished, your interviewer will ask you to analyze the complexity of the code in Big O notation.
Walk through your code line by line and assign a complexity to each line.
Remember how to analyze how “good” your solution is: how long does it take for your solution to complete? Watch this video to get familiar with Big O Notation.
• Ask clarifying questions. Talk through the problem and ask follow-up questions to make sure you understand the exact problem you’re trying to solve before you jump into building the solution. • Let the interviewer know if you’ve seen the problem previously. That will help us understand your context. • Present multiple potential solutions, if possible. Talk through which solution you’re choosing and why.
While you code
• Don’t forget to talk! While your tech screen will focus heavily on coding, the engineer you’re interviewing with will also be evaluating your thought process. Explaining your decisions and actions as you go will help the interviewer understand your choices. • Be flexible. Some problems have elegant solutions, and some must be brute forced. If you get stuck, just describe your best approach and ask the interviewer if you should go that route. It’s much better to have non-optimal but working code than just an idea with nothing written down. • Iterate rather than immediately trying to jump to the clever solution. If you can’t explain your concept clearly in five minutes, it’s probably too complex. • Consider (and be prepared to talk about): • Different algorithms and algorithmic techniques, such as sorting, divide-and-conquer, recursion, etc. • Data structures, particularly those used most often (array, stack/queue, hashset/hashmap/hashtable/dictionary, tree/binary tree, heap, graph, etc.) • O memory constraints on the complexity of the algorithm you’re writing and its running time as expressed by big-O notation. • Generally, avoid solutions with lots of edge cases or huge if/else if/else blocks, in most cases. Deciding between iteration and recursion can be an important step
After you code
• Expect questions. The interviewer may tweak the problem a bit to test your knowledge and see if you can come up with another answer and/or further optimize your solution. • Take the interviewer’s hints to improve your code. If the interviewer makes a suggestion or asks a question, listen fully so you can incorporate any hints they may provide. • Ask yourself if you would approve your solution as part of your codebase. Explain your answer to your interviewer. Make sure your solution is correct and efficient, that you’ve taken into account edge cases, and that it clearly reflects the ideas you’re trying to express in your code.
FAANGM Screening/Phone Interview Examples:
Arrays
Reverse to Make Equal: Given two arrays A and B of length N, determine if there is a way to make A equal to B by reversing any subarrays from array B any number of times. Solution here
Contiguous Subarrays: You are given an array arr of N integers. For each index i, you are required to determine the number of contiguous subarrays that fulfills the following conditions:
The value at index i must be the maximum element in the contiguous subarrays, and These contiguous subarrays must either start from or end on index i. Solution here
Add 2 long integer (Example: “1001202033933333093737373737” + “934019393939122727099000000”) Solution here
# Python3 program to find sum of
# two large numbers.
# Function for finding sum of
# larger numbers
def findSum(str1, str2):
# Before proceeding further,
# make sure length of str2 is larger.
if (len(str1) > len(str2)):
t = str1;
str1 = str2;
str2 = t;
# Take an empty string for
# storing result
str = "";
# Calculate length of both string
n1 = len(str1);
n2 = len(str2);
# Reverse both of strings
str1 = str1[::-1];
str2 = str2[::-1];
carry = 0;
for i in range(n1):
# Do school mathematics, compute
# sum of current digits and carry
sum = ((ord(str1[i]) - 48) +
((ord(str2[i]) - 48) + carry));
str += chr(sum % 10 + 48);
# Calculate carry for next step
carry = int(sum / 10);
# Add remaining digits of larger number
for i in range(n1, n2):
sum = ((ord(str2[i]) - 48) + carry);
str += chr(sum % 10 + 48);
carry = (int)(sum / 10);
Rotational Cipher: One simple way to encrypt a string is to “rotate” every alphanumeric character by a certain amount.
Rotating a character means replacing it with another character that is a certain number of steps away in normal alphabetic or numerical order. For example, if the string “Zebra-493?” is rotated 3 places, the resulting string is “Cheud-726?”. Every alphabetic character is replaced with the character 3 letters higher (wrapping around from Z to A), and every numeric character replaced with the character 3 digits higher (wrapping around from 9 to 0). Note that the non-alphanumeric characters remain unchanged. Given a string and a rotation factor, return an encrypted string. Solution here
Matching Pairs: Given two strings s and t of length N, find the maximum number of possible matching pairs in strings s and t after swapping exactly two characters within s. A swap is switching s[i] and s[j], where s[i] and s[j] denotes the character that is present at the ith and jth index of s, respectively. The matching pairs of the two strings are defined as the number of indices for which s[i] and t[i] are equal. Note: This means you must swap two characters at different indices. Solution here
Minimum Length Substrings: You are given two strings s and t. You can select any substring of string s and rearrange the characters of the selected substring.
Determine the minimum length of the substring of s such that string t is a substring of the selected substring. Solution here
Recursion
Encrypted Words: You’ve devised a simple encryption method for alphabetic strings that shuffles the characters in such a way that the resulting string is hard to quickly read, but is easy to convert back into the original string.
When you encrypt a string S, you start with an initially-empty resulting string R and append characters to it as follows: Append the middle character of S (if S has even length, then we define the middle character as the left-most of the two central characters) Append the encrypted version of the substring of S that’s to the left of the middle character (if non-empty) Append the encrypted version of the substring of S that’s to the right of the middle character (if non-empty) For example, to encrypt the string “abc”, we first take “b”, and then append the encrypted version of “a” (which is just “a”) and the encrypted version of “c” (which is just “c”) to get “bac”. If we encrypt “abcxcba” we’ll get “xbacbca”. That is, we take “x” and then append the encrypted version “abc” and then append the encrypted version of “cba”.
Slow Sums: Suppose we have a list of N numbers, and repeat the following operation until we’re left with only a single number: Choose any two numbers and replace them with their sum. Moreover, we associate a penalty with each operation equal to the value of the new number, and call the penalty for the entire list as the sum of the penalties of each operation.For example, given the list [1, 2, 3, 4, 5], we could choose 2 and 3 for the first operation, which would transform the list into [1, 5, 4, 5] and incur a penalty of 5. The goal in this problem is to find the worst possible penalty for a given input.
Reverse Operations: You are given a singly-linked list that contains N integers. A subpart of the list is a contiguous set of even elements, bordered either by either end of the list or an odd element. For example, if the list is [1, 2, 8, 9, 12, 16], the subparts of the list are [2, 8] and [12, 16].Then, for each subpart, the order of the elements is reversed. In the example, this would result in the new list, [1, 8, 2, 9, 16, 12].The goal of this question is: given a resulting list, determine the original order of the elements. Solution Here.
Hash Tables
Pair Sums: Given a list of n integers arr[0..(n-1)], determine the number of different pairs of elements within it which sum to k. If an integer appears in the list multiple times, each copy is considered to be different; that is, two pairs are considered different if one pair includes at least one array index which the other doesn’t, even if they include the same values. Solution here.
Note: These exercises assume you have knowledge in coding but not necessarily knowledge of binary trees, sorting algorithms, or related concepts. • Topic 1 | Arrays & Strings • A Very Big Sum • Designer PDF Viewer • Left Rotation
/**
* Your solution in here. Just need to add the number in a variable type long so you
* don't face overflow.
*/
static long aVeryBigSum(int n, long[] ar) {
long sum = 0;
for (int i = 0; i < n; i++) {
sum += ar[i];
}
return sum;
}
/**
* HackerRank provides this code.
*/
public static void main(String[] args) {
Scanner in = new Scanner(System.in);
int n = in.nextInt();
long[] ar = new long[n];
for(int ar_i = 0; ar_i < n; ar_i++){
ar[ar_i] = in.nextLong();
}
long result = aVeryBigSum(n, ar);
System.out.println(result);
}
}
static int[] leftRotation(int[] a, int d) {
// They say in requirements that these inputs should not be considered.
// However, noting that we should prevent against those.
if (d == 0 || a.length == 0) {
return a;
}
int rotation = d % a.length;
if (rotation == 0) return a;
// Please note that there is an implementation, circular arrays that could be considered here,
// but that one has an edge case (Test#1)
// As, we don't need to optimize for memory, let's keep it simple.
int [] b = new int[a.length];
for (int i = 0; i < a.length; i++) {
b[i] = a[indexHelper(i + rotation, a.length)];
}
return b;
}
/**
* Takes care of the case where the rotation index. You have to take into account
* how it is rotated towards the left. To compute index of B, we rotate towards the right.
* If we were to do a[i] in the loop, then these method would need to be slightly chnaged
* to compute index of b.
*/
private static int indexHelper(int index, int length) {
if (index >= length) {
return index - length;
} else {
return index;
}
}
public static void main(String[] args) {
Scanner in = new Scanner(System.in);
int n = in.nextInt();
int d = in.nextInt();
int[] a = new int[n];
for(int a_i = 0; a_i < n; a_i++){
a[a_i] = in.nextInt();
}
int[] result = leftRotation(a, d);
for (int i = 0; i < result.length; i++) {
System.out.print(result[i] + (i != result.length - 1 ? " " : ""));
}
System.out.println("");
in.close();
}
}
Sparse Array in Java
import java.io.*;
import java.util.*;
public class Solution {
public static void main(String[] args) {
Scanner scanner = new Scanner(System.in);
final int totalN = Integer.parseInt(scanner.nextLine());
final Map<String, Integer> mapWords = buildCollectionOfStrings(scanner, totalN);
final int numberQueries = Integer.parseInt(scanner.nextLine());
printOcurrenceOfQueries(scanner, numberQueries, mapWords);
}
/**
* This method construcs a map with the collection of Strings and occurrence.
*/
private static Map<String, Integer> buildCollectionOfStrings(Scanner scanner, int n) {
final Map<String, Integer> map = new HashMap<String, Integer>();
for (int i = 0; i < n; i++) {
final String line = scanner.nextLine();
if (map.containsKey(line)) {
map.put(line, map.get(line) + 1);
} else {
map.put(line, 1);
}
}
return map;
}
private static void printOcurrenceOfQueries(Scanner scanner, int numberQueries, Map<String, Integer> mapWords) {
for (int i = 0; i < numberQueries; i++) {
// for each query, we look for how many times it occurs and we print on screen the value.
final String line = scanner.nextLine();
if (mapWords.containsKey(line)) {
System.out.println(mapWords.get(line));
} else {
System.out.println(0);
}
}
}
}
The interviewer will be thinking about how your skills and experience might help their teams.
Help them understand the value you could bring by focusing on these traits and abilities. • Communication: Are you asking for requirements and clarity when necessary, or are you just diving into the code? Your initial tech screen should be a conversation, so don’t forget to ask questions. • Problem solving: They are evaluating how you comprehend and explain complex ideas. Are you providing the reasoning behind a particular solution? Developing and comparing multiple solutions? Using appropriate data structures? Speaking about space and time complexity?Optimizing your solution? • Coding. Can you convert solutions to executable code? Is the code organized and does it capture the right logical structure? • Verification. Are you considering a reasonable number of test cases or coming up with a good argument for why your code is correct? If your solution has bugs, are you able to walk through your own logic to find them and explain what the code is doing?
– Please review Amazon Leadership Principles, to help you understand Amazon’s culture and assess if it’s the right environment for you (For Amazon) – Review the STAR interview technique (All)
FAANGM Compensation
Legend – Base / Stocks (Total over 4 years) / Sign On
I am not a lawyer, but I believe there is some sort of legal restrictions on what an internship can be (in California, at least). In my previous company, we had discussed cases on whether or not someone who was out of school can be an intern, and we got an unequivocal verdict that you have to be in school, or about to start school (usually college or graduate school, but high school can work too), in order to be considered for an internship.
I suspect the reason for this is to protect a potential intern from exploitation by companies who would offer temporary jobs to people while labeling them as learning opportunities in order to pay less.
Personally, I feel for people like you, in an ideal world you should be allowed to have the same opportunities as the formally schooled people, and occasionally there are. For example, some companies offer residency programs which are a bit similar to internships. In many cases, though, these are designed to help underrepresented groups. Some examples include:
This is a common bug in the thinking of people, doing the wrong thing harder in the hope that it works this time. Asking tough questions is like hitting a key harder when the broken search function in LinkedIn fails. No matter how hard you hit the key, it won’t work.
Given the low quality of the LinkedIn platform from a technical perspective it seems hard to imagine that they hire the best or even the mediocre. But it may be that because their interviews are too tough to hire good programmers.
Just because so few people can get a question right, does not mean it is a good question, merely that it is tough. I am also a professional software developer and know a bunch of algorithms, but also am wholly ignorant of many others. Thus it is easy to ask me (or anyone else) questions they can’t answer. I am (for instance) an expert on sort/merging, and partial text matching, really useful for big data, wholly useless for UX.
So if you ask about an obscure algorithm or brain teaser that is too hard you don’t measure their ability, but their luck in happening to know that algo or having seen the answer to a puzzle. More importantly how likely are they to need it ?
This is a tricky problem for interviewers, if you know that a certain class of algorithms are necessary for a job, then if you’re a competent s/w dev manager then odds are that you’ve already put them in and they just have to integrate or debug them. Ah, so I’ve now given away one of my interview techniques. We both know that debugging is more of our development effort than writing code. Quicksort (for instance) is conceptually quite easy being taught to 16 year old Brits or undergraduate Americans, but it turns out that some implementations can go quadratic in space and/or time complexity and that comparison of floating point numbers is a sometimes thing and of course some idiot may have put = where he should have put == or >= when > is appropriate.
Resolving that is a better test, not complete of course, but better.
For instance I know more about integrating C++ with Excel than 99.9% of programmers. I can ask you why there are a whole bunch of INT 3’s laying around a disassembly of Excel, yes I have disassembled Excel, and yes my team was at one point asked to debug the damned thing for Microsoft. I can ask about LSTRs, and why SafeArrays are in fact really dangerous. I can even ask you how to template them. That’s not easy, trust me on this.
Are you impressed by my knowledge of this ?
I sincerely hope not.
Do you think it would help me build a competent search engine, something that the coders at LinkedIn are simply unable to do ?
No.
I also know a whole bunch of numerical methods for solving PDEs. Do you know what a PDE even is ? This can be really hard as well. Do you care if you can’t do this ? Again not relevant to fixing the formless hell of LI code. fire up the developer mode of your browser and see what it thinks of Linkedin’s HTML, I’ve never seen a debugger actually vomit in my face before.
A good interview is a measure not just of ability but of the precise set of skills you bring to the team. A good interviewer is not looking for the best, but the best fit.
Sadly some interviewers see it as an ego thing that they can ask hard questions. So can I, but it’s not my job. My job is identfying those that can deliver most, hard questions that you can’t answer are far less illuminating that questions you struggle with because I get to see the quality of your thinking in terms of complexity, insight, working with incomplete and misleading information and determination not to give up because it is going badly.
Do you as a candidate ask questions well ? If I say something wrong, do you a) notice, b) have the soft skills to put it to me politely, c) have the courage to do so ?
Courage is an under-rated attribute that superior programmers have and failed projects have too little of.
Yes, some people have too much courage, which is a deeper point, where for many things there is an optimum amount and the right mix for your team at this time. I once had to make real time un-reversible changes to a really important database whilst something very very bad happened on TV news screens above my head. Too much or too little bravery would have had consequences. Most coding ain’t that dramatic, but the superior programmer has judgement, when to try the cool new language/framework feature in production code, when to optimise for speed, or for space and when for never ever crashes even when memory is corrupted by external effects. When does portability matter or not ? Some code will only be used once and we both know some of it will literally never be executed in live, do we obsess about it’s quality in terms of performance and maintainability .
The right answer is it depends, and that is a lot more important to hiring the best than curious problems in number theory or O(N Log(N)) for very specific code paths that rarely execute.
Also programming is a marathon, not a sprint, or perhaps more like a long distance obstacle course, stretches of plodding along with occasional walls. Writing a better search engine than LinkedIn “programmers” manage is a wall, I know this because my (then) 15 year old son took several weeks, at 15, he was only 2 or 3 times better than the best at LinkedIn, but he’s 18 now and as a professional grade programmer, him working for LinkedIn would be like putting the head of the Vulcan Science Academy in among a room of Gwyneth Paltrow clones.
And that ultimately may be the problem.
Good people have more options and if you have a bad recruitment process then they often will reject your offer. We spend more of our lives with the people we work with than sleep with and if at interview management is seen as pompous or arrogant then they won’t get the best people.
There’s now a Careers advice space on Quora, you might find it interesting.
![r/dataisbeautiful - Where does the world's non-renewable energy come from? Zoom in to see a point for each power plant! [OC]](https://preview.redd.it/jh7lxu3qwpt61.png?width=960&crop=smart&auto=webp&s=3678d21b2fc94780eadd6eb4f191e47d4af50daf)
![r/dataisbeautiful - [OC] Racial Diversity of Each State (Based on US Census 2019 Estimates)](https://preview.redd.it/9f2irpzr9tv61.png?width=960&crop=smart&auto=webp&s=42b635adf1f73e359c54659eeaed161d2538c625)
![r/dataisbeautiful - Share of US Wealth by Generation [OC]](https://preview.redd.it/wvrdlzm24ix61.jpg?width=960&crop=smart&auto=webp&s=480189d7d91ad6bfc67b6499e061524cd0fec674)
![r/dataisbeautiful - [OC] Emigrants from the UK by Destination](https://preview.redd.it/3pwaowy9qoz61.png?width=960&crop=smart&auto=webp&s=291473d1c1152fa746bb89812c7bbe87ecf77668)
![r/dataisbeautiful - [OC] US Rivers and Streams](https://i.redd.it/tsdy2srrgoz61.png)
![r/dataisbeautiful - [OC] National Parks of North America](https://preview.redd.it/s27jcb7p0hz61.png?width=960&crop=smart&auto=webp&s=0852c9e0ea6fdb6849dcf7a1e032e220bd833682)
![r/dataisbeautiful - [OC] Inflation of Bitcoin and DogeCoin vs. Federal Reserve target](https://preview.redd.it/ml071bet4cz61.png?width=960&crop=smart&auto=webp&s=f4b407d8d2783dc74cf42d719c40284549a21f63)
![r/dataisbeautiful - [OC] Comparing Emissions Sources - How to Shrink your Carbon Footprint More Effectively](https://preview.redd.it/if9brfwt32171.png?width=960&crop=smart&auto=webp&s=dbdec18bcbaf6762ff8f609630352705cd7be647)
![r/dataisbeautiful - [OC] Oil and gas-fired power plants in the world](https://preview.redd.it/hdzhpxwa0b171.png?width=960&crop=smart&auto=webp&s=aca0f916b4015847547f95ce8579a0233797d692)
![r/dataisbeautiful - [OC] I recently made a graph showing the Top 100 Reddit posts of all time. Some people said I should make a pie chart too, so here it is!](https://preview.redd.it/clh7xlpp0b171.jpg?width=960&crop=smart&auto=webp&s=c399509668f1285715e279cd459f7a595e83adaf)
![r/dataisbeautiful - Countries of the world sorted by those that have warmed the most in the last 10 years, showing temperatures from 1890 to 2020 [OC]](https://preview.redd.it/hdy9tqbtbv171.jpg?width=960&crop=smart&auto=webp&s=d519a7772144abdc0ca1bcff7061febbf655f440)
![r/dataisbeautiful - [OC] Climate change concern vs personal spend to reduce climate change](https://preview.redd.it/s78w2y8whq171.png?width=960&crop=smart&auto=webp&s=aa07421a85869867a8ff3c5303cc4aefee19c2b9)
![r/dataisbeautiful - [OC] Yearly Software Sales on PlayStation Consoles since 1994](https://preview.redd.it/yxrt8lszz8271.png?width=960&crop=smart&auto=webp&s=cf2bfedc56ab7ec692c43c75ce9d4b608b2563e0)
![r/dataisbeautiful - [OC] Yearly Hardware Sales of PlayStation Consoles since 1994](https://preview.redd.it/8wghxchvp1271.png?width=960&crop=smart&auto=webp&s=3540dd80d91e74687d4bf1a853ae559abb5c722b)
![r/dataisbeautiful - [OC] Mass Transit Use in America](https://preview.redd.it/pqyy2v9h34271.png?width=960&crop=smart&auto=webp&s=2d37dd6728215b333e19eb58cdecd683140c1b3e)
![r/dataisbeautiful - [OC] Cybertruck vs F150 Lightning pre-orders, by time since debut](https://preview.redd.it/tr3qd0sax6271.jpg?width=960&crop=smart&auto=webp&s=8168cb67d5eb2223724864cf855b45e6115216f9)
![r/dataisbeautiful - (Fixed once again) Top 100 Most Populous City Proper in the world. [OC]](https://i.redd.it/8ggxqh9oas271.jpg)
![r/dataisbeautiful - [OC] What do Europeans feel most attached to - their region, their country, or Europe?](https://preview.redd.it/0gy650wtd8371.png?width=640&crop=smart&auto=webp&s=cd4a8b76a2a845e4e1f1b1a9447ba4638c3ef47a)
![r/dataisbeautiful - [OC] Percentage of female students in higher education by subject area](https://i.redd.it/ppl2jqoxv9471.png)
![r/dataisbeautiful - [OC] Percentage of students with disabilities in higher education by subject area](https://i.redd.it/83839ss8qg471.png)
![r/dataisbeautiful - [OC] Arrests for Hate Crimes in NYC by Category, 2017-2020](https://preview.redd.it/cnzb7i9b3h471.png?width=960&crop=smart&auto=webp&s=a26561aa4cf61553ce90d27d0717b4e3563ec46f)
![r/dataisbeautiful - [OC] The Most Successful U.S. Sports Franchises](https://preview.redd.it/dthvusdixg471.png?width=960&crop=smart&auto=webp&s=498b3fe02357d088d668df7073920e8f11e6ff52)
![r/dataisbeautiful - G7 Corporate Tax rate 1980 - 2020 [OC]](https://i.redd.it/26rpdhxa3h471.jpg)
![r/dataisbeautiful - [OC] Euro 2020 (played in 2021) Group Stage Predictions Based of a Bayesian Linear Item Response Model](https://preview.redd.it/gwbm1s0lgd471.png?width=960&crop=smart&auto=webp&s=5850220718a41802c4eeb4625c6338aa5647b19a)
![r/dataisbeautiful - [OC] The 15 most shared musicians on Reddit](https://preview.redd.it/4nhiluzew4471.png?width=960&crop=smart&auto=webp&s=ef1c23f395db57ab712b2f52e076340d586f5548)
![r/dataisbeautiful - Spam vs. Legitimate Email, Average Global Emails per Day [OC]](https://preview.redd.it/fjbzsoelfh471.png?width=960&crop=smart&auto=webp&s=325b533ce0f4b24b03f7ce3ed3ead3386cc7d282)
![r/dataisbeautiful - Europe Covid-19 waves [OC]](https://i.redd.it/rqdjn5k8ju471.png)
![r/dataisbeautiful - Who is going to win EURO 2020? Predicted probabilities pooled together from 18 sources [OC]](https://preview.redd.it/g8a398k0pp471.png?width=960&crop=smart&auto=webp&s=6364747ea8d1f29d80a7344acf7f3087edb10011)
![r/dataisbeautiful - [OC] Population Density of Canada 2020](https://preview.redd.it/ej6rd6sx1x471.png?width=960&crop=smart&auto=webp&s=08ed614cb36226a709b55c9824a7e22d72ea9a5a)
![r/dataisbeautiful - [OC] The portion of a country's population that is fully vaccinated for COVID (as of June 2021) scales with GDP per capita.](https://preview.redd.it/v9gj0kpql1571.png?width=960&crop=smart&auto=webp&s=c7cd61f0d0055c1fd89a4ce4c4f9663f92d45d81)
![r/dataisbeautiful - [OC] These are the world’s most livable cities in 2021.](https://preview.redd.it/m7czrinyna571.png?width=960&crop=smart&auto=webp&s=d4a5b8cd531fce78c7b6d8cbe15f8c0b75f37036)
![r/dataisbeautiful - What businesses in different countries search for when they look for a marketing agency - "creative" or "SEO"? [OC]](https://preview.redd.it/ldjh8smq69571.png?width=960&crop=smart&auto=webp&s=13d5a62f5fc0410e7e39d30f473493975ee082c3)
![r/dataisbeautiful - [OC] Percentage of Adults with HIV/AIDS in Africa](https://preview.redd.it/xjppz40fpo571.png?width=960&crop=smart&auto=webp&s=e4cd946b5d9ead10f3859c98572ded29c73f75b6)
![r/dataisbeautiful - [OC] Recorded CDC deaths (2014 - June 16, 2021) from Symptoms, signs and abnormal clinical and laboratory findings, not elsewhere classified (R00-R99)](https://preview.redd.it/1hvgfo2shr571.png?width=960&crop=smart&auto=webp&s=eb561d2f881f0fe10f1243ef026975a26f544151)
![r/dataisbeautiful - What are the long term gains on cryptocurrencies? [OC]](https://preview.redd.it/qp9sseqsao571.png?width=960&crop=smart&auto=webp&s=ded99430f080b4cb3a25943795a43ae93b338ecc)
![r/dataisbeautiful - [OC] Life Expectancy and Death Probability by Age and Gender](https://preview.redd.it/cuxmlze98n571.png?width=960&crop=smart&auto=webp&s=90fd693eaab1e4e3e9ec22408549214907d6106f)
![r/dataisbeautiful - Daily Coronavirus cases in Canada vs % of Population Vaccinated [OC]](https://preview.redd.it/sqyq4v0xym571.png?width=960&crop=smart&auto=webp&s=12e786d67a9b8323a03f6cf82689f554caa2868b)
![r/dataisbeautiful - [OC] Do stricter gun laws reduce firearms homicides?](https://preview.redd.it/99d71o2o7r571.png?width=960&crop=smart&auto=webp&s=251d96c1ae867e11da31c7ae9e4e6703a303e753)
![r/dataisbeautiful - [OC] Relative frequency of words in economics textbooks vs their frequency in mainstream English (the Google Books corpus)](https://preview.redd.it/5r0ljnoeii571.png?width=960&crop=smart&auto=webp&s=c846904a06f9e6c1e66241ce9ec5cf0763c896b4)
![r/dataisbeautiful - [OC] Hours per day spent on mobile devices by US adults](https://preview.redd.it/h37v8y3lxf571.png?width=960&crop=smart&auto=webp&s=c4d12abc6f1b9fbb6ed82057914f7fd3953727fe)
![r/dataisbeautiful - [OC] Environmental Impact of Coffee Brewing Methods](https://i.redd.it/57sap4gk4g571.png)
![r/dataisbeautiful - [OC] Murders in major U.S. Cities: 2019 vs. 2020](https://preview.redd.it/bqrhband9w571.png?width=960&crop=smart&auto=webp&s=dfe540bd013f012671a729dc97fea488235aab9a)
![r/dataisbeautiful - Support for same-sex marriage by religious group [OC]](https://i.redd.it/peewdefhi2671.png)
![r/dataisbeautiful - Daily chance of dying for Americans [OC]](https://i.redd.it/1axa0pajeg671.png)
![r/dataisbeautiful - [OC] Mapping Global Carbon Emission Intensity (Dec 2020)](https://preview.redd.it/aoh1zkvfmm671.png?width=960&crop=smart&auto=webp&s=c2d6a5b7ac2af3deda9e5a65a191ed83f78dab06)
![r/dataisbeautiful - [OC] Total number of streams per artist vs. number of Top 200 hits (Spotify Top 200 since 2017)](https://preview.redd.it/k3itv2g5me671.png?width=960&crop=smart&auto=webp&s=9d432fe5c26d34eb7303f9ad6a6ffe23f368f899)
![r/dataisbeautiful - [OC] Number of Miss Americas by U.S. State](https://preview.redd.it/3rdqv88ip4671.png?width=960&crop=smart&auto=webp&s=69c199b2b7da770eeca2a71167ff0d40db12455c)
![r/dataisbeautiful - [OC] The World's Nuclear Warheads](https://preview.redd.it/wy6vfb27ca671.png?width=960&crop=smart&auto=webp&s=2af05396ac3bce6ad227bce13be096fcb93aa6d0)
![r/dataisbeautiful - [OC] The population of Las Vegas over time](https://i.redd.it/cuc8zkijla671.png)
![r/dataisbeautiful - [OC] The Alpha to Omega of Wikipedia](https://preview.redd.it/ura7wskjdn671.jpg?width=960&crop=smart&auto=webp&s=6b8bf16fdfee230978c5b1c3fb1fc737b13c89a2)
![r/dataisbeautiful - [OC] Crime Rates in the US: 1960-2021](https://preview.redd.it/d5c23282zt671.png?width=960&crop=smart&auto=webp&s=0eb18045a538863347a1f99dddd4132c3ee809a5)
![r/dataisbeautiful - [OC] A network visualisation of privacy research (83k nodes, 462k edges)](https://preview.redd.it/0erin1vlzz671.png?width=960&crop=smart&auto=webp&s=bf014b31b336979d5fe8157423024f0d2777e24d)
![r/dataisbeautiful - [OC] GDP (at purchasing power parity) per capita in international dollars](https://preview.redd.it/p9ohlc9xjw671.png?width=960&crop=smart&auto=webp&s=f66a545bc6766fac53c164296680cc79a3c262f1)
![r/dataisbeautiful - Phone Call Anxiety is a real thing for Millennials and Gen Z [OC]](https://i.redd.it/horan1lg9v671.png)
