Executing Gradient Descent on the Earth

Source: fosterelli.co

Chris Foster:

A common analogy for explaining gradient descent goes like the following: a person is stuck in the mountains during heavy fog, and must navigate their way down. The natural way they will approach this is to look at the slope of the visible ground around them and slowly work their way down the mountain by following the downward slope.

This is a pretty fun thought experiment carried through into real code. I’m a little sad that it stops where it does, though. I’d love to have seen the paths created for RMS Prop, the Adam optimiser, learning rate annealing1, etc. Also: It would be fun to see what would happen if this was expended to include elevations below sea level.

  1. Which I can’t find a good link for. This is where the gradually reduce the learning rate over the lifetime of training.

Machine Learning’s ‘Amazing’ Ability to Predict Chaos

Source: www.quantamagazine.org

Natalie Wolchover:

In a series of results reported in the journals Physical Review Letters and Chaos, scientists have used machine learning — the same computational technique behind recent successes in artificial intelligence — to predict the future evolution of chaotic systems out to stunningly distant horizons. The approach is being lauded by outside experts as groundbreaking and likely to find wide application.

There are a couple of disclaimers here, I think. The main one is that this is a case of “for eight times longer than existing methods”, rather than the “for the lifetime of the system” which the title seems to imply. But still: amazing result. Being British, obviously the must exciting potential application for this lies in improved weather prediction. But I’m sure there are other uses, too…

As always, I’d love to know how much computing power this took, but I couldn’t fin that information in either this article or the original paper.

Earn.com Joins Coinbase

Source: news.earn.com

It look me a little while to parse what earn.com actually is. It seems as though it has been many things (including a bitcoin mine) during its years of operation. What it is now though, at root, seems to be something very similar to Google Surveys, but paying out in Bitcoin, rather than Play Store credit.

Over the last several years, the primary way most people have obtained cryptocurrency is through buying it, with many of these transactions facilitated by Coinbase. With this acquisition, we allow users to also earn crypto by doing things they already know how to do — like replying to emails and filling out surveys.

It seems as though Coinbase is trying to kickstart a bitcoin based services market. The former CEO of Earn.com is also becoming the CTO of Coinbase as part of the deal. So Coinbase must have quite a bit of faith in this deal.

But still, it’s not entirely clear to me what advantages Earn.com has over similar services which pay out in fiat currency.

Lessons Learned Reproducing a Deep Reinforcement Learning Paper

Source: amid.fish

Matthew Rahtz:

I’ve seen a few recommendations that reproducing papers is a good way of levelling up machine learning skills, and I decided this could be an interesting one to try with. It was indeed a super fun project, and I’m happy to have tackled it – but looking back, I realise it wasn’t exactly the experience I thought it would be.

This whole post got me really excited, despite the difficulties mentioned. It also took me back to the mindset I needed during my PhD. When it takes a couple of hours to figure out if something worked, you need to spend a lot more time considering what that something should be. Reimplamenting someone else’s paper can be a really great exercise. You get really intimate knowledge of how the system works. You level up your skills. Most importantly: you get ideas for what to try next.

Reading the post, I suspect the first thing I would do differently is aim lower than he did:

The first surprise was in terms of calendar time. My original estimate was that as a side project it would take about 3 months. It actually took around 8 months. (And the original estimate was supposed to be pessimistic!) Some of that was down to underestimating how many hours each stage would take, but a big chunk of the underestimate was failing to anticipate other things coming up outside the project. It’s hard to say how well this generalises, but for side projects, taking your original (already pessimistic) time estimates and doubling them might not be a bad rule-of-thumb.

That’s more or less the standard rule of thumb for all engineering projects, in my experience. Even so, eight months is a long time to spend on a side project. Like I said above: I’ll be aiming lower. My plan of attack starts with the 2018 and version of fast.ai’s “Cutting Edge Deep Learning for Coders” when it’s released later this year.

A Little More about Why I’m Fascinated By Short Selling

This isn’t strictly relevant to the subject of this blog, but as I noted in that last link post I’ve been slightly fascinated by the subject of short sellers recently. Since this is my blog, I’m going to talk a little about why that is. Besides, I think understanding fiat currency and financial markets is increasingly relevant to crypto currency, which is one of the reasons I’m learning about them.

In the first instance, short selling is interesting because it’s dangerous. When you buy a stock (or go long) the most you can lose is 100% of your stake. But when you bet against (or short) a stock there’s basically no limit on what you can potentially lose, since there’s no theoretical upper limit of what the value of the stock could rise to1.

Shorting a stock is in some ways like betting on the don’t pass line at craps. At the craps table it’s an extremely rational action. Your odds of winning are actually better than at the pass line2, which is the “normal” place to bet. But you’re betting against the table, essentially attempting to profit from the other players losing3. In much the same way, when you short a stock you’re betting against the success of the company and attempting profit on the losses of those who bought the stock. It’s a negative action. People tend not to like it when you bet against them.

On the other hand, shorting can be useful, or even somewhat noble. In a bubble, shorting a stock can cause a gradual deflation, rather than a pop. As noted in that last post, this might have been what happened to bitcoin after the futures market was introduced. There’s no futures market for other cryptocurrencies, but their price tends to be somewhat correlated to bitcoin (so far), so they also lost value.

The ability to bet against a stock also provides an incentive to find overvalued stocks. One reason a company might be overvalued is because they’re fraudulent or even criminal (Enron is a good example). Hence, there are short sellers who specialise in finding and researching fraudulent companies. They profit by shorting the stock and then publishing their findings to the widest audience possible.

Obviously this also means there’s some incentive for a firm to skimp on the research and just publish negative information about a company anyway. It seems as though something like this might have happened to AMD last month.

If you are interested in the subject of short selling, I found the following documentaries (and one feature film) both entertaining and informative4:

Betting on Zero

The story of Pershing Square Capital head Bill Ackman’s quest to prove that Herbalife is an obvious fraud and pyramid scheme (spoiler: it is).

Watch it: Netflix, iTunes, Amazon UK, Amazon US

Dirty Money: Drug Short

Remember Martin Shkrelli? He was actually a pretty small fish. This short doc is about the short sellers who went after the shark. It makes a good follow up to Betting on Zero, because Bill Ackman was on the other side of this trade.

Watch it: Netflix

The China Hustle

This is about shady Chinese companies using a convoluted process to list themselves on the US stock market. It gets into the lengths short sellers can go to in order to perform their research.

Watch it: Netflix, Amazon US

The Big Short

Based on the Micheal Lewis book of the same name. Won on Oscar for best adapted screenplay. Manages to succeed as a comedy whilst also telling the story of the 2008 financial crash. Which is impressive. And a little scary.

Watch it: Netflix, iTunes, Amazon UK, Amazon US

Disclosure: Some of the links on this page are affiliate links. This means that if you click one and then buy something, I might earn a commission at no additional cost to you. That being said: everything I write, present or recommend on this site is my honest opinion.

  1. Obviously there is a practical limit. The value of a stock can’t actually grow to infinity. Likewise, my understanding is that stocks tend to grow steadily. They might crash down, but they’re unlikely to crash up.
  2. At the pass line, the house has an edge of 1.41%. At the don’t pass line it’s 1.36%.
  3. Of course, the distinction between the pass and don’t pass lines at a craps table is completely arbitrary. The casino just wants you to take the option which has better odds for them.
  4. All of which are available on Netflix UK, so perhaps in your region as well. Netflix make it really hard for me to check without a lot of messing with my VPN.

Bitcoin Was Prone to Bubbles Until Bears Could Bet Against It

Source: www.bloomberg.com

Noah Smith:

Limits to arbitrage can help explain why Bitcoin has been so bubble-prone. Until recently, it was easy enough to take a long position, but expensive and risky to bet against the cryptocurrency. Things really changed in December, when U.S. regulators allowed the trading of Bitcoin futures. That move came in the middle of a historic runup in the price of Bitcoin and other cryptocurrencies. But as soon as futures contracts began to trade, an interesting thing happened — futures prices suggested that Bitcoin’s growth would slow.

What happened next is historic. Bitcoin’s price crashed from a high of about $19,000 to less than $7,000 as of the writing of this article[.]

This is certainly not the only theory as to why the most recent bitcoin bubble deflated1, but it is quite compelling. I’m posting it mainly because I’ve been fascinated by the role, mechanism and psychology of short sellers just recently. It would seem like the development of these markets would tend to take bitcoin closer to “regular” finance. That might be a good thing if you like stability. But it might be a bad thing if you like purity, and want “finance professionals” as far away from your trust minimised medium of evangelical as possible. There are myriad arguments for both sides, I’m sure.

Personally, I like stability. But I like a solid market for goods and services as a basis for that stability. Which I guess is a little bit of a chicken and egg problem right now.

  1. Which seems like a more accurate description than “popped”, thankfully.

Making $100K Trading CryptoKitties

Source: HackerNoon.com

Ivan Bogatyy:

Next morning I called Oleg, a close buddy who runs a top crypto hedge fund in Russia to discuss this unusual phenomenon. While the CryptoKitties game was merely a curiosity for me, Oleg immediately spotted a lucrative investment opportunity. First, we were clearly among the earlier entrants. Second, the game looked like it had all the precursors of going viral, similar to Pokemon Go, and was in the early stage of a hockey stick explosion.

Now the question was, which Kitties should we buy? The marketplace offered 4 ways to sort Kitties: cheapest first, most expensive first, newest first, oldest first (note: the website had been redesigned since). The first 3 options are clearly transient: you can always put a cheaper, more expensive, or a newer Kitty on the market. Oldest, however, is like diamonds: forever. Thus we decided to buy single-digit Founder Cats, despite their already hefty price tags: somebody just snatched them at 25 ETH and re-listed the lineup at 50 ETH ($25K), with Founder Cat #1 trading even higher at 150 ETH.

On the surface that’s an insane amount of money to make buying and selling cats which do not exist. Spoiler: the story of how the majority of that money was made (a single trade involving an early CryptoKitty) isn’t really all that exciting.

The description of the arbitrage bot which pulled an additional $8k out of the network is a little more interesting, though. That’s possible because of the underlying technology of Cryptokitties; namely: non fungible crypto assets. Cryptocollectables are now a thing. I’m interested to see whether this grows into anything beyond the obvious use case of being linked to real world assets.

Cryptokitties also raise some interesting questions about the value of digital goods. How does a Cryptokitty compare to an Amiibo, for example? Which holds more inherent value? The Amiibo because it’s a physical object you can hold in your hand? Or the CryptoKitty which is digitally linked to you and can be bred to make more kitties?

Do You Trust This Computer? (Documentary)

Source: DoYouTrustThisComputer.org

I watched this over the weekend. It’s very well put together, and features an impressive panel of experts. That said: it’s somewhat unfocused and alarmist. Having watched it, I found it interesting and thought provoking. But I’m not 100% certain what its thesis or conclusion are. It seems to amount to:

  • Artificial Intelligence is an exciting field;
  • If we’re not careful it might be dangerous;
  • We should probably do… something about that.

How Cambridge Analytica’s Facebook Targeting Model Really Worked

Source: theconversation.com

Side note: I originally read this piece at NiemanLab.org, but since it was first published by The Conversation, that’s what I’m linking to.

Matthew Hindman:

The researcher whose work is at the center of the Facebook-Cambridge Analytica data analysis and political advertising uproar has revealed that his method worked much like the one Netflix uses to recommend movies.

In an email to me, Cambridge University scholar Aleksandr Kogan explained how his statistical model processed Facebook data for Cambridge Analytica. The accuracy he claims suggests it works about as well as established voter-targeting methods based on demographics like race, age and gender.

As noted in the article, Cambridge Analytica’s system is far from being a crystal ball. It seems like it’s even further from being a voodoo doll, allowing people be nudged towards CA’s clients’ ends. After all: Netflix can make reasonable guesses at the movies and TV shows you’re likely to enjoy. It can’t change your taste by strategically recommending particular media1.

If you’re interested in going deeper into this subject, I’d be inclined to recommend two places to look:

First: educational material on the type of machine learning concerned. Weeks 8 (specifically the lectures on Dimensionality Reduction) and 9 (specifically Recommender Systems) of Andrew Ng’s Machine Learning course on Coursera (non-affiliate link);

Second: a work of speculative fiction2. The novel Interface (Amazon US, Amazon UK) by Neal Stephenson and George Jewsbury, writing as Stephen Bury.

  1. I hope.
  2. Which is also a somewhat paranoid and eerily prescient political techno-thriller.

CoinList Announces Series A Funding

Source: CoinList @ medium.com

This is pretty cool. I have to admit that I want CoinList to succeed. Partly because I think they represent the “sensible” side of blockchain. They treat the law as something which should be worked within, rather than circumvented. They also do a lot of due diligence. I believe that if an ICO is offered on their platform, it is at least legitimate (i.e. not an outright fraud). That’s not say it isn’t still spectacularly risky, but that’s a different matter all together.

The other part of why I want them to succeed is because it was through them that I invested in FileCoin1. Which is, to date, the only money I’ve ever put into anything crypto or blockchain related. Obviously I don’t want that (or my conduit to it) to fail.

They’re also the love child of Protocol Labs and AngelList, which is pretty good parentage.

  1. Time will tell whether “invested” is actually the right word there…