Spotify’s journey with Google Cloud
Spotify’s mission is to connect a million creators with a billion fans. Data is the core of running their business, as they host over 130 million audio files. The challenge is not just the number of Spotify users, but the amount of content that Spotify has. As a result of having numerous users and a huge catalog, they experience up to 8 million events per second. With these increasing daily, scaling is a key determinant for Spotify, so that they can get insights about their users and key statistics about their business. They initially opted to run these workloads on-prem, which inevitably led to their systems not scaling anymore. This is when they made the wise decision to move to the cloud. So let’s look at Spotify’s journey as they moved to the cloud with Google.
In 2016, Spotify announced that it would be moving to Google Cloud. Contrary to the expectations, it was the Spotify engineers who advocated for this move. They saw that it could solve their scaling problem and provide a foundation for them to build their business on. Spotify did not choose Google Cloud based on their brand and scale alone, they also considered their reputation of being a data-driven and engineering-oriented company. Being fully managed, would ease the burden of managing storage, compute, and network capacity, which was becoming challenging for a global company. In addition, they could also utilize some services on Google Cloud such as BigQuery and Pub/Sub to aid in their operations.
Google Cloud solutions for Spotify
Cloud migration for such a large enterprise is intricate. Consequently, Google Cloud was flexible in letting them choose their migration strategy. To make this process seamless, they migrated their services, users, data, and teams separately. For each step, migration was only half of it, the other half comprised reliability testing and educating their staff about GCP, including:
GCP’s support
Pricing
Networking
GCP Products they could utilize
By May 2017, Spotify had fully migrated to Google Cloud. With this migration, the company is now able to enjoy full infrastructure management. This allows their engineers to focus on Spotify-specific actions, which resulted in faster product development.
Because of their distributed computing environment, Spotify previously had the largest Hadoop cluster in Europe on their premises, running up to 20,000 jobs per day, working with over 100 PB of data. After their migration, they did not entirely give up on this. They still run the clusters but on the cloud. The speed of analyzing their data has multiplied with minimum effort.
They also recruited the services of Google Cloud Dataflow to run the largest serverless Dataflow job ever run on the platform. Since Dataflow was quite new to Spotify’s engineers, they utilized Scio (a Scala API) to simplify Dataflow’s less familiar functions.
Spotify was one of the early adopters of the microservices architecture and has up to 1200. They initially run containerized microservices on Docker. When they were migrating to the cloud, they did the same for Kubernetes and are now using GKE to manage their containers.
Spotify’s current data stack on GCP relies on BigQuery for Ad Hoc Querying and Pub/Sub for event delivery. BigQuery handles over 10 million queries and scheduled jobs per month, while Pub/Sub handles over 1 trillion requests per day at ultra-low latencies. Dataproc has also been recruited for processing large chunks of data.
Spotify’s journey with Google Cloud has been a revelation for them. They are now able to deliver services and innovate faster without having the burden of infrastructure management.