Docker grabs conductor's baton, jumps into cluster orchestration
Forget containers – are you ready for a platform?
DockerCon EU Linux container wrangler Docker rounded out the first day of its DockerCon EU conference on Thursday by unveiling three new services that represent the next steps in Docker's transition from toolset to full-fledged platform.
"We're incubating a series of projects separately from the core runtime," Docker cofounder and CTO Solomon Hykes explained during his Thursday evening keynote. "The core runtime is focused on stability. These new projects, these new tools, are focused on attacking a new problem in an independent, orthogonal way."
The trio of services are designed to make it easier for admins to provision and orchestrate distributed applications made up of multiple services running on independent Docker containers, which Hykes said was the one feature most requested by Docker customers.
The first of the three, dubbed Docker Machine, automates the process of provisioning the Docker Engine onto any host – whether it's a local laptop, a data center VM, or a cloud host – eliminating much of the manual configuration and scripting that was previously necessary to get Docker running in production.
Solomon Hykes had container orchestration on his mind at the first-ever DockerCon EU conference
Initially, Docker Machine will ship with drivers that facilitate creating Docker hosts on Amazon Web Services, DigitalOcean, IBM SoftLayer, Microsoft Azure, Microsoft Hyper-V, OpenStack, Rackspace, VirtualBox, and VMware, with more targets to come.
Further building on those capabilities is the next service, Docker Swarm, a native clustering service for containers. Docker Swarm can create a resource pool from multiple Docker Engine hosts and then automatically schedule workloads according to their resource requirements. It can also automatically rebalance workloads in the event that hosts fail.
Hykes hastened to add, however, that Swarm would not be the sole clustering option for use with Docker – which he described as Docker's "batteries included, but batteries removable" philosophy.
"The goal is not to create the clustering solution that does out of the box everything for everybody," Hykes said. "Instead, what we're trying to provide a standard default implementation that's good enough to be really sweet to use out of the box, and then make it really easy to swap out that default backend and bring in your favorite backend."
As an example of that, Hykes added, Docker is partnering with Mesosphere to make Mesos a first-class citizen within the Docker ecosystem.
Make that two and a half services
The last of the three new services is Docker Compose, and it's also the least mature – so much so that Hykes described it as a "tentative proposal for a tool." The idea here is to allow developers to define multi-container distributed applications by specifying services and their dependencies using simple YAML configuration files, independent of any underlying architecture.
By way of demonstration, Hykes showed how a single command –
docker up – could be used to bring up an entire distributed application composed of multiple containers. Still, he said, Docker Compose is "still not finished" and he called on the community to help make it fully baked.
All three orchestration services are designed to be scalable to any number of containers running in whatever configuration the user needs. As Docker marketing veep David Messina explained to El Reg in a briefing earlier this week, "We literally mean that all these capabilities that we're talking about scale from, let's say, a two-container application that runs on my laptop, all the way up to hundreds of services running on hundreds of hosts."
Public APIs are also planned for all three services, although only the API for Docker Machine is available as of Thursday. APIs for the other two are expected to launch in the first half of 2015.
The debut of these new services should come as no surprise to followers of the Docker open source project, which develops the software in full public view. All three services have been available as pull requests on GitHub and they have been developed based on feedback from the Docker community. Thursday's announcement was more of a coming-out party, where Docker showed them off to more casual consumers of the tech for the first time.
But not everyone agrees that piling these sorts of features onto Docker is the right approach. On Monday, CoreOS – makers of a lightweight Linux distro for massive-scale server deployments – announced Rocket, a minimalist container format created specifically in response to Docker's platform aspirations.
At a community event in San Francisco, CoreOS CEO Alex Polvi criticized Docker's process model, saying, "Having a service that runs as root on your servers that does everything is fundamentally flawed." He added that with Rocket, CoreOS would focus on developing lightweight, standalone tools for managing "minimal containers" for large-scale web infrastructure.
How this budding rivalry pans out remains to be seen. It's safe to say there will be no mass exodus from Docker any time soon, though, as the Rocket software is still pre-alpha. And the Docker ecosystem is strong and growing; partners that are already experimenting with the new orchestration services include IBM, Joyent, Microsoft, and Mesosphere, among others.
Customers are now free to kick the new services around, too. More information on each is available at the Docker blog, and all three services are available in alpha versions – even including the "unfinished" Docker Compose – beginning on Thursday (see above links). As for when the services will be part of the shipping product, Docker says it's shooting for general availability in the second quarter of 2015. ®