Mesos With Google Container Registry

Hey you crazy kids! Have you ever set up a Mesos cluster inside Google Compute Engine and wanted to pull Docker containers from Google’s Container Registry rather than having to stand up your own registry? Did you try it and get a bit lost? Because I certainly did, but it turns out that it’s really not that hard.

First, you’ll need to create a JSON service account key from the API Manager in the GCE console. Download that and create a new instance inside your environment (this instance is expendable - I just preferred making sure I wasn’t leaving anything on the Mesos control node myself).

Copy the JSON file to that instance and then log in. Then, issue this command:

    docker login -e 1234@5678.com -u _json_key -p "$(cat [JSON_FILE])" https://gcr.io

(the email address doesn’t have to be a legitimate address you control, but it does have to be a legal email address. Which is odd, but there you go)

This saves the required credentials in that user’s .docker directory. So, let’s tar that up!

    tar czf docker.tar.gz .docker

Now, copy that file across to all your worker nodes (may I suggest Ansible? Just add that to your playbook during worker node creation), preferably to an obvious place like /etc (you could also store it on Google Cloud Storage instead!)

(oh, and you can destroy that sacrificial instance now!)

The JSON below is a (simplified!) Marathon entry for pulling a dashboard container from Google’s Container Registry instead of DockerHub:

 {
    “id”: “dashboard”,
    “cpus”: 0.2,
    “mem”: 512,
    “instances”: 1,
    “container”: {
        “type”: “DOCKER”,
        “docker”: {
            “image”: “gcr.io/[PROJECT]/dashboard”,
            “network”: “HOST”
        }
    },
    “uris”: [“file:///etc/docker.tar.gz”]
}

The magic is the uris array that tells the Mesos worker where to find the private registry and the credentials needed to login (in this case, the Google registry).

And as for pushing your images up to CR? That’s pretty simple too:

Tag your image:
- docker tag -f dashboard gcr.io/[PROJECT]/dashboard
Push your image:
- gcloud docker push gcr.io/[PROJECT]/dashboard

And voilà! Your Mesos cluster can now pull from your private Container Registry!

(note, you may want to check the docs if you’re not in the US - you can push to and pull from a registry hosted nearer to you rather than just gcr.io - and for those of us in the US, we can’t rely on gcr.io always being the US, though it is currently)

Ian Pointer

Mesos With Google Container Registry