Cloud Foundry Container Runtime
- Slack: #cfcr on https://slack.cloudfoundry.org
- Pivotal Tracker: https://www.pivotaltracker.com/n/projects/2093412
Table of Contents
- Deploying CFCR
- Accessing the CFCR Cluster with kubectl
- Backup & Restore
Accessing the master:
- Single Master: Set up a DNS name pointing to your master's IP address
- Multiple Masters: A TCP load balancer for your master nodes.
- Use a TCP load balancer configured to connect to the master nodes on port 8443.
- Add healthchecks using either a TCP dial or HTTPS by looking for a
200 OKresponse from
- if you have used BOSH Bootloader on GCP then you need to manually create a firewall rule. Allow access to port TCP 8443 to VMs in your BBL network tagged
cfcr-masterfrom your load balancer's IP.
Cloud Config with
small-highmem(See cf-deployment for reference)
- three availability zones
Note: the cloud-config properties can be customized by applying ops-files. See
manifests/ops-filesfor some examples.
If using loadbalancers then apply the
cfcr-master-loadbalancerto the cloud-config to add the instances to your loadbalancers. See BOSH documentation for information on how to configure loadbalancers.
Kubernetes uses etcd as its datastore. The official infrastructure requirements and example configurations for the etcd cluster can be found here.
Upload the latest Xenial stemcell to the director.
Untar the kubo-deployment tarball and rename it
Option 1. Single Master
cd kubo-deployment bosh deploy -d cfcr manifests/cfcr.yml \ -o manifests/ops-files/misc/single-master.yml \ -o manifests/ops-files/add-hostname-to-master-certificate.yml \ -v api-hostname=[DNS-NAME]
Option 2. Three Masters
cd kubo-deployment bosh deploy -d cfcr manifests/cfcr.yml \ -o manifests/ops-files/add-vm-extensions-to-master.yml \ -o manifests/ops-files/add-hostname-to-master-certificate.yml \ -v api-hostname=[LOADBALANCER-ADDRESS]
Note: Loadbalancer address should be the external address (hostname or IP) of the loadbalancer you have configured.
Check additional configurations, such as setting Kubernetes cloud provider, in docs.
Add Kubernetes system components
bosh -d cfcr run-errand apply-specs
Run the following to confirm the cluster is operational
bosh -d cfcr run-errand smoke-tests
Please check out our manifest and ops-files in kube-deployment for examples on how to configure kubo-release. Additionally, we have a doc page to describe how to configure Kubernetes components for the release.
CFCR can be deployed with Pod Security Policies. Check for more details in the doc
CFCR allows you to configure proxy for all components. Check recommendations for no proxy settings first.
CFCR clusters on BOSH Lite are intended for development. We run the deploy_cfcr_lite script to provision a cluster with the latest stemcell and master of kubo-release. This requires that the cloned kubo-release repository can be found from
cd ../kubo-release from within the kubo-deployment directory.
cd kubo-deployment ./bin/deploy_cfcr_lite
Login to the Credhub Server that stores the cluster's credentials:
Find the director name by running
cd kubo-deployment ./bin/set_kubeconfig <DIRECTOR_NAME>/cfcr https://[DNS-NAME-OR-LOADBALANCER-ADDRESS]:8443
We use BBR to perform backups and restores of the etcd node within a CFCR cluster, for both single and three master deployments. Our backup currently takes an etcd snapshot without interruptions to the cluster. However, for restore we take both the kube-apiserver and etcd offline to restore the cluster with the specified snapshot. Restore is a destructive operation that will completely overwrite any existing data on the cluster. For a closer look at the bbr scripts, check out:
To run the
bbr cli against a CFCR cluster, follow the steps under "BOSH Deployment" on the BBR documentation page.
Follow the recommendations in etcd's documentation for monitoring etcd metrics.
By default CFCR runs with CoreDNS in preference of Kube-DNS.
If you are migrating from an earlier version of CFCR, Kube-DNS can be removed by running:
kubectl delete deployment -n kube-system kube-dns
You may notice that a
kube-dns service remains, this is also required by the CoreDNS spec.
CFCR had a set of scripts, including
deploy_k8s, that were the primary mechanism we supported to deploy BOSH and Kubernetes clusters. We no longer support these and have removed the corresponding documentation from https://docs-cfcr.cfapps.io
The BOSH oriented method documented in this README.md is the supported method to deploy Kubernetes clusters with CFCR.
K8s 1.11 release kicked off the deprecation timeline for the Heapster component, see here for more info. As a result, we're in the process of replacing Heapster with Metrics Server in the upcoming releases of kubo-release.
Heapster can be removed by running:
kubectl delete deployment -n kube-system heapster