nixos/kubernetes: adding manual section for kubernetes and writing release note for NixOS 19.03

3 files changed, 169 insertions, 0 deletions

diff --git a/nixos/doc/manual/configuration/configuration.xml b/nixos/doc/manual/configuration/configuration.xml
index cebc4122c6c66..138d1d86d7fc9 100644
--- a/nixos/doc/manual/configuration/configuration.xml
+++ b/nixos/doc/manual/configuration/configuration.xml
@@ -23,5 +23,6 @@
  <xi:include href="linux-kernel.xml" />
  <xi:include href="../generated/modules.xml" xpointer="xpointer(//section[@id='modules']/*)" />
  <xi:include href="profiles.xml" />
+ <xi:include href="kubernetes.xml" />
 <!-- Apache; libvirtd virtualisation -->
 </part>
diff --git a/nixos/doc/manual/configuration/kubernetes.xml b/nixos/doc/manual/configuration/kubernetes.xml
new file mode 100644
index 0000000000000..ddc026c0c0109
--- /dev/null
+++ b/nixos/doc/manual/configuration/kubernetes.xml
@@ -0,0 +1,127 @@
+<chapter xmlns="http://docbook.org/ns/docbook"
+         xmlns:xlink="http://www.w3.org/1999/xlink"
+         xmlns:xi="http://www.w3.org/2001/XInclude"
+         version="5.0"
+         xml:id="sec-kubernetes">
+ <title>Kubernetes</title>
+
+ <para>
+   The NixOS Kubernetes module is a collective term for a handful of
+   individual submodules implementing the Kubernetes cluster components.
+ </para>
+
+ <para>
+   There are generally two ways of enabling Kubernetes on NixOS.
+   One way is to enable and configure cluster components appropriately by hand:
+<programlisting>
+services.kubernetes = {
+  apiserver.enable = true;
+  controllerManager.enable = true;
+  scheduler.enable = true;
+  addonManager.enable = true;
+  proxy.enable = true;
+  flannel.enable = true;
+};
+</programlisting>
+  Another way is to assign cluster roles ("master" and/or "node") to the host.
+  This enables apiserver, controllerManager, scheduler, addonManager,
+  kube-proxy and etcd:
+<programlisting>
+<xref linkend="opt-services.kubernetes.roles"/> = [ "master" ];
+</programlisting>
+  While this will enable the kubelet and kube-proxy only:
+<programlisting>
+<xref linkend="opt-services.kubernetes.roles"/> = [ "node" ];
+</programlisting>
+  Assigning both the master and node roles is usable if you want a single
+  node Kubernetes cluster for dev or testing purposes:
+<programlisting>
+<xref linkend="opt-services.kubernetes.roles"/> = [ "master" "node" ];
+</programlisting>
+  Note: Assigning either role will also default both
+  <xref linkend="opt-services.kubernetes.flannel.enable"/> and
+  <xref linkend="opt-services.kubernetes.easyCerts"/> to true.
+  This sets up flannel as CNI and activates automatic PKI bootstrapping.
+ </para>
+
+ <para>
+   As of kubernetes 1.10.X it has been deprecated to open
+   non-tls-enabled ports on kubernetes components. Thus, from NixOS 19.03 all
+   plain HTTP ports have been disabled by default.
+   While opening insecure ports is still possible, it is recommended not to
+   bind these to other interfaces than loopback.
+
+   To re-enable the insecure port on the apiserver, see options:
+   <xref linkend="opt-services.kubernetes.apiserver.insecurePort"/>
+   and
+   <xref linkend="opt-services.kubernetes.apiserver.insecureBindAddress"/>
+ </para>
+
+ <note>
+  <para>
+   As of NixOS 19.03, it is mandatory to configure:
+   <xref linkend="opt-services.kubernetes.masterAddress"/>.
+   The masterAddress must be resolveable and routeable by all cluster nodes.
+   In single node clusters, this can be set to <literal>localhost</literal>.
+  </para>
+ </note>
+
+ <para>
+   Role-based access control (RBAC) authorization mode is enabled by default.
+   This means that anonymous requests to the apiserver secure port will
+   expectedly cause a permission denied error. All cluster components must
+   therefore be configured with x509 certificates for two-way tls communication.
+   The x509 certificate subject section determines the roles and permissions
+   granted by the apiserver to perform clusterwide or namespaced operations.
+   See also:
+   <link
+     xlink:href="https://kubernetes.io/docs/reference/access-authn-authz/rbac/">
+     Using RBAC Authorization</link>.
+ </para>
+
+  <para>
+   The NixOS kubernetes module provides an option for automatic certificate
+   bootstrapping and configuration,
+    <xref linkend="opt-services.kubernetes.easyCerts"/>.
+   The PKI bootstrapping process involves setting up a certificate authority
+   (CA) daemon (cfssl) on the kubernetes master node. cfssl generates a CA-cert
+   for the cluster, and uses the CA-cert for signing subordinate certs issued to
+   each of the cluster components. Subsequently, the certmgr daemon monitors
+   active certificates and renews them when needed. For single node Kubernetes
+   clusters, setting <xref linkend="opt-services.kubernetes.easyCerts"/> = true
+   is sufficient and no further action is required. For joining extra node
+   machines to an existing cluster on the other hand, establishing initial trust
+   is mandatory.
+ </para>
+
+ <para>
+   To add new nodes to the cluster:
+   On any (non-master) cluster node where
+   <xref linkend="opt-services.kubernetes.easyCerts"/> is enabled, the helper
+   script <literal>nixos-kubernetes-node-join</literal> is available on PATH.
+   Given a token on stdin, it will copy the token to the kubernetes
+   secrets directory and restart the certmgr service. As requested
+   certificates are issued, the script will restart kubernetes cluster
+   components as needed for them to pick up new keypairs.
+ </para>
+
+ <note>
+  <para>
+   Multi-master (HA) clusters are not supported by the easyCerts module.
+  </para>
+ </note>
+
+ <para>
+   In order to interact with an RBAC-enabled cluster as an administrator, one
+   needs to have cluster-admin privileges. By default, when easyCerts is
+   enabled, a cluster-admin kubeconfig file is generated and linked into
+   <literal>/etc/kubernetes/cluster-admin.kubeconfig</literal> as determined by
+   <xref linkend="opt-services.kubernetes.pki.etcClusterAdminKubeconfig"/>.
+   <literal>export KUBECONFIG=/etc/kubernetes/cluster-admin.kubeconfig</literal>
+   will make kubectl use this kubeconfig to access and authenticate the cluster.
+   The cluster-admin kubeconfig references an auto-generated keypair owned by
+   root. Thus, only root on the kubernetes master may obtain cluster-admin
+   rights by means of this file.
+ </para>
+
+</chapter>
diff --git a/nixos/doc/manual/release-notes/rl-1903.xml b/nixos/doc/manual/release-notes/rl-1903.xml
index 6f78983d48294..269f27f74fbde 100644
--- a/nixos/doc/manual/release-notes/rl-1903.xml
+++ b/nixos/doc/manual/release-notes/rl-1903.xml
@@ -54,6 +54,13 @@
      </itemizedlist>
      <para>to <literal>false</literal> and enable your preferred display manager.</para>
     </note>
+     <para>
+       A major refactoring of the Kubernetes module has been completed.
+       Refactorings primarily focus on decoupling components and enhancing
+       security. Two-way TLS and RBAC has been enabled by default for all
+       components, which slightly changes the way the module is configured.
+       See: <xref linkend="sec-kubernetes"/> for details.
+     </para>
    </listitem>
   </itemizedlist>
  </section>
@@ -564,6 +571,40 @@
      provisioning.
     </para>
    </listitem>
+   <listitem>
+     <para>
+       The use of insecure ports on kubernetes has been deprecated.
+       Thus options:
+       <varname>services.kubernetes.apiserver.port</varname> and
+       <varname>services.kubernetes.controllerManager.port</varname>
+       has been renamed to <varname>.insecurePort</varname>,
+       and default of both options has changed to 0 (disabled).
+     </para>
+    </listitem>
+    <listitem>
+      <para>
+        Note that the default value of
+        <varname>services.kubernetes.apiserver.bindAddress</varname>
+        has changed from 127.0.0.1 to 0.0.0.0, allowing the apiserver to be
+        accessible from outside the master node itself.
+        If the apiserver insecurePort is enabled,
+        it is strongly recommended to only bind on the loopback interface. See:
+        <varname>services.kubernetes.apiserver.insecurebindAddress</varname>.
+      </para>
+    </listitem>
+    <listitem>
+      <para>
+        The option <varname>services.kubernetes.apiserver.allowPrivileged</varname>
+        and <varname>services.kubernetes.kubelet.allowPrivileged</varname> now
+        defaults to false. Disallowing privileged containers on the cluster.
+      </para>
+    </listitem>
+    <listitem>
+      <para>
+        The kubernetes module does no longer add the kubernetes package to
+        <varname>environment.systemPackages</varname> implicitly.
+      </para>
+    </listitem>
   </itemizedlist>
  </section>
 </section>