Wednesday, August 26, 2020

kubeflow Istio configuration for trustworthy JWTs on rancher 2.x


For some reason some of the default feature gates are not turned on in rancher. 
So deploying Kubeflow or any workload that uses Istio version 1.3.1 with SDS enabled you need to enable TokenRequest and TokenRequestProjection.

Issue symptoms:

  1. istio-pilot and everything dependent will fail to start in Kubeflow deployment.
  2. pod events / log similar to "MountVolume.SetUp failed for volume "istio-token" : failed to fetch token: the API server does not have TokenRequest endpoints enabled"

How to prepare Rancher for Istio 1.3 and up (tested on 2.x)

Option 1, use server configuration file (yaml edit)

  1. Login to your Rancher2.0 UI
  2. Select relevant cluster
  3. Click on options and edit
  4. On cluster options choose "Cluster Options" and edit ad YAML
  5. go to: "kube-api:"
    and add :
           service-account-issuer: "kubernetes.default.svc"
           service-account-signing-key-file: "/etc/kubernetes/ssl/kube-service-account-token-key.pem"
  6. Save the file / configuration
Cluster will reconfigure. 

Option 2, feature gates flags via Rancher API

Follow the instructions in this thread.


solving error: Your current user or role does not have access to Kubernetes objects on this EKS cluster.

Trying to access EKS cluster with kubectl you might get an error similar to: Your current user or role does not have access to Kubernetes ob...