kubeflow Istio configuration for trustworthy JWTs on rancher 2.x


For some reason some of the default feature gates are not turned on in rancher. 
So deploying Kubeflow or any workload that uses Istio version 1.3.1 with SDS enabled you need to enable TokenRequest and TokenRequestProjection.

Issue symptoms:

  1. istio-pilot and everything dependent will fail to start in Kubeflow deployment.
  2. pod events / log similar to "MountVolume.SetUp failed for volume "istio-token" : failed to fetch token: the API server does not have TokenRequest endpoints enabled"

How to prepare Rancher for Istio 1.3 and up (tested on 2.x)

Option 1, use server configuration file (yaml edit)

  1. Login to your Rancher2.0 UI
  2. Select relevant cluster
  3. Click on options and edit
  4. On cluster options choose "Cluster Options" and edit ad YAML
  5. go to: "kube-api:"
    and add :
           service-account-issuer: "kubernetes.default.svc"
           service-account-signing-key-file: "/etc/kubernetes/ssl/kube-service-account-token-key.pem"
  6. Save the file / configuration
Cluster will reconfigure. 

Option 2, feature gates flags via Rancher API

Follow the instructions in this thread.



Popular posts from this blog

How to set or disable auto-logout on bash shell

Setting up updates with public yum server for Oracle Enterprise Linux

Setting Filezilla Server on Amazon EC2 instance with passive ftp