kubeflow Istio configuration for trustworthy JWTs on rancher 2.x

Introduction:

For some reason some of the default feature gates are not turned on in rancher. 

So deploying Kubeflow or any workload that uses Istio version 1.3.1 with SDS enabled you need to enable TokenRequest and TokenRequestProjection.

Issue symptoms:

1. istio-pilot and everything dependent will fail to start in Kubeflow deployment.
2. pod events / log similar to "MountVolume.SetUp failed for volume "istio-token" : failed to fetch token: the API server does not have TokenRequest endpoints enabled"

How to prepare Rancher for Istio 1.3 and up (tested on 2.x)

Option 1, use server configuration file (yaml edit)

1. Login to your Rancher2.0 UI
2. Select relevant cluster
3. Click on options and edit
4. On cluster options choose "Cluster Options" and edit ad YAML
5. go to: "kube-api:"
   and add :
   extra_args:
          service-account-issuer: "kubernetes.default.svc"
          service-account-signing-key-file: "/etc/kubernetes/ssl/kube-service-account-token-key.pem"
6. Save the file / configuration

Cluster will reconfigure. 

Option 2, feature gates flags via Rancher API

Follow the instructions in this thread.

references:

Kubernetes documentation

Istio blog

Rancher git issue