Error in validating kubeconfig

PX-Backup
#1

Happy Friday All!

I’m running a AWS EKS instance and I am trying to install PX-Backup. I followed the installation instructions here: https://backup.docs.portworx.com/install/

I can log into the PX-Central GUI, when attempting to add a k8s cluster, I get an error “error in validating kubeconfig”.

Could use an assist to troubleshoot!

Via kubectl -n px-backup get pods I see all the pods except pxcentral-post-install-hook are running. Checking the logs I see OIDC errors:
OIDC client secret patch command status: 0, output: I1105 20:13:40.697919 20 cached_discovery.go:130] failed to write cache to /.kube/cache/discovery/10.100.0.1_443/servergroups.json due to mkdir /.kube: permission denied
I1105 20:13:40.700934 20 cached_discovery.go:87] failed to write cache to /.kube/cache/discovery/10.100.0.1_443/v1/serverresources.json due to mkdir /.kube: permission denied

Otherwise no glaring errors in the logs.

Appreciate help here,

Am

#2

Hi Arthur,

Can you confirm that you are using AWS cloud credentials with your kubeconfig? It is required for EKS clusters and that might be causing the auth to fail.

#3

Thanks for responding Prashant,

I believe I am, do you know how can I verify?

Am

#4

@Arthur_Miser This page has more information on how to add EKS clusters: https://backup.docs.portworx.com/use-px-backup/cluster/aws-eks/

You need to make sure you do the following:

  1. Create a CloudCredential using your AWS credentials: https://backup.docs.portworx.com/use-px-backup/credentials/aws/
  2. When adding the cluster, choose EKS as the “Kubernetes Service” and select the Credential you created above.

This is required since EKS uses AWS credentials to authenticate with the Kubernetes cluster.

#5

Evening Dinesh,

I’ve followed the outlined steps and I’m still getting the same error. I can confirm the account that I’m adding as a cloud account in Step #4 has enough EC2 & S3 permissions. I can log into S3 via S3 browser and create buckets etc.

FWIW I did log into the px-backup node and I’m seeing this error:

time=“2020-11-09T01:15:50Z” level=error msg="error validating kubeconfig: error initializing k8s instance: error getting cluster version: the server has asked for the client to provide credentials

Based on that error and some googling, my only thought here is that the role / credentials that I used for EKS cluster creation are different than AWS authentication?

Any ideas what I’m missing here?

Am

#6

Afternoon Folks,

Just an update here, I think I figured out what was causing the issue.

Looks like it was related to my EKS cluster configuration and permissions associated with authentication to AWS. My authentication was happening via role vs actual user. Ie. kubectl config view needed to have a real user vs assumed role in AWS.

As this is a test cluster I rebuild EKS using an actual user vs assumed role and I was able to successfully deploy PX-Backup.

I’m confident that this could have also been resolved by adding additional authorized users.

Am

#7

Hi @Arthur_Miser,

Thanks for the update. Good to know you were able to add the cluster.

Can you share the env variables keys you were passing in via the kubeconfig? I want to make sure we document this for other users too.

Thanks
Dinesh