"License has expired" with Essentials package

Portworx Essentials
1

I am trying to install kafka and zookeeper on the kubernetes cluster defined by the following command

gcloud beta container --project “my-project” clusters create “gke-px” --zone “us-central1-a” --no-enable-basic-auth --cluster-version “1.14.10-gke.27” --machine-type “n1-standard-4” --image-type “UBUNTU” --disk-type “pd-ssd” --disk-size “100” --metadata disable-legacy-endpoints=true --scopes “https://www.googleapis.com/auth/cloud-platform” --num-nodes “3” --enable-stackdriver-kubernetes --enable-ip-alias --network “projects/my-project/global/networks/default” --subnetwork “projects/my-project/regions/us-central1/subnetworks/default” --default-max-pods-per-node “110” --enable-autoscaling --min-nodes “3” --max-nodes “5” --no-enable-master-authorized-networks --addons HorizontalPodAutoscaling,HttpLoadBalancing,KubernetesDashboard,Istio,GcePersistentDiskCsiDriver --istio-config auth=MTLS_PERMISSIVE --enable-autoupgrade --enable-autorepair

as per the corresponding article however I am met with an increadibly frustrating error when starting the zookeeper stateful set as described in the following kubectl describe output:

Events:
Type Reason Age From Message

Normal NotTriggerScaleUp 17m (x6 over 18m) cluster-autoscaler pod didn’t trigger scale-up (it wouldn’t fit if a new node is added):
Normal NotTriggerScaleUp 4m19s (x72 over 19m) cluster-autoscaler pod didn’t trigger scale-up (it wouldn’t fit if a new node is added): 1 in backoff after failed scale-up
Warning FailedScheduling 68s (x19 over 19m) stork pod has unbound immediate PersistentVolumeClaims (repeated 3 times)

I then traced the error to describe pvc (persistent volume claim of the above pod) and was met with the following error messages:

Events:
Type Reason Age From Message

Warning ProvisioningFailed 9m20s (x14 over 24m) persistentvolume-controller Failed to provision volume with StorageClass “portworx-sc”: rpc error: code = Internal desc = Failed to create volume: Volume (Name: pvc-9e3cc75a-7ab7-11ea-95f1-42010a8000af) create failed: License has expired
Warning ProvisioningFailed 7s (x11 over 8m22s) persistentvolume-controller Failed to provision volume with StorageClass “portworx-sc”: rpc error: code = Internal desc = Failed to create volume: Volume (Name: pvc-9e3cc75a-7ab7-11ea-95f1-42010a8000af) create failed: License has expired
Mounted By: zk-0

As noted by the attached image, the PX license I have is for the Essentials tier. However I still receive the above message. Could someone please advise, thanks.

portworx
portworx1916×945 32.4 KB

2

Hi, Please post output of your ‘pxctl status’ for someone from the community support team to look at. Thanks.

3

The Portworx alerts will indicate why exactly license has expired. You can run the following command to get the alerts

PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[0].metadata.name}')
kubectl exec $PX_POD -n kube-system -- /opt/pwx/bin/pxctl alerts show

You can also check kubernetes events for the latest set of Portworx alerts.

kubectl get events | grep portworx
4

‘dial tcp connection refused’ is fixed, due to docker proxy. run pvc create at 14:00, no dial tcp connection refused.

/opt/pwx/bin/pxctl alerts show

Type ID Resource Severity Count LastSeen FirstSeen Description
NODE NodeStateChange 0bedbc3f-d437-4e71-b14d-ab74153ca01b ALARM 1 Sep 24 07:43:58 UTC 2020 Sep 24 07:43:58 UTC 2020 Node is not in quorum. Waiting to connect to peer nodes on port 9002.
NODE NodeStateChange 7d368c94-17e9-4ba4-9ecf-3f47a4987ad0 ALARM 1 Sep 24 07:46:57 UTC 2020 Sep 24 07:46:57 UTC 2020 Node is not in quorum. Waiting to connect to peer nodes on port 9002.
NODE NodeStateChange b4715367-6a83-439a-bb03-e6723677dbd5 ALARM 1 Sep 24 07:47:11 UTC 2020 Sep 24 07:47:11 UTC 2020 Node is not in quorum. Waiting to connect to peer nodes on port 9002.
CLUSTER LicenseExpired px-cluster-b62327f4-67a7-4794-868a-3c8893d1fcbe ALARM 1 Sep 24 13:44:13 UTC 2020 Sep 24 13:44:13 UTC 2020 PX-Essential license expired 0 days, 0:59 ago
VOLUME VolumeCreateFailure pvc-060f25ad-3e6f-4b87-a71b-48f45b94d518 ALARM 9 Sep 28 13:39:28 UTC 2020 Sep 28 13:35:45 UTC 2020 Volume (Name: pvc-060f25ad-3e6f-4b87-a71b-48f45b94d518) create failed: License has expired
VOLUME VolumeCreateFailure pvc-5d396fa6-da59-48d3-b2a1-22551e83f0eb ALARM 3 Sep 28 13:41:28 UTC 2020 Sep 28 13:40:58 UTC 2020 Volume (Name: pvc-5d396fa6-da59-48d3-b2a1-22551e83f0eb) create failed: License has expired
NODE CallHomeFailure 0bedbc3f-d437-4e71-b14d-ab74153ca01b ALARM 52 Sep 28 13:45:05 UTC 2020 Sep 24 07:44:15 UTC 2020 Error while calling home: node (10.230.13.253) node (10.230.13.253) Post http://logs-01.loggly.com/inputs/*token*/tag/http/: dial tcp 54.236.129.133:80: connect: connection refused
CLUSTER MeteringAgentCritical ALARM 102 Sep 28 13:46:10 UTC 2020 Sep 24 07:44:14 UTC 2020 Unable to reach to billing server: Cannot reach Billing endpoint: Cannot reach TCP endpoint pxessentials.portworx.com:http in timeout 1, Err: dial tcp 18.211.233.187:80: connect: connection refused
VOLUME VolumeCreateFailure pvc-c9848479-ed04-455a-9fce-4f36eedaaf93 ALARM 15 Sep 28 14:00:28 UTC 2020 Sep 28 13:43:16 UTC 2020 Volume (Name: pvc-c9848479-ed04-455a-9fce-4f36eedaaf93) create failed: License has expired
VOLUME VolumeCreateFailure pvc-ae3050d6-6d3f-43bb-8844-8728aec7753f ALARM 1 Sep 28 14:05:29 UTC 2020 Sep 28 14:05:29 UTC 2020 Volume (Name: pvc-ae3050d6-6d3f-43bb-8844-8728aec7753f) create failed: License has expired

5

@kaitaklam What version of PX are you running?

If the connection between your nodes and PX-Central is fixed, then the nodes should auto-recover within 24 hours. Do you still see the license expired error ?