"License has expired" with Essentials package

I am trying to install kafka and zookeeper on the kubernetes cluster defined by the following command

gcloud beta container --project “my-project” clusters create “gke-px” --zone “us-central1-a” --no-enable-basic-auth --cluster-version “1.14.10-gke.27” --machine-type “n1-standard-4” --image-type “UBUNTU” --disk-type “pd-ssd” --disk-size “100” --metadata disable-legacy-endpoints=true --scopes “https://www.googleapis.com/auth/cloud-platform” --num-nodes “3” --enable-stackdriver-kubernetes --enable-ip-alias --network “projects/my-project/global/networks/default” --subnetwork “projects/my-project/regions/us-central1/subnetworks/default” --default-max-pods-per-node “110” --enable-autoscaling --min-nodes “3” --max-nodes “5” --no-enable-master-authorized-networks --addons HorizontalPodAutoscaling,HttpLoadBalancing,KubernetesDashboard,Istio,GcePersistentDiskCsiDriver --istio-config auth=MTLS_PERMISSIVE --enable-autoupgrade --enable-autorepair

as per the corresponding article however I am met with an increadibly frustrating error when starting the zookeeper stateful set as described in the following kubectl describe output:

Events:
Type Reason Age From Message


Normal NotTriggerScaleUp 17m (x6 over 18m) cluster-autoscaler pod didn’t trigger scale-up (it wouldn’t fit if a new node is added):
Normal NotTriggerScaleUp 4m19s (x72 over 19m) cluster-autoscaler pod didn’t trigger scale-up (it wouldn’t fit if a new node is added): 1 in backoff after failed scale-up
Warning FailedScheduling 68s (x19 over 19m) stork pod has unbound immediate PersistentVolumeClaims (repeated 3 times)

I then traced the error to describe pvc (persistent volume claim of the above pod) and was met with the following error messages:

Events:
Type Reason Age From Message


Warning ProvisioningFailed 9m20s (x14 over 24m) persistentvolume-controller Failed to provision volume with StorageClass “portworx-sc”: rpc error: code = Internal desc = Failed to create volume: Volume (Name: pvc-9e3cc75a-7ab7-11ea-95f1-42010a8000af) create failed: License has expired
Warning ProvisioningFailed 7s (x11 over 8m22s) persistentvolume-controller Failed to provision volume with StorageClass “portworx-sc”: rpc error: code = Internal desc = Failed to create volume: Volume (Name: pvc-9e3cc75a-7ab7-11ea-95f1-42010a8000af) create failed: License has expired
Mounted By: zk-0

As noted by the attached image, the PX license I have is for the Essentials tier. However I still receive the above message. Could someone please advise, thanks.

Hi, Please post output of your ‘pxctl status’ for someone from the community support team to look at. Thanks.

The Portworx alerts will indicate why exactly license has expired. You can run the following command to get the alerts

PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[0].metadata.name}')
kubectl exec $PX_POD -n kube-system -- /opt/pwx/bin/pxctl alerts show

You can also check kubernetes events for the latest set of Portworx alerts.

kubectl get events | grep portworx

‘dial tcp connection refused’ is fixed, due to docker proxy. run pvc create at 14:00, no dial tcp connection refused.

/opt/pwx/bin/pxctl alerts show

Type ID Resource Severity Count LastSeen FirstSeen Description
NODE NodeStateChange 0bedbc3f-d437-4e71-b14d-ab74153ca01b ALARM 1 Sep 24 07:43:58 UTC 2020 Sep 24 07:43:58 UTC 2020 Node is not in quorum. Waiting to connect to peer nodes on port 9002.
NODE NodeStateChange 7d368c94-17e9-4ba4-9ecf-3f47a4987ad0 ALARM 1 Sep 24 07:46:57 UTC 2020 Sep 24 07:46:57 UTC 2020 Node is not in quorum. Waiting to connect to peer nodes on port 9002.
NODE NodeStateChange b4715367-6a83-439a-bb03-e6723677dbd5 ALARM 1 Sep 24 07:47:11 UTC 2020 Sep 24 07:47:11 UTC 2020 Node is not in quorum. Waiting to connect to peer nodes on port 9002.
CLUSTER LicenseExpired px-cluster-b62327f4-67a7-4794-868a-3c8893d1fcbe ALARM 1 Sep 24 13:44:13 UTC 2020 Sep 24 13:44:13 UTC 2020 PX-Essential license expired 0 days, 0:59 ago
VOLUME VolumeCreateFailure pvc-060f25ad-3e6f-4b87-a71b-48f45b94d518 ALARM 9 Sep 28 13:39:28 UTC 2020 Sep 28 13:35:45 UTC 2020 Volume (Name: pvc-060f25ad-3e6f-4b87-a71b-48f45b94d518) create failed: License has expired
VOLUME VolumeCreateFailure pvc-5d396fa6-da59-48d3-b2a1-22551e83f0eb ALARM 3 Sep 28 13:41:28 UTC 2020 Sep 28 13:40:58 UTC 2020 Volume (Name: pvc-5d396fa6-da59-48d3-b2a1-22551e83f0eb) create failed: License has expired
NODE CallHomeFailure 0bedbc3f-d437-4e71-b14d-ab74153ca01b ALARM 52 Sep 28 13:45:05 UTC 2020 Sep 24 07:44:15 UTC 2020 Error while calling home: node (10.230.13.253) node (10.230.13.253) Post http://logs-01.loggly.com/inputs/*token*/tag/http/: dial tcp 54.236.129.133:80: connect: connection refused
CLUSTER MeteringAgentCritical ALARM 102 Sep 28 13:46:10 UTC 2020 Sep 24 07:44:14 UTC 2020 Unable to reach to billing server: Cannot reach Billing endpoint: Cannot reach TCP endpoint pxessentials.portworx.com:http in timeout 1, Err: dial tcp 18.211.233.187:80: connect: connection refused
VOLUME VolumeCreateFailure pvc-c9848479-ed04-455a-9fce-4f36eedaaf93 ALARM 15 Sep 28 14:00:28 UTC 2020 Sep 28 13:43:16 UTC 2020 Volume (Name: pvc-c9848479-ed04-455a-9fce-4f36eedaaf93) create failed: License has expired
VOLUME VolumeCreateFailure pvc-ae3050d6-6d3f-43bb-8844-8728aec7753f ALARM 1 Sep 28 14:05:29 UTC 2020 Sep 28 14:05:29 UTC 2020 Volume (Name: pvc-ae3050d6-6d3f-43bb-8844-8728aec7753f) create failed: License has expired

@kaitaklam What version of PX are you running?

If the connection between your nodes and PX-Central is fixed, then the nodes should auto-recover within 24 hours. Do you still see the license expired error ?