PX-Essential - ERROR: License is expired, PX-Central server not reachable

#1

About one hour into configuring a new cluster and the message pops up about the license.
Managed to bring 2 pods without issues, all mounted ok

It picked up a license initially and all nodes formed a quorum : PX-Essential (lease renewal in 23h, 37m)

One hour later (with no config changes) afterward:

Cannot reach Billing endpoint: Cannot reach TCP endpoint pxessentials.portworx.com:http in timeout 1, Err: dial tcp: i/o timeout

curl to pxessentials.portworx.com: returns 308 Permanent Redirect<

Any idea what is going on?

#2

Thank you for reporting this. We’ll take a look.

@Tapas-Sharma can you help?

#3

Hey,
Thank you for reaching out, we have a fix for this in the latest release for PX-Essentials, i.e. 2.5.5
If you can share your spec with us, I can verify which version you are trying to install and help guide you to update it.
Also, if you could share more details about your architecture, it will help us debug this issue quickly.
Thanks,
Tapas Sharma

#4

Hi, thanks for your reply.
My setup was straightforward.All latest versions. Kube on 1.18.6, 5 nodes with 3 etcd dedicated devices. All was running fine on previous setup with combined kvdb volumes, but when tried to setup a new cluster things break right from the start. After initial setup few hours the license expired. Tried twice from brand new cluster. The same result. No idea how to troubleshoot as there is no docs about detail in depth process.

#5

Do you have one or two Portworx Essential clusters active at the same time? Only one is allowed and you have to unlink your first cluster to use a new one as shown here -https://docs.portworx.com/portworx-install-with-kubernetes/operate-and-maintain-on-kubernetes/troubleshooting/unlink-a-portworx-essentials-cluster/

#6

I’m aware of that , the cluster always one at the time , else the newly created won’t even register and allow me to create resources etc… Magically after an hour or so the error message about the timeout to the billing point and that was game over from this point on. At first instance I waited more than 24 hrs, forced bounce all nodes, reboot all nodes etc etc. Once the error message kicked it this was end of the road and no way to clear it.

#7

I’ve just tested it again,
Exactly after 1 hr the License: PX-Essential changes from PX-Essential (lease renewal in 23h, 0m) to License is expired, PX-Central server not reachable

image
image1741×1007 376 KB

Is this a software bug? If yes, when do you think it will be fixed?

#8

Hi @darioneto, essentials requires access to the internet to function… so that it can check it’s license criteria. The message seems to indicate that it cannot reach the central.portworx.com license server.

#9

I can reach this endpoint, unless custom outbound ports needs to be open to this destination? On the FW there is nothing blocking that traffic from the logs.

image
image3007×868 240 KB

#10

this clearly indicate that Portworx endpoint has issues

404 from their server not gonna work

image

#11

@darioneto The Portworx endpoint is setup such that you cannot reach the head URL. To check the health you can run the following curl from your setup
curl -k https://pxessentials.portworx.com/osb/ping
As of writing this post the service is running and serving requests. Please update with the output of the above ping.
Also, what is the output of
pxctl alert show

#12

Hi there,
I’ve wiped the px cluster already. Unfortunately can’t afford weeks or months of troubleshooting.
For now moved on to test another project. If anything changes will let you know.

#13

Hi,

I am getting the same error message even after forcibly refresh a license.

Also, there is no issue in connecting these links

Could you please help?

#14

Hi Hardeep,
Can you login to one of the portworx pods and check the following
curl -k https://pxessentials.portworx.com/osb/ping
As of writing this post the service is running and serving requests. Please update with the output of the above ping.
Also, what is the output of
/opt/pwx/bin/pxctl alerts show