I have a PX Essentials server running that failed to renew it’s license in the last 24 hours and therefore the license expired. It was working fine up until now.
The error is “License is expired, Cannot reach PX-Central for license renewal”
I am able to ping and Curl both licensing endpoints from the nodes without issue.
How do I force PX to refresh its license?
Can you paste the alerts generated by Portworx? The alerts will show why license refresh failed.
pxctl alerts show
You can force refresh a license by restarting Portworx on one of the nodes. You can restart Portworx on a node by labeling them in kubernetes
kubectl label nodes <node-name> px/service=restart
unfortunately I recently did a pxctl alerts purge (a couple of hours ago) so, I can’t give you an error as no new errors show up for license. the error that was there was talking about connection being refused to the loggly server on port 80. However I am able to ping and curl that server successfully.
I have set the label on nodes but it had no effect and the nodes are still with that label. I have also restored all of the Portworx pods and the services on the nodes by using systemctl restart portworx
Still the same error.
If this matters, this is a bare metal cluster with Flatcar OS on it.
So even after restart of Portworx the license is not refreshed? Did it generate a new alert now ?
No sir, here is what it looks now. The errors are form me restarting services and one K8S PVC that keeps failing to bind due to the license issue.
Oh, a new error just showed up:
CLUSTER MeteringAgentCritical ALARM 2m3s Unable to reach to billing server: Cannot send metric to the billing endpoint: Get http://logs-01.loggly.com/inputs/225dfd1d-30d9-4fa2-a60b-ad7767b928b4/tag/http/: dial tcp 0.0.0.0:80: connect: connection refused
Can you try the curl command from within the Portworx container?
On a node, you can exec into the Portworx container using
/opt/pwx/bin/runc exec -t portworx bash
Ran it, got the same output. response:ok
dgrekov@node1 ~ $ sudo /opt/pwx/bin/runc exec -t portworx bash
root@node1:/# curl http://logs-01.loggly.com/inputs/225dfd1d-30d9-4fa2-a60b-ad7767b928b4/tag/http -v
* Trying 52.73.201.133...
* Connected to logs-01.loggly.com (52.73.201.133) port 80 (#0)
> GET /inputs/225dfd1d-30d9-4fa2-a60b-ad7767b928b4/tag/http HTTP/1.1
> Host: logs-01.loggly.com
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.12.1
< Date: Thu, 18 Jun 2020 17:45:13 GMT
< Content-Type: text/html
< Content-Length: 19
< Connection: keep-alive
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Headers: Authorization,Host,Content-Type,X-Forwarded-For,X-LOGGLY-TAG,X-Real-IP
<
* Connection #0 to host logs-01.loggly.com left intact
{"response" : "ok"}Since one of the nodes is going to refresh the license, can you run the above tests from all your nodes?
Also do you have a proxy setup in your cluster?
Can you share PX diagnostics with us?
pxctl service diags -a -c
The above command will generate a tarball under /var/cores on all the nodes.
ok, got something … one node is reporting
curl http://logs-01.loggly.com/inputs/225dfd1d-30d9-4fa2-a60b-ad7767b928b4/tag/http -v
* Trying 0.0.0.0:80...
* TCP_NODELAY set
* connect to 0.0.0.0 port 80 failed: Connection refused
* Trying :::80...
* TCP_NODELAY set
* connect to :: port 80 failed: Connection refused
* Failed to connect to logs-01.loggly.com port 80: Connection refused
* Closing connection 0
curl: (7) Failed to connect to logs-01.loggly.com port 80: Connection refused
I’ll troubleshoot this and will get back to you. Also generating the tarball as well.
Ok, I was able to fix that node, issue was with a bad DNS server. Now I will wait an hour to see if it tries to get a license again?
That fixed it. Thank you for your help!