Hi, I’m tring to install Portworx on OpenShift 4.5.18 using operator.
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
One of five container doesn’t start correctly.@ccr-svil-dc2hv-worker-westeurope1-k559k systemd[1]: Started libcontainer container 8a41879d7e153b4977cbb6e23dc4eda7a746e2d4769a22fb4e65b95a20cf5bc3.@ccr-svil-dc2hv-worker-westeurope1-k559k portworx[1888528]: time=“2020-11-18T11:35:15Z” level=warning msg=“503 Node status not OK (STATUS_INIT)” Driver=“Cluster API” ID=nodeHealth Request=“Cluster API”@ccr-svil-dc2hv-worker-westeurope1-k559k portworx[1888528]: time=“2020-11-18T11:35:25Z” level=warning msg=“503 Node status not OK (STATUS_INIT)” Driver=“Cluster API” ID=nodeHealth Request=“Cluster API”@ccr-svil-dc2hv-worker-westeurope1-k559k portworx[1888528]: time=“2020-11-18T11:35:35Z” level=warning msg=“503 Node status not OK (STATUS_INIT)” Driver=“Cluster API” ID=nodeHealth Request=“Cluster API”@ccr-svil-dc2hv-worker-westeurope1-k559k portworx[1888528]: time=“2020-11-18T11:35:45Z” level=warning msg=“503 Node status not OK (STATUS_INIT)” Driver=“Cluster API” ID=nodeHealth Request=“Cluster API”@ccr-svil-dc2hv-worker-westeurope1-k559k portworx[1888528]: time=“2020-11-18T11:35:55Z” level=warning msg=“503 Node status not OK (STATUS_INIT)” Driver=“Cluster API” ID=nodeHealth Request=“Cluster API”@ccr-svil-dc2hv-worker-westeurope1-k559k portworx[1888528]: time=“2020-11-18T11:36:05Z” level=warning msg=“503 Node status not OK (STATUS_INIT)” Driver=“Cluster API” ID=nodeHealth Request=“Cluster API”
How to resolve this?
aleks
November 19, 2020, 2:40pm
2
The output from the pods themselves will not be sufficient - these are monitoring components and the Portworx software has to get to a certain point in order for the pod’s collection of logs to be functional. This appears to not yet be the case here.
Can you instead gather the output of journalctl -au portworx* from one of the nodes themselves please? Then make a snippet on a pastebin and provide a link here - You will need to log in as either root (if using EL8) or the core user and sudo to root if using RHCOS (the latter of which we’ve seen the most number of issues with due to the rate at which the kernel version changes (PX has support for only certain versions).
Hi thanks for your response.
$ cat /etc/redhat-release
$ uname -a#1 SMP Fri Oct 16 13:38:49 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux
$ journalctl -au portworx*
Nov 17 16:24:29 ccr-svil-dc2hv-worker-westeurope1-k559k portworx[29509]: time=“2020-11-17T16:24:29Z” level=info msg="Rootfs found at /var/opt/pwx/oci>
16:24:31,698 INFO spawned: ‘lttng’ with pid 258
aleks
November 19, 2020, 5:04pm
5
It will be difficult to post this in this forum as this log is likely very long, can you please utilize a pastebin service (or a gist on github ) and post a link to the paste instead?
Sure:
Thanks in advance
aleks
November 19, 2020, 5:34pm
7
Hi Vincenzo,
I just had a look at the pastebin link, but it also looks truncated/cut off short, on that pastebin - can you re-capture it from the node, or perhaps a different node if the issue affects more than one node?
In the one you provided, Portworx is still initializing and between the first message to last is only a span of two minutes - Portworx typically needs anywhere from 2-5 minutes to initialize/install the first time (depending on the environment and configuration).
Hi Aleks,
Thanks
aleks
November 20, 2020, 12:13pm
9
I had a look, the other node’s logs just show it regularly updating its essentials license hourly (as expected) and no errors. Though it’s hard to tell, the other node appears that Portworx would be functioning normally (although as no other messages are seen, it may just be idle and not doing anything just yet). Can you run pxctl status on that other node and let us know what you see? Also ideally we’d want to capture diagnostics from any node with an issue and gather these files, the procedure for that is: pxctl sv diags -a -f (run on each node) and the generated file will be in /var/cores/diags.tar.gz - you may need to post links to a file sharing service as these are binary files and may not be compatible with pastebins (and this forum only allows images to be posted)
Hi Aleks,
The problem was I initially deployed 8 worker node, then I scaled down to 5 nodes.pxctl status on a new node and I saw:Status: PX is operational License: PX-Essential (ERROR: License is expired, Another cluster registered with the same userID. Unlink the previously registered cluster on PX-Central.)
So I decided to reinstall.
Best regards
aleks
November 20, 2020, 4:02pm
11
Glad you were able to sort out the cluster. We limit the number of Portworx Essential clusters to exactly one per PX-Central user, so if you try to set up a second cluster, the first cluster’s “license” may be invalidated (or vice versa). If you really needed another cluster, there’s little to stop you from creating a different user on px-central with another email, but we don’t advocate that as a proper solution (those limitations are there as we prefer people upgrade to Portworx Enterprise if they need unhindered flexibility (and the support there is 24x7x365 via Jira rather than a forum that is largely intended to be community provided (as the Essentials product is relatively new, we still have engineers keep an eye here from time to time but it is not as high a priority to do so). Thanks for understanding and if you need anything further (such as more info on Enterprise), please ask on here.