Why did my node go down?

There are several potential causes for a node in your Pextra CloudEnvironment® cluster to suddenly go down. Manual intervention may be required to diagnose and resolve the issue. In most cases, the node will self-recover after the underlying problem is fixed.

note

It may not be possible to access the node remotely if it is down due to hardware failure. Physical access or out-of-band management tools (like IPMI, iLO, or DRAC) may be required.

Resource Exhaustion

Running out of critical resources like CPU, memory, or disk space can cause a node to crash or freeze. You may need to consider upgrading hardware resources if resource exhaustion is a recurring issue.

Disk Space

warning

Do not, under any circumstances, delete system files or directories unless you are absolutely sure of their purpose. This includes directories like /bin, /sbin, /usr, /etc, and /var. This will destroy your installation and may lead to data loss.

Check disk space usage on all mounted partitions:
```
df -h
```

If any partitions are at or near 100% usage, take immediate action to free up space.

Find large files consuming disk space:
```
du -ah / | sort -rh | head -n 20
```
Clear cache and temporary files:
```
apt clean
```

Remove old log files or archive them to free up space:

find /var/log -type f -name "*.log" -exec rm -f {} \;

Memory

Check memory usage:
```
free -h
```

If memory usage is critically high, consider adding more RAM or stopping unnecessary instances.

Hardware Failures

Hardware issues like disk failures, memory errors, or power supply problems can lead a node to fail and become unresponsive. If any component is identified as failing, schedule replacement immediately to minimize downtime.

Check for hardware errors:

dmesg | grep -i error
tail -n 100 /var/log/syslog

Memory (RAM) test using Memtest86+ can identify RAM issues. You may run it from a bootable USB.

Inspect disk health:

smartctl -a /dev/sdX	# Replace /dev/sdX with your disk identifier

Review power supply and cooling systems to ensure they are functioning correctly.

Network Issues

Network connectivity problems can result from interface failures, network partitions, or misconfigurations. Verify network status and connectivity to other nodes in the cluster.

Check network interface status:
```
ip a
```
Test connectivity to other nodes:
```
ping <other-node-ip>
```
Review network configuration and logs for errors.
```
cat /var/log/syslog | grep -i network
```
Restart network services if necessary:
```
systemctl restart systemd-networkd
```

Software Bugs or Misconfigurations

Although it is highly unlikely, bugs in the operating system or misconfigurations can lead to instability and crashes. To update your system, refer to the System Upgrade section.

Keyboard shortcuts