Saturday, January 24, 2015

Mount a partition from a Proxmox hardware node to openvz container(s)

Sometime we need to access another partition outside our OpenVZ container. In Proxmox PVE you can do this easily, but require manual configuration (can not be done from Proxmox admin) and will require your OpenVZ container reboot.

[STEP 1] create the destination directory in the container:

in this example, I am mounting /mnt/<name> from hardware node to /data_ssd in container

mkdir /data_ssd
chmod -R 777 /data_ssd

shutdown the container:

shutdown -h now

[STEP 2] prepare the mount to whatever partition you would like to access (ON HARDWARE NODE)

on the proxmox HN:

mkdir /mnt/<name>    (name example 'ssd_shared')
chmod -R 777 /mnt/<name>

in this example I am mounting an LVM partition called lvm_ssd.
lets make sure the partition will be mounted automatically on HN boot.

blkid            (copy the UUID of the newly created LVM partition)

Then make sure your fstab file has the following entry:

nano /etc/fstab

UUID=<uuid> /mnt/<name> ext4 defaults,noatime,nodiratime,noacl,data=writeback,barrier=0,nobh,discard,errors=remount-ro 0 2

To make sure the fstab setting work, execute the command below to auto-mount all fstab entries

mount -a

[STEP 3] setup the container's configuration file to allow container to access the shared mount

cd /etc/pve/openvz
nano <ctid>.mount                 (ctid is the container ID for example 112)

#!/bin/bash
. /etc/vz/vz.conf
. ${VE_CONFFILE}
SRC=/mnt/<name>
DST=/data_ssd
if [ ! -e ${VE_ROOT}${DST} ]; then mkdir -p ${VE_ROOT}${DST}; fi
mount -o noatime -n -t simfs ${SRC} ${VE_ROOT}${DST} -o ${SRC}


[STEP 4] Test by starting the container

once container has started back up, execute the 'mount' command to see all your mounted partition in your container. Mine looks like this:

/mnt/md0/private/112 on / type simfs (rw,relatime)
/mnt/ssd_shared on /data_ssd type simfs (rw,noatime)
proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
none on /dev/pts type devpts (rw,nosuid,noexec,relatime,mode=600,ptmxmode=000)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
none on /run type tmpfs (rw,nosuid,noexec,relatime,size=419432k,mode=755)
none on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
none on /run/shm type tmpfs (rw,relatime)

The one I highlighted in yellow is the /mnt/ssd_shared (from HN) that was already mounted to /data_ssd inside container 112.

You can also use 'df' (disk free) command to check the available disk space, mine looks like this:

Filesystem      1K-blocks      Used Available Use% Mounted on
/dev/simfs      167772160 113525032  54247128  68% /
/dev/simfs       16513960   4398600  11276500  29% /data_ssd
none               419432       996    418436   1% /run
none                 5120         0      5120   0% /run/lock
none              2097152         0   2097152   0% /run/shm


You are done!

Add SMART smartmontools SATA smart drive monitoring tool smartd to Proxmox PVE hardware node

I recommend always installing SMARTMONTOOLS to any server with physical disc hard drive(s). Meaning if you have a spinning hard drive (not SSD) you will eventually have to replace it, because it will fail soon or later.

I hate to be surprised when it is too late to replace a failing hard drive. SmartMonTools stands for SMART Monitoring Tool, will query your hard drive for its health status.  If you do this daily and setup an alert system to your email, you will most likely avoid a bad surprise in the future.

I highly recommend installing and using this smartmontools monitoring and alert for any server.

Here is how I have deployed on EACH on of my server:

THIS CAN BE INSTALLED ON ANY BARE METAL SERVER, FOR PROMOX PVE, THIS MEANS YOUR HARDWARE NODE.


1. install smartmontools

aptitude update && aptitude -y install smartmontools


2. edit default daemon start configuration:

nano /etc/default/smartmontools

unremark all commented lines

enable_smart="/dev/sda /dev/sdb /dev/sdc"
start_smartd=yes
smartd_opts="--interval=28800"

3. edit smartd.conf  (in this example I have 3 SATA drives: sda, sdb, sdc)

nano /etc/smartd.conf

/dev/sda -d sat -a -s L/../../7/4 -m john@smith.com,jack@jill.com
/dev/sdb -d sat -a -s L/../../7/5 -m john@smith.com,jack@jill.com
/dev/sdc -d sat -a -s L/../../7/6 -m john@smith.com,jack@jill.com

The above example will do the following:
1. scan sda at 4am Saturday
2. scan sdb at 5am Saturday
3. scan sdc at 6am Saturday

Email alert will be sent to john@smith.com and jack@jill.com if there is something wrong.

NOTE about the -s parameter:

The second from the last is the DAY parameter:

            Sunday is day # 1
            Monday is day # 2
            ...
            Saturday is day #7


4. restart smartmontools

/etc/init.d/smartmontools restart

5. check current HEALTH status:

smartctl -H /dev/sda
smartctl -H /dev/sdb
smartctl -H /dev/sdc

DONE!

Install ntpdate ntp client to update time from openvz proxmox CT container

By default OpenVZ container does not allow ntpdate operation to set date + time.

If you are getting error message like:

Can't adjust the time of day: Operation not permitted

on the Proxmox hardware node (HN) do the following to override default Proxmox OpenVZ setting to allow sys_time override:

vzctl stop 108
vzctl set 108 --capability sys_time:on --save
vzctl start 108

apt-get update
apt-get install ntpdate



Add the following to your crontab to auto-update date-time daily:

crontab -e

22 2 * * * /usr/sbin/ntpdate 192.168.1.1 > /log/ntpdate.log

How to Re-Sync Lost out of sync Proxmox Cluster

If cluster gets out of sync with messages like 'failed sending messages' in syslog or Quorum failed during pve-cluster restart, the right solution is to stop the entire cluster, then start it all again.

another symptom of this problem is when the hardware node icons in your Proxmox control panel are showing RED instead of green.

Do the following:

WARNING: you should backup all your Containers and VMs just in case. This works for me, but use at your own risk.

1. STOP all pve-clusters first

2. then stop cman

3. after all pve-clusters are stopped
(you will need to enter root password everytime script need to connect via ssh to start cman for each node.)

Containers are not affected (keeps running unaffected) during this process!

Proxmox PVE install boot options

I prefer to set my initial root partition size to be smaller than the default Proxmox setting.

Here are some example usage and list of other setup boot options:

Examples:

linux ext4 hdsize=8
linux ext4 hdsize=10
linux ext4 hdsize=12

  • linux ext4 – sets the partition format to ext4. The default is ext3.
  • hdsize=nGB – this sets the total amount of hard disk to use for the Proxmox installation. This should be smaller than your disk size.
  • maxroot=nGB – sets the maximum size to use for the root partition. This is the max size so if the disk is too small, the partition may be smaller than this.
  • swapsize=nGB – sets the swap partition size in gigabytes.
  • maxvz-nGB – sets the maximum size in gigabytes that the data partition will be. Again, this is similar to maxroot and the final partition size may be smaller.
  • minfree=nGB – sets the amount of free space to remain on the disk after the Proxmox installation.

Upgrade Proxmox PVE HN hardware node from version 3.1.x to 3.3.x

*** MAKE SURE ALL CONTAINERS AND VMS ARE SHUTDOWN ***
*** MAKE BACKUP OF ALL CTS AND VMS ***


edit sources.list

nano /etc/apt/sources.list

add the following line in between debian main and debian security:

# PVE pve-no-subscription repository provided by proxmox.com, NOT recommended for production use
deb http://download.proxmox.com/debian wheezy pve-no-subscription

Remark this line to avoid error for subscription repo:

nano /etc/apt/sources.list.d/pve-enterprise.list
# deb https://enterprise.proxmox.com/debian wheezy pve-enterprise

Update packages list

apt-get update


Perform upgrade

apt-get dist-upgrade

Upgrade Proxmox hardware node from version 2.3.x to 3.1.x

*** MAKE SURE ALL CONTAINERS AND QEMU vms ARE SHUTDOWN ***
*** MAKE BACKUP OF ALL CONTAINERS AND QEMU VMS ***

cd /root


chmod +x pve-upgrade-2.3-to-3.0

./pve-upgrade-2.3-to-3.0

reboot

-----------------------

after reboot confirm to make sure your Proxmox version has been upgraded:

pveversion -v

(for version 3.3-5 - updated on 2014/11/02)

proxmox-ve-2.6.32: 3.3-138 (running kernel: 2.6.32-33-pve)
pve-manager: 3.3-5 (running version: 3.3-5/bfebec03)
pve-kernel-2.6.32-19-pve: 2.6.32-96
pve-kernel-2.6.32-33-pve: 2.6.32-138
pve-kernel-2.6.32-26-pve: 2.6.32-114
pve-kernel-2.6.32-18-pve: 2.6.32-88
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.7-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.10-1
pve-cluster: 3.0-15
qemu-server: 3.3-1
pve-firmware: 1.1-3
libpve-common-perl: 3.0-19
libpve-access-control: 3.0-15
libpve-storage-perl: 3.0-25
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-8
vzctl: 4.0-1pve6
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 2.1-10
ksm-control-daemon: 1.1-1
glusterfs-client: 3.5.2-1