Saturday, November 13, 2021

AVERMEDIA LIVE STREAMER WEBCAM

 Quick and honest review of the AVERMEDIA LIVE STREAMER WEBCAM.

This avermedia webcam is one of the best valued webcam around. It offers all the standard features you will ever need for just under $40. It is a 1080p FULL HD webcam with microphone. It is the perfect zoom, web conference, online meeting camera! If you want to buy this Avermedia Live Streamer Webcam please use my link: 👉 👉 ✅ ✅ https://amzn.to/3w1aofu Please LIKE and SUBSCRIBE! #avermedia #webcam #zoomcam

Sunday, November 7, 2021

How to restart proxmox cluster services (also will reconnect to cluster)

If your node has been disconnected from your Proxmox Cluster (PVE Cluster). You may want to rejoin without rebooting.

Here are the commands you should execute (no pause needed in between command):

killall -9 corosync
systemctl restart pve-cluster
systemctl restart pvedaemon
systemctl restart pveproxy

systemctl restart pvestatd 



Friday, November 5, 2021

How to fix this error: E: Repository 'http://security.debian.org buster/updates InRelease' changed its 'Suite' value from 'stable' to 'oldstable'

If you are getting this error:

E: Repository 'http://security.debian.org buster/updates InRelease' changed its 'Suite' value from 'stable' to 'oldstable'

You can fix it using the following command: 

apt-get --allow-releaseinfo-change update

apt-get update


Wednesday, November 3, 2021

How to troubleshoot / debug "lxc_init failed to run lxc.hook.pre-start for container ###"?

 Are you seeing this error while starting your Proxmox container?


lxc_init: 797 Failed to run lxc.hook.pre-start for container "###"

__lxc_start: 1896 Failed to initialize container "###"



Here is the command to find out more (debug to LOG) what is going on...


lxc-start -n ### -F -lDEBUG -o your_log_file.log


replace the ### with your Container ID 

then examine the content of 'your_log_file.log'

cat your_log_file.log


Output of log file:

In my case my container failed to start because I ran out of disk space (disk quota limit exceeded). here is the content of my log file:


lxc-start: 157: conf.c: run_buffer: 323 Script exited with status 1

lxc-start: 157: start.c: lxc_init: 797 Failed to run lxc.hook.pre-start for container "157"

lxc-start: 157: start.c: __lxc_start: 1896 Failed to initialize container "157"

lxc-start: 157: tools/lxc_start.c: main: 308 The container failed to start

lxc-start: 157: tools/lxc_start.c: main: 314 Additional information can be obtained by setting the --logfile and --logpriority options

root@e8:/etc/pve/lxc# ls

157.conf  164.conf  168.conf  176.conf lxc-157.log

root@e8:/etc/pve/lxc# cat lxc-157.log

lxc-start 157 20211103125836.780 INFO     lsm - lsm/lsm.c:lsm_init:29 - LSM security driver AppArmor

lxc-start 157 20211103125836.788 INFO     conf - conf.c:run_script_argv:340 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "157", config section "lxc"

lxc-start 157 20211103125837.708 DEBUG    conf - conf.c:run_buffer:312 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 157 lxc pre-start produced output: unable to open file '/fastboot.tmp.32694' - Disk quota exceeded


lxc-start 157 20211103125837.714 DEBUG    conf - conf.c:run_buffer:312 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 157 lxc pre-start produced output: error in setup task PVE::LXC::Setup::pre_start_hook


lxc-start 157 20211103125837.723 ERROR    conf - conf.c:run_buffer:323 - Script exited with status 1

lxc-start 157 20211103125837.731 ERROR    start - start.c:lxc_init:797 - Failed to run lxc.hook.pre-start for container "157"

lxc-start 157 20211103125837.739 ERROR    start - start.c:__lxc_start:1896 - Failed to initialize container "157"

lxc-start 157 20211103125837.747 INFO     conf - conf.c:run_script_argv:340 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "157", config section "lxc"

lxc-start 157 20211103125838.259 INFO     conf - conf.c:run_script_argv:340 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "157", config section "lxc"

lxc-start 157 20211103125839.156 ERROR    lxc_start - tools/lxc_start.c:main:308 - The container failed to start

lxc-start 157 20211103125839.163 ERROR    lxc_start - tools/lxc_start.c:main:314 - Additional information can

Sunday, February 14, 2021

Basic network commands in Proxmox PVE

If you use Proxmox PVE, you may have noticed that popular ethernet / IP / networking tools commands are not working in proxmox.  I am talking about basic commands like ifconfig.

So I just want to create this simple post as reminder for myself and for others using proxmox to use the following commands instead:


ip a

The ip command kind of replaces the ifconfig command.  The above command 'ip a' will list all network adapter detected and show their states (up/down).


ifup

The ifup command will bring up (activate) (make online) a particular network adapter.


ifdown

The ifup command will bring down (de-activate) (make offline) a particular network adapter.


The commands above works on virtual / bonded adapters too such as vmbr0.


LOCATION OF LIST OF INTERFACES CONFIGURATION

If you want to edit the list of interfaces, you can find them here:

/etc/network/interfaces

to edit simply use nano command like this:

nano /etc/network/interfaces


INSTALL NET-TOOLS

If you don't want to deal with new commands that you have to remember like 'ip a' etc... maybe you just want to install the same networking commands you are used to like ifconfig, you can just install them by typing the commands below:

apt-get update

apt-get install net-tools

Thursday, August 27, 2020

How to replace a bad hard drive in ZFS Raid

How to replace a bad hard drive in ZFS Raid and start the regeneration (resilvering process).


STEP 1 - INFORMATION OF THE FAILED DRIVE

Get GUID of the failed drive:

root@localhost# zdb
raid1:
    version: 5000
    name: 'raid1'
    state: 0
    txg: 1178836
    pool_guid: 8019483820723122312
    errata: 0
    hostid: 3155752912
    hostname: 'a6'
    com.delphix:has_per_vdev_zaps
    vdev_children: 1
    vdev_tree:
        type: 'root'
        id: 0
        guid: 8019483820723122312
        create_txg: 4
        children[0]:
            type: 'mirror'
            id: 0
            guid: 11864727355575360377
            metaslab_array: 256
            metaslab_shift: 34
            ashift: 12
            asize: 2000384688128
            is_log: 0
            create_txg: 4
            com.delphix:vdev_zap_top: 129
            children[0]:
                type: 'disk'
                id: 0
                guid: 15304200656844780564
                path: '/dev/sdb1'
                devid: 'ata-HITACHI_HUA723020ALA640_YGKU6BBG-part1'
                phys_path: 'pci-0000:00:1f.2-ata-2'
                whole_disk: 1
                create_txg: 4
                com.delphix:vdev_zap_leaf: 130
            children[1]:
                type: 'disk'
                id: 1
                guid: 980353070042574228
                path: '/dev/sdc1'
                devid: 'ata-HITACHI_HUA723020ALA640_YGKT7U6G-part1'
                phys_path: 'pci-0000:00:1f.2-ata-3'
                whole_disk: 1
                DTL: 384
                create_txg: 4
                com.delphix:vdev_zap_leaf: 131
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data

In this example we will pretend /dev/sdc is the bad drive. We will find the GUID for /dev/sdc which is 980353070042574228

Get serial #:

root@localhost# smartctl -a /dev/sdc | grep Serial Serial Number: YGKT7U6G


STEP 2 - REMOVE THE FAILED DRIVE

zpool offline raid1 980353070042574228


STEP 3 - REPLACE THE HARD DRIVE PHYSICALLY

Please replace the broken hard drive with a new hard drive.


STEP 4 - COPY PARTITION TABLE

Please note the first device in the command below is the TARGET and the second device is the SOURCE.

sgdisk --replicate=[TARGET] [SOURCE]
sgdisk --replicate=/dev/sdc /dev/sdb


STEP 5 - GENERATE RANDOM GUID

sgdisk --randomize-guids /dev/sdc


STEP 6 - ADD NEW HARD DRIVE TO ZFS POOL

zpool replace raid1 /dev/sdc


FINAL - CHECK AND MONITOR SILVERING PROCESS

watch zpool status raid1 -v




Saturday, June 6, 2020

Proxmox PVE APT UPDATE "not authorized" Error Message - How to configure No-Subscription APT Repository

Proxmox PVE is a great software and mature Virtualization Environment.  I am very thankful for the team and grateful for the software. I wish I can afford to pay for subscription, but so far I am not that much profitable yet. Because of that I am still not a subscriber :-(    However I plan to be and recommend anyone who can to subscribe.

Having said the above, I would like to show you what steps are necessary to disable Enterprise Repository and Enable the No-Subscription Repository.  This is for the New Proxmox PVE 6.x using Debian Buster.

STEP 1 - ADD NO-SUBSCRIPTION REPOSITORY

nano /etc/apt/sources.list

Add the following lines at the end of your /etc/apt/sources.list

# PVE pve-no-subscription repository provided by proxmox.com,
# NOT recommended for production use
deb http://download.proxmox.com/debian/pve buster pve-no-subscription

Your new sources.list file should now look like this:

deb http://ftp.us.debian.org/debian buster main contrib

deb http://ftp.us.debian.org/debian buster-updates main contrib

# security updates
deb http://security.debian.org buster/updates main contrib

# PVE pve-no-subscription repository provided by proxmox.com,
# NOT recommended for production use
deb http://download.proxmox.com/debian/pve buster pve-no-subscription


STEP 2 - COMMENT OUT THE ENTERPRISE REPOSITORY FILE


nano /etc/apt/sources.list.d/pve-enterprise.list

Put a pound sign in front of this line:

# deb https://enterprise.proxmox.com/debian/pve buster pve-enterprise

Your pve-enterprise.list file should now look like this:

# deb https://enterprise.proxmox.com/debian/pve buster pve-enterprise



STEP 3 - TEST BY EXECUTING APT UPDATE

You should now be able to execute 'apt update' without errors.



# deb https://enterprise.proxmox.com/debian/pve buster pve-enterprise



STEP 3 - CONGRATULATIONS! YOU CAN NOW UPDATE FROM PROXMOX

Remember to Subscribe to Proxmox when you can!

Monday, May 18, 2020

Proxmox PVE version 6.2 has been released! (Proxmox Virtualization Environment)

Proxmox, one of the best open source project for virtualization.  Proxmox PVE has released its 6th edition with version 6.2.

Here are the list of features and changes for Proxmox 6.2 (thanks to magic of copy-and-paste):

Advanced options for the web-based management interface:
  • Proxmox VE implements built-in validation of domains for Let's Encrypt TLS certificates via the DNS-based challenge mechanism, in addition to the already existing HTTP-based validation mode.
  • Full support for up to eight corosync network links is available. The more links are used, the higher the cluster availability.
  • In the storage content view, administrators can now filter the stored data with the new column ‘Creation Date’ which, for example, simplifies to search for a backup from a certain date.
  • The language of the web interface can be seamlessly changed without the need to restart the session. An Arabic translation has been added and thus Proxmox VE supports 20 languages in total.
Linux Container:
  • The integrated container technology has been updated to LXC 4.0.2 and lxcfs 4.0.3. Proxmox VE 6.2 now allows to create templates for containers on directory-based storage.
  • New LXC templates for Ubuntu 20.04, Fedora 32, CentOS 8.1, Alpine Linux and Arch Linux.
Zstandard for Backup/Restore:
  • The integrated backup manager supports the fast and highly efficient lossless data compression algorithm Zstandard (zstd).
User and permission management:
  • Proxmox VE uses a role-based user and permission management for all objects such as VMs, storage, nodes, etc. The new LDAP sync enables synchronization of LDAP users and groups into the Proxmox user and group permission framework.
  • Full support and the integration for API tokens has been added allowing stateless access to most parts of the REST API by another system, software or API client. API Tokens can be generated for individual users and can optionally be configured with separate permissions and expiration dates to limit the scope and duration of the access. Should the API token get compromised it can be revoked without having to disable the user itself.

Further notable enhancements

  • QEMU/KVM: Support for Live Migration with replicated disks (storage replication with zfs) is enabled.
  • Testing the Ceph storage has become easier as the uninstall process has been simplified.

Wednesday, February 26, 2020

Converting LVM to LVM Thin so Proxmox PVE will support Snapshot Backup and Thin Provisioning

I have been using Proxmox PVE since version 2.x.  Back in Proxmox version 2 and 3, a regular LVM partition with enough free space in its LVM Volume Group would be enough for Proxmox Backup Snapshot function to work.  However starting Proxmox version 5 you have to use LVM Thin (thin provisioning) in order to make backup with snapshot to work and support over provisioning.

Backup with snapshot is important because it allows the VM or CT to continue working during the backup process (uninterrupted).  This is crucial and makes backup seamless and effortless without downtime.

Over provisioning allow you to create VM and CT without actually taking up the storage size specified in the VM or CT.  For example, lets say you create a CT with 100GB storage, however the CT actually only use 20GB, with thin provisioning the LVM Thin will only take up 20GB instead of 100GB therefore allowing you to assign / provision more storage and you actually physically have capacity of.  Of course if all your CT / VM actually takes up 100% of their capacity and you over provision all your CT / VM you will run out of space!

Here are the steps I have done to convert my existing partition (/dev/md0) from typical ext4 mounted as a directory to my Proxmox (which currently does not work for backup with snapshot) and converting it to LVM Thin:


STEP 1 - Move all your VM and CT

First you need to empty and remove all your VM and CT if the partition you are working on is being used by some VM / CT.  I simply just STOP them and back them up one at a time and restore at a different proxmox server.   If you backup and restore, you will then still have to remove all the VM and CT in the hardware node you are working on.  The point here is, you have to empty the partition we will be converting to LVM Thin (because the data will be erased during the process).


STEP 2 - UNMOUNT THE PARTITION

Unmount the partition using the command:

umount /mnt/md0

in case the above fails you can try using 'force':

umount -f /mnt/md0


STEP 3 - CHANGE PARTITION TYPE TO LVM

See the list of devices recognize by fdisk and find the device ID you are working on:

fdisk -l

My device ID is /dev/md127

Next, I will use fdisk to create a new single partition (100% size) and change its type to LVM

fdisk /dev/md127

Type command 'n' to create new partition, select default options (press enter a few times to select default) until the partition has been created.

Type command 't' to change its type, type '8e' to select Linux LVM

Type command 'w' to commit changes.


STEP 4 - CREATE PHYSICAL VOLUME AND VOLUME GROUP

Use fdisk again to find the newly created LVM partition:

fdisk -l

My new device ID for the partition is /dev/md127p1

Create physical volume first:

pvcreate /dev/md127p1

Check to make sure physical volume has been created:

pvdisplay



Then create volume group, I would like to call my volume group local_md0

vgcreate local_md0 /dev/md127p1

Check to make sure volume group has been created:



STEP 5 - CREATE LOGICAL VOLUME WITH THIN OPTION

The lvcreate command below will create a THIN logical volume because of the -T option and will leave 5% of the volume group free.  The 5% is enough in my case because this is a 2TB partition.  You only need to reserve enough space for snapshot changes during backup operation.  Meaning lets say your backup takes 30 min, during that 30 min time, is your 5% free space enough to record all the changes that may happen within that 30 min window?

lvcreate -T -n local_md0 -l 95%FREE local_md0


STEP 6 - ADD STORAGE IN PROMOX PVE AS LVM THIN


Click on Datacenter > Storage > Add




STEP 7 - CONFIRM IT HAS BEEN ADDED AS STORAGE






Congratulations!  You know have LVM Thin which means you can do Backup with Snapshot and your VM and CT can be over provisioned!


Monday, December 2, 2019

Clean up logs, cache, docs, man files from Proxmox pve-root to free up disk space

My main pve-root partition was getting full (I only reserved 4GB, I should have reserved more space here, at least 8GB). I searched and found many areas where I can remove files to gain back a few megabytes.

DELETING UN-NEEDED FILES IN /usr/share/doc

cd /usr/share/doc
rm -rf *


DELETING UN-NEEDED FILES IN /usr/share/man

cd /usr/share/man
rm -rf *


CLEAN UP JOURNAL LOGS FROM /run/log/journal

journalctl --disk-usage
nano /etc/systemd/journald.conf
     add or edit this line:  SystemMaxUse=50M
Signal or restart systemd-journald service:
     systemctl kill --kill-who=main --signal=SIGUSR2 systemd-journald.service
       -or-
     systemctl restart systemd-journald.service


CLEAN UP /var/cache/apt

apt clean
apt purge
apt autoclean

SUMMARY

The commands above freed up about 650MB of space in my Proxmox PVE 5.1
I hope this article has been useful for you.

Tuesday, September 10, 2019

[EASY 5 MIN] Install Ubiquiti Unifi Controller on Proxmox VE LXC Container





This is a detailed video to show you how to setup/install Ubiquiti Unifi Controller on Proxmox VE LXC Container. I am using Ubuntu 16.04 / 18.04 in this video. This method is very fast, only takes about 5 min to do.

Help support my channel, if you are looking to buy ubiquiti products please use my Amazon link below:

https://amzn.to/2N8Ihro

#unifi #ubiquiti #unificontroller


Monday, September 9, 2019

Installing Ubiquiti UNIFI controller as a Proxmox VE LXC Container

This guide will help you Install a Ubiquiti UNIFI controller as a Proxmox LXC Container.

There are many benefits of having an always ONLINE Unifi controller, one of them that I am interested in is the Hot Spot / Captive Portal landing page customization.

Since I already have a Proxmox VE server running, I thought this will be the best method for me to have a Ubiquiti Unifi controller 24/7.


STEP 1 - Download Template for Ubuntu 18.04

Go to your Proxmox VE, go to your storage, and make sure you have downloaded a template for Ubuntu 18.04 Standard

STEP 2 - Create the LXC Container

Create an LXC container using the following settings:
Hostname: unifi
Disk size: 32GB
Memory: 2GB (but 1GB is also okay)
Network:  192.168.8.4/24

STEP 3 - Login to your container and Update APT

sudo apt update

STEP 4 - Installing CA Certificates

sudo apt-get install ca-certificates wget -y

STEP 5 - Download installation script


wget https://get.glennr.nl/unifi/install/unifi-5.11.39.sh

STEP 6 - Make the script executable (change filename to the script version you downloaded)

chmod +x unifi-5.11.39.sh

STEP 6 - Run the Install Script

./unifi-5.11.39.sh

STEP 7 - When finish installing go to your browser and type in

https://ip.of.your.server:8443

----------------------------------------

Thank you for using this guide.

Please help me support this blog by purchasing your Ubiquiti products from Amazon using my link below:

https://amzn.to/2A4WdtW

Much appreciated! :-)




Wednesday, July 10, 2019

[SOLVED] Mounting external USB drive to Proxmox LXC Container

I needed to mount an external USB drive directly to an LXC container.  On initial research, this process seems simple enough.

I am using Ubuntu / Debian for this guide.

Basically I do the following to get it mounted on the Proxmox Hardware Node first:

mkdir /mnt/mountpoint
chmod -R 777 /mnt/mountpoint (this is optional)
mount /dev/bus/usb/001/004 /mnt/mountpoint

BUT! when I tried to use it in the LXC container I got 'READ ONLY' permission issues. Some error message like this:

Read only filesystem!

So I did a lot more research and found many people have similar issues. I wasted 30 minutes trying different suggestions until I found one that worked.  The solution is the fact that linux was mounting the partition as fat32 / exfat. Therefore it could not perform Read and Write.  Here is the solution that worked for me...

STEP 1 - INSTALL ntfs-3g

sudo apt-get update
sudo apt-get install ntfs-3g

STEP 2 - MOUNT using NTFS file system type (my usb drive is at USB 001:004)

mkdir /mnt/mountpoint
chmod -R 777 /mnt/mountpoint (this is optional)
mount -t ntfs-3 /dev/bus/usb/001/004 /mnt/mountpoint

STEP 3 - CHECK IF YOU CAN READ & WRITE

cd /mnt/mountpoint
touch test

If step 3 did not produce any error and you can clearly see file 'test' being created, then you are good to go.

Next you just need to add one line to your LXC container configuration to set the mountpoint.

nano /etc/pve/lxc/123.conf   (assuming 123 is your CT ID)

Tuesday, May 21, 2019

How to mount NFS share to Proxmox LXC container

This is a quick how to guide to mount NFS share to your Proxmox VE LXC container.

If you are experiencing permission issue, you may want to read this article:

for example: May 21 12:42:31 e7 systemd[1]: Failed to start PVE LXC Container: 164.

Enabling NFS sharing mount bind in Proxmox

STEP 1 - EDIT YOUR LXC CONTAINER CONFIGURATION

nano /etc/pve/lxc/[CTID].conf


STEP 2 - ADD THE FOLLOWING LINE TO END OF CONFIGURATION FILE


mp0: [/host/nfs/shared_path],mp=[/lxc/container/mount_point_path]

for example:

mp0: /mnt/pve/nfs_share/,mp=/data_shared


(note: please make sure you pre-created the /data_shared (mount point) directory in your LXC container first)

STEP 3 - RESTART YOUR LXC


pct shutdown [CTID]
pct start [CTID]


Unlike openvz you do not have to edit / add anything to your fstab file. The next time you enter your container, you will have the NFS share automatically mounted.

Enabling NFS sharing (mount bind) for mounting NFS shares into LXC container

If you are running Proxmox 5.x and you are trying to mount an NFS share to your LXC container you may encounter permission denied issues.

such as:
May 21 12:42:31 e7 systemd[1]: Failed to start PVE LXC Container: 164.


This issue is caused by security permission in apparmor. We just need to reconfigure it so that it will support NFS shares.

This article will describe how to overcome this issue permission issue.

STEP 1 - COPY APPARMOR CONFIGURATION (on Proxmox Host)

cd /etc/apparmor.d/lxc
cp lxc-default-cgns lxc-container-default-with-nfs
nano lxc-container-default-with-nfs

Change lxc-container-default-cgns with lxc-container-default-with-nfs

Add the following lines right before the } (closing curly brace)

  mount fstype=nfs*,
  mount fstype=rpc_pipefs,


STEP 2 - RELOAD APPARMOR

systemctl reload apparmor



The above 2 steps will fix the AppArmor permission issues.

Next, you may want to know how to Bind Mount your NFS share to your LXC container, please click on the next article How to mount NFS share to Proxmox LXC container.



SAMPLE OF MY LXC CONFIGURATION FILE:


arch: amd64
cores: 4
hostname: blahblah
memory: 8192
nameserver: 10.0.0.1 8.8.8.8 4.4.4.4
net0: name=eth0,bridge=vmbr0,gw=10.0.0.1,hwaddr=_______________,ip=10.0.110.42/16,type=veth
ostype: ubuntu
rootfs: local_md0:164/vm-164-disk-1.raw,size=64G
searchdomain: localhost
swap: 8192
lxc.apparmor.profile: lxc-container-default-with-nfs
mp0: /mnt/pve/vepublicb1/tf/data_shared,mp=/data_shared

Debugging a proxmox LXC container that will not start

Sometime after making changes in your LXC configuration file in /etc/pve/lxc your LXC container may have problem starting. You will get a message like this:

Job for pve-container@165.service failed because the control process exited with error code.
See "systemctl status pve-container@165.service" and "journalctl -xe" for details.
command 'systemctl start pve-container@165' failed: exit code 1

Then it recommends to get status of the start by typing...

systemctl status pve-container@165.service

And its output is...

● pve-container@165.service - PVE LXC Container: 165
   Loaded: loaded (/lib/systemd/system/pve-container@.service; static; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2019-05-21 08:13:01 CDT; 10s ago
     Docs: man:lxc-start
           man:lxc
           man:pct
  Process: 3560220 ExecStart=/usr/bin/lxc-start -n 165 (code=exited, status=1/FAILURE)

May 21 08:12:59 e2 systemd[1]: Starting PVE LXC Container: 165...
May 21 08:13:01 e2 lxc-start[3560220]: lxc-start: 165: lxccontainer.c: wait_on_daemonized_start: 865 Received container state "ABORTING" instead of "RUNNING"
May 21 08:13:01 e2 lxc-start[3560220]: lxc-start: 165: tools/lxc_start.c: main: 330 The container failed to start
May 21 08:13:01 e2 lxc-start[3560220]: lxc-start: 165: tools/lxc_start.c: main: 333 To get more details, run the container in foreground mode
May 21 08:13:01 e2 lxc-start[3560220]: lxc-start: 165: tools/lxc_start.c: main: 336 Additional information can be obtained by setting the --logfile and --logpriority options
May 21 08:13:01 e2 systemd[1]: pve-container@165.service: Control process exited, code=exited status=1
May 21 08:13:01 e2 systemd[1]: pve-container@165.service: Killing process 3560226 (3) with signal SIGKILL.
May 21 08:13:01 e2 systemd[1]: Failed to start PVE LXC Container: 165.
May 21 08:13:01 e2 systemd[1]: pve-container@165.service: Unit entered failed state.

May 21 08:13:01 e2 systemd[1]: pve-container@165.service: Failed with result 'exit-code'.


As you can see the output still does not tell me enough information.

So I recommend using the following command to start LXC with debug and output log to temporary file:


lxc-start --logfile /tmp/lxc-start.log -n [CTID]



if the debug or log above is still not enough, the next command below will provide an EXTREMELY detailed log:

strace -f lxc-start -l trace -o /tmp/trace.log -n [CTID]

Wednesday, May 15, 2019

Debugging (verbose logging) LXC startup process in Proxmox VE

Sometime you run into problem and your LXC container do not start.  Proxmox VE graphical UI is very good, but it does not provide enough information about what is wrong.

To get the most detailed information, you should try to start the LXC container from command using verbose logging. Here is the command you should use:


(replace [ID] with your actual LXC container ID)

lxc-start -n [ID] -F -l DEBUG -o /tmp/lxc-[ID].log


Then, to view the log use this command:

cat /tmp/lxc-[ID].log

Upgrading Proxmox VE

To upgrade your Proxmox virtual environment to the next version do the following steps.

Step 1 - Get latest APT updates

apt-get update


Step 2 - Perform distribution upgrade

apt-get dist-upgrade


You may be asked if you would like to use existing or newer configuration file. I usually answer N (no) to keep using the current version which is usually safer.

Can not start LXC container with error: unsupported Ubuntu version '18.04'

I restored an LXC container I have backed up from a newer Proxmox VE to an older node. To my surprise... it would not start. This is the error message I got when I debugged the LXC start-up process.


lxc pre-start with output: unsupported Ubuntu version '18.04'


When I press start button, it quickly fails and displays:




I thought it may be caused by file system (the LXC image) corruption. So I tried to mount and unmount it first using

pct mount 165
pct unmount 165

both worked perfectly so that was not the issue.

Then I made sure the file system is clean by running fsck against it, using this fsck command

pct fsck 165

That also ran perfectly clean!

After being puzzled, I researched and found people with similar issue, it turned out that Proxmox LXC start up script checks for version of Ubuntu and did not have the latest version 18.04 LTS Bionic Beaver.  So I updated this file:

nano /usr/share/perl5/PVE/LXC/Setup/Ubuntu.pm

look for this section:
my $known_versions = {
  
    '17.10' => 1, # artful
    '17.04' => 1, # zesty
    '16.10' => 1, # yakkety
    '16.04' => 1, # xenial
    '15.10' => 1, # wily
    '15.04' => 1, # vivid
    '14.04' => 1, # trusty LTS
    '12.04' => 1, # precise LTS
};

and replace it with this:
my $known_versions = {
    '22.04' => 1,
    '20.04' => 1,
    '18.04' => 1, # Bionic Beaver LTS
    '17.10' => 1, # artful
    '17.04' => 1, # zesty
    '16.10' => 1, # yakkety
    '16.04' => 1, # xenial
    '15.10' => 1, # wily
    '15.04' => 1, # vivid
    '14.04' => 1, # trusty LTS
    '12.04' => 1, # precise LTS
};


This SOLVED my issue and I was able to start the LXC container.

If you don't feel comfortable editing /usr/share/perl5/PVE/LXC/Setup/ 
You can also simply upgrade your Proxmox VE node to get the latest update from Proxmox.

Proxmox LXC Backup issue: INFO: mode failure - some volumes do not support snapshots

I am getting this notice when performing LXC container backup on Proxmox 5.x

INFO: mode failure - some volumes do not support snapshots




At first I thought there was something wrong with the container, I checked for strange things that I may have done such as mounting external NFS share, etc... but I did not find anything strange.

Then I realize this disk partition or space that the container is on is not an LVM type of partition.

Because of this I can NOT do a 'SNAPSHOT' backup.  Suspend somehow does not work either.

But backup did work after full shutdown. Or you can just start the backup right away with SHUTDOWN mode.

Hope this help someone with same issue.