Recovering from email delivery failure requires exim4 exim_tidydb to clear the error

My cyrus imap server stopped working without any real reason. It is very strange. But the recovery was to purge jansson library, reinstall it, and force-reinstall cyrus.

IMAPD stopped working. I have never seen this before. My guss is file data corruption but then, this is also very unlikely as the system is on RAID1 ZFS. (I am aware ZFS has some cache/corruption issue but my system is NOT busy at all.)

sudo pkg delete -f jansson
sudo pkg install jansson
sudo pkg install -f cyrus-imapd310

After this, IMAPD came back to life.
Yet, exim refused to deliver since the LMTP endpoint was down long time. I had to clear off the error status of the LMTP endpoint.

sudo exim_tidydb -t 1m /var/spool/exim retry

My email was bouncing for multiple days. This now reminds me that I need a way to know imapd/lmtp down. I lost a few days of emails.

Framework 16 laptop

Keyboard

As in QMK, copy this to /etc/udev/rules.d/50-qmk.rules.
Then append these lines.

# Framework
SUBSYSTEMS=="usb", ATTRS{idVendor}=="32AC", ATTRS{idProduct}=="0012", TAG+="uaccess"
SUBSYSTEMS=="usb", ATTRS{idVendor}=="32AC", ATTRS{idProduct}=="0013", TAG+="uaccess"
KERNEL=="hidraw*", SUBSYSTEM=="hidraw", ATTRS{idVendor}=="32ac", ATTRS{idProduct}=="0012", MODE="0660", GROUP="users", TAG+="uaccess", TAG+="udev-acl"
KERNEL=="hidraw*", SUBSYSTEM=="hidraw", ATTRS{idVendor}=="32ac", ATTRS{idProduct}=="0013", MODE="0660", GROUP="users", TAG+="uaccess", TAG+="udev-acl"

Then, reload rules, and trigeer the new rule.

sudo udevadm control --reload-rules
sudo udevadm trigger

Now, VIA should work.

xcp-ng + xoa installation

XOA if you wish (optional)

Once xcp-ng is up and running, you’ want XOA up for VM migration. Here is the fastest way.
Log into xen host,
bash -c "$(curl -s http://xoa.io/deploy)"
Answer pretty basic Q and off you go.

If you can run Docker image (Linux, WSL, MacOS (not sure for Apple silicone)), you can run XOA locally to connect to xcp-ng server. This saves the disk/memory on the server.

The default username/password is admin@admin.net / admin.

Cheat sheet

auto start

xe vm-list
xe vm-param-set uuid=$VMUUID other-config:auto_poweron=true

get IP

xe vm-param-get param-name=networks uuid=$VMUUID

PCI passthrough

source

First, make dom-0 to not use the PCI devices.

/opt/xensource/libexec/xen-cmdline --set-dom0 "xen-pciback.hide=(02:00.0)(06:00.0)(06:00.1)"

check what’s the boot command with

cat /boot/efi/EFI/xenserver/grub.cfg

(if this is EFI boot.)

Reboot.

xl pci-assignable-list shows the devices, and use the devices for the guest.

0000:02:00.0
0000:06:00.0
0000:06:00.1

xe vm-param-set uuid=$VMUUID other-config:pci=0/0000:02:00.0,0/0000:06:00.0,0/0000:06:00.1

Reaching to client VNC console

First you need to allow sshd to port forward in /etc/ssh/sshd_config

PermitTunnel yes

systemctl restart sshd

ssh 

Gotchas

When you import a VM from ESXi, the ethernet device name changes. If you are using netplan for static IP address, adjust the device name in /etc/netplan.

Debian/Ubuntu

You may need to install xe-daemon. Without it, the VM status like network IP does not show up.

sudo apt install xe-guest-utilities
systemctl enable xe-daemon
systemctl start xe-daemon

XigmaNAS

pkg install sysutils/xe-guest-utilities xen-guest-tools 

Go over to “System > Advanced > rc.conf” and add ” xenguest_enable” -> “YES”

and service xenguest start. Once it works, you can see the IP address on XOA.

XOA in a docker

This is one of good reasons on my book to go with xcp-ng. I don’t have to run XOA on the server.
Following the instructions on https://github.com/ronivay/xen-orchestra-docker, I make a docker image and running it locally.

Create dirs for the docker:

sudo mkdir -p /var/lib/xen/xo-server /var/lib/xen/redis

then run it.

docker run -dit -p 3380:80 -v /var/lib/xen/xo-server:/var/lib/xo-server -v /var/lib/xen/redis:/var/lib/redis --name xoa --stop-timeout 30 xen-orchestra

I put this in a shell script and run it when I want to run XOA and open the console on localhost:3380.

nct6755 on xcp-ng – SuperMicro x12sae motherboard

I purchased a used SuperMicro x12sae motherboard for xcp-ng host.
Unfortunately, sensors-detect does not recognize the sensor chip.

Trying family `VIA/Winbond/Nuvoton/Fintek'...               Yes
Found unknown chip with ID 0xd423

The sensor chip on this board uses nct6755 kernel module.

So in /etc/modules-load.d/nct6775.conf,

nct6775

You can load it with modprobe nct6775, and sensors command should report the temps and everything.

“coretemp” – coretemp-module-alt is also useful. Add both.

End of VMware ESXi free version and migration to other hypervisor

Broadcom’s buyout of VMware resulted the end of free version of ESXi.
With it, I have to start moving out my VMs from ESXi to something else. There are a few, and for me
– proxmox
– xcp-ng / xen
– bhyve
Majority seem to like proxmox, and I think I understand why. Pretty nice UI and for small user like me, it should work pretty well, especially Linux only users.
To me, proxmox is “too busy”. I want something simple. ESXi (vSphere?) free was comprehesive and did what I want it to do.
As a FreeBSD usre, bhyve is a possibility. The downside is, there is no good web UI at the moment, so everything needs to be done from command line.
Xen, especially using FreeBSD as dom0 is an interesting option but it would add extra layer of learning.
xcp-ng is a ESXi wanna be but the lingos and UI is lacking. I spent a few hours going through the menus and I am somewhat lost. Feature wise, it does what I want to do.
Well, the 4th option is to go back to bare metal. It’s not like I have too many VMs. (I am running 5 VMs.) and some can be consolidated. Some VMs can be done with docker or VirtualBox as well.
This is truely a PITA situation to undo the VMs. It is likely I have to write down a few posts for the transition.
TBH, at the moment, the front runner is xcp-ng. If I can run XOA in a docker, it would solve the most everything I want to do.

[HACK] Ubuntu Server booting to not wait for 120 seconds for unconnected network interface

While setting up a new network server for my volunteer work, the server with 1+4 ethernet port machine waits for network up and running. This is very annoying.
All I want is to make this timeout shorter. After sniffing around, I spotted the thing that waits.
/lib/systemd/system/systemd-networkd-wait-online.service has

[Service]
Type=oneshot
ExecStart=/lib/systemd/systemd-networkd-wait-online -q --timeout=10
RemainAfterExit=yes

Add -q and --timeout=10. It still waits but only 10 seconds, not 120.
Do sudo systemctl daemon-reload which I think not needed, but does no harm either, and reboot.

PROFIT!

Ubuntu – enabling USB wakeup

First, you need a command to set “enabled” to the USB ports.

Quick hack of shell script. Let’s call this /usr/local/bin/usb-standby-power-on.sh.

#!/bin/bash

for usbport in /sys/bus/usb/devices/usb*/power/wakeup ; do
    echo enabled > $usbport
done
exit 0

Then, you need a systemd unit file. Name this /etc/systemd/system/usb-wakeup-enable.service.

[Unit]
Description=Enable USB's stand-by power

[Service]
Type=oneshot
ExecStart=/usr/local/bin/usb-stanby-power-on.sh
User=root
Group=root
RemainAfterExit=true  # Indicates that the service remains active after the main process exits

[Install]
WantedBy=basic.target

Then,

sudo systemctl daemon-reload 
sudo systemctl enable usb-wakeup-enable.service
sudo systemctl start usb-wakeup-enable.service

Ubuntu Discord generates a lot of logging – : apparmor=”DENIED” operation=”ptrace” class=”ptrace” profile=”snap.discord.discord”

https://forum.snapcraft.io/t/discord-ptrace-apparmor-denials/5099/13

If you see Discord generating the logging

: apparmor="DENIED" operation="ptrace" class="ptrace" profile="snap.discord.discord"

Ues following command supresses it. See the first line link.

snap connect discord:system-observe :system-observe
snap connect discord:unity7 :unity7