A quick one as no-one seems to have written it down properly.
Mailcow (https://mailcow.email) is a dockerized mail system. By default it will run on port 80 and port 443 of an IP address, so it occupies the web addresses of, say, a cheap VM.
Nginx Proxy Manager (https://www.nginxproxymanager.com) is a dockerized way of managing an nginx proxy.
The theoretical right way of connecting the two is by bridging or sharing the docker networks, but a more straightforward way is connecitng the exposed ports on the VM IP address. Not localhost or 0.0.0.0 because they're in the docker network, the IP address of the VM. Annoying simple but not necessarilly obvious.
Category: Technical
Fixing a frozen screen caused by systemd 256
Systemd version 256 has been rolled out in several distributions recently, including Arch and Debian and variants. It adds a feature where user sessions are frozen when the system enters sleep, but in my case, running Manjaro Linux with KDE on a Gen 1 Lenovo T14, this meant that when the system woke, the screen remained frozen and required a reboot by switching to the command line with Ctrl-Alt-F3.
At first I perhaps unfairly assumed that this was KDE as it affected the desktop but left the mouse working and indeed the system accessible with through the console, but after a bit of digging around in the system journal with journalctl -b-1 -p4 --no-pager (a really useful command that I didn't know before), I found the error Cannot start frozen unit Session 10 of User xxxxx from systemd.
Searching on that brought up the problem and conclusively pointed at the 256 update, and a solution, or at least a fix until the issue is fixed.
The fix is to add an override to the systemd-suspend service. This doesn't actually have a service file in /etc/systemd/system in Manjaro at least, so you need to create the directory /etc/systemd/system/systemd-suspend.service.d and create an override file called disable_freeze_user_session.conf with these contents:
[Service]
Environment="SYSTEMD_SLEEP_FREEZE_USER_SESSIONS=false"Save this, and restart systemd with systemctl daemon-reload.
This disables user freeze on sleep and for me at least, reverted to the previous behaviour. I found that a lot of the examples mentioned Nvidia video hardware, which I don't have, so it's probably a wider issue than I've found.
I've added a comment to the Github issue.
A fix for ‘cert is required but missing for this certificate’ in certbot
Yes, this is probably a bit search spammy (like search actually works any more), but it\'s an interesting little fix and a view into how certbot works. Apart from when new posts are published, a server I manage is served wholly from Cloudflare. We have recently migrated it and apart from the initial caching phase it runs with no load. As the key components were synced over by rsync, this copied over the LetsEncrypt certificates, which, because of the Cloudflare caching, failed to renew at some point, silently. Certbot\'s version control is in the /etc/letsencrypt/archive folder. Certificate files are written here with a number appended to them and symlinked to /etc/letsencrypt/live, so while certbot works correctly, there will be an archive file with a number, so cert1.pem, cert2.pem etc. In this case however, the certificate files in /archive weren\'t numbered, so when attempting to renew the certificate manually, certbot fails with the error cert is required but missing for this certificate. The fix is, fortunately, simple: rename the cert files in /archive with a number and relink them to the files in /live. Run the renew command again and the certificates will be reissued correctly.
Adventures in BTRFS: Replacing an array
Preamble ramble
There aren't a lot of real world examples of using BTRFS around online. Some of the most useful advice I found was in r/BTRFS on Reddit but it's hard to actually find user experiences beyond the theroretical.
This is sort of one, but the tl;dr is have good backups and learn from your experiences. The practical stuff starts at 'The Practical Stuff'.
I've had a home disk array for a long time for various reasons. As much as anything I have a home music streaming system as I prefer to buy and own music, even if it is as a download, and over the years I have digitised CDs, originally into iTunes but currently using Funkwhale.
The current disk array is a Terramaster D5-300, a five disk JBOD array that has Windows based software RAID support, but, being Just A Bunch of Disks, can be set up with other on-disk RAID architectures, so, as a Linux user, there were a couple of immediate choices: ZFS, or BTRFS.
ZFS is an enterprise level filesystem that was originally developed by Sun (RIP) but was ported to Linux when Sun's BSD based OS Solaris was open sourced before the Oracle takeover and was forked as OpenZFS. ZFS itself is a great filesystem, but at the time (2016) OpenZFS development had slowed and it's not really a good fit for the kind of low powered machine that goes with a home setup (I think I set it up on an Acer Revo at first).
BTRFS, on the other hand, is in the Linux kernel, so if something runs the kernel, it should work, and just needs the btrfs-tools package to manage it, so it can be configured and moved around if necessary.
At the beginning of this month, owing to a full fibre upgrade and a declining powerline network that meant my home office wasn't enjoying the benefits of the kind of connectivity that was unimaginable when I first plugged in a modem, I had ethernet run around the house by Tom at WestworldIT, which required the study to look less of a tip, facilitiating a much needed clean but also the shutdown of the server for a while, and as is often the case in these situations, checking the disks with smartctl showed that a couple of the aging second hand 3TB disks in it failed their tests. I had a pair of 4TB disks so went looking for some more slightly less aging second hand ones and found some via eBay at the Edinburgh Remakery.
As the array disks were holding up, I backed them up to a spare 4TB and started the replacement.
The Practical Stuff
Note: there are mistakes here. If you know anything about BTRFS you will probably be shouting at me.
The array was originally set up as RAID10 with metadata striped 1c3, so on three disks.
I replaced one of the failing disks with the spare and ran btrfs balance /mount. I should have possibly done btrfs replace here but both seem to be valid at this point. However, balance is a rich command and works best with filters, which are well described in the manual page for btrfs-balance, so running it as is is going to take some time.
When the new (second-hand) 4TB disks arrived the next day, I left them to warm up and let the balance complete.
The disks were a variety of 'desktop' (does anyone have a 4TB drive in a desktop these days - I suppose they do) and NAS disks, all around six years old. They all checked out with smartctl, so I started to replace the older 3TB disks using btrfs replace start <id> /dev/sdx /mount.
Each disk took about 24 hours, for around 2.5TB of data. The first two worked perfectly. Progress can be monitored with btrfs replace status /mount.
I did two disks and resized them, but the third appeared to be showing problems when trying to mount the volume. It appeared it might be the disk itself but smartctl didn't show anything. I swapped it out for the other spare 4TB that I used to back up (copying the data onto one of the 3TB disks and a bit of juggling).
However, here was my mistake: I picked out the last disk I replaced for the backup, and that must have been the one that had metadata on it, so I was now down to two metadata disks and the replace had got stuck and the volume would only mount read only, indicating that it was time to get data off and start again. You are welcome to shout now, as I did.
As I have backups I decided not to fight and just recreated the volume and started restoring and it looks like it's going to be a day or so of rsyncs.
Lessons learned
- Maintain backups
- Watch what state your metadata is in between swapping disks - I'm not 100% sure that this was the issue but it's what I suspect, and balancing metadata would probably have avoided it.
- Have patience - it's a slow process and rightly so. Cloning disks appears to be an alternative but I'm not sure how that would work with a RAID scheme like 1c3.
- Maintain backups.
As a note the current storage host is a 8GB Raspberry Pi 4, which is excellent as a server but less so in doing maintenance work, so that gets handed off to a laptop.
Closing down an instance
I set up offworld.fedisonic.cloud when it was apparent that Twitter was going to start circling the drain pretty quickly. I had an idea of running a Mastodon instance for anyone in my circle who wanted it, but with my usual overambition I started to think that it could be a business of some kind.
I first looked at Mastodon when it was announced back in 2016 or so, when it was becoming apparent when Twitter was getting gamed, and put up an instance on a VM, but the discovery process wasn't there and it was hard to find people and I took it down again. I made a mental note to watch ActivityPub and developments around it. I joined mastodon.social (and chaos.social, which is the opposite) when the toxicity in Twitter was starting to become the norm but didn't find it compelling, but still kept an eye on what was going on.
When That Guy squandered enough money to end world hunger on being That Guy on Twitter every day, enough was enough. I had a couple of underused servers so threw up a instance, invited people to see what the fuss was about and imported my followers as everyone was doing the same.
Now it's just short of a year later and I've closed that server. I'm running one for myself. set up in a better way, and of the people I invited, one has stayed on Mastodon.
Over the last year I've realised a few things:
- Mastodon is not the Fediverse. It's shorthand for the network because it is the most popular platform, and we like to have an easy handle for these platforms.
- Mastodon the application is technically simple to set up. I am a sysadmin of several million lifetimes' experience to be sure but it is made of common components and the documentation is well written, and if you can set up WordPress or Drupal it's not too much of a challenge.
- Mastodon the service is challenging to maintain. The abillty to block instances and indeed users at the server level is powerful, but a year ago it relied on a hashtag which of which it quickly became apparent was open to abuse and indeed straightforward politics.
Now, common blocklists have emerged and operators can choose what to use, but with any kind of success comes abuse, and it's obvious that public instances have bigger operating overheads than scaling their hardware. - Mastodon the software is resource hungry. At scale it is probably very efficient. With two (two!) active users it's overkill and things like scheduled deletes of busy follows could grind my server to a halt if that server was in the same timezone (if that makes sense). As a system that's designed with privacy in mind it's also difficult to troubleshoot as there are no logs by default (they can be switched on of course) so the main source of information is the job queues.
- Technically, use cloud storage rather than on server storage, or at least don't use the same disk device. Cache as well, also not on the same disk device. It creates a lot of I/O.
- Legal implications are getting more onerous across the world, and while they're a grey area for small instances, in the UK at least, we're not far from operators having to take a lot more responsibility for their users.
Running a small server with and for other people is a committment. Doing it for money would be something else again. I'm glad that there are businesses setting that up
On the actual social side of social media (remember that?), my experience of the Fediverse has been generally good. My feed seems to be largely cat pictures (especially on #caturday, but finding good people has been relatively easy. I'm not a big socialiser at all but I've long liked how the Internet works in that respect, and as ActivityPub is basically marshalling RSS feeds, we're back where were before we were rudely interrupted, just doing it in a way that makes more sense now.
I am going to try a couple of other platforms as I don't think Mastodon is the best option for a single user fediverse instance, or even two or three users. and work out how to aggregate different applications, like WordPress for example, so that's the next year sorted.
Adding S3-compatible cloud storage to Pixelfed
Pixelfed is a federated photo sharing service that is an alternative to Instagram and uses ActivityPub to share posts across other services such as Mastodon, Pleroma and indeed WordPress. It is one of the more mature Fediverse apps and has been in continuous development for several years.
As with Mastodon and other apps that can share images and other larger media files, it supports saving attachments to S3-compatible cloud storage. However, this isn't well documented so far, so here's a few notes on getting it to work, specifically with iDrive E2, a low cost E2 storage provider.
Pixelfed is a Laravel application and its user config is stored in a .env file in the application root. The S3 section looks like this:
## S3 Configuration (Post-Installer)
PF_ENABLE_CLOUD=true
FILESYSTEM_CLOUD=s3
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_DEFAULT_REGION=
AWS_BUCKET=
AWS_URL=
AWS_ENDPOINT=
AWS_USE_PATH_STYLE_ENDPOINT=falseIDrive E2 appears to be based on Minio but the priniciple is broadly similar for all S3 compatible services:
- Create a bucket
- Create credentials for that bucket
- Configure the application with those credentials
I think they're fairly self-explanatory in that the service will usually present them in the same way. The default region can be a bit of odd one but the naming seems to be arbitrary - I'm using iDrive's London region, which is labelled LDN, so go with what the service gives you.
AWS_URL is as it says, a standard URL, so it's https:// AWS_ENDPOINT / AWS_BUCKET. AWS_ENDPOINT is an important thing though - IDrive has the option of public and private endpoints with different hostnames. I'm not entirely clear on how a private endpoint works in this context - I would think it's a location that needs to be accessed with a shared key - but as you're publishing images online that's not going to be possible, and in testing I couldn't get Pixelfed to write to a private endpoint. In addition they're going to need to be public on an open server, so following the principle of least privilege, public and read-only works.
I had to comment AWS_USE_PATH_STYLE_ENDPOINT out as setting it to false didn't work for some reason. There's probably something in the code that assumes it's going to be an AWS URL or similar.
The Horizon dashboard was very useful for debugging. Again, this is a Laravel tool and part of the admin suite. I'm not that familiar with Laravel and but it basically appears to be the application manager. It writes to a log but that doesn't say much, and all the useful information is in the dashboard.
If you want to share images in the Fediverse, Pixelfed is the application you need. Its author, Daniel Supernault has been steadily improving it for years, and he's also currently working on a secure and federated messaging system intended as a drop-in replacement for direct messages. Cloud storage reduces your hosting costs as it's usually cheaper in bulk than VPS, and it's portable, so now you can replace Instagram with something under your control.
Adding an IPv6 default route with network-scripts in an EL8 server at OVH
This is a bit of a niche one, not least because network-scripts is supposed to be on its way out as a configuration method, despite cPanel still not supporting NetworkManager properly. No problem people, it's only been around for ten years.
This may also be specific to OVH's network, but I haven't found a resource that covers it, so here are my notes.
Dedicated servers at OVH are allocated a /64 IPV6 range. The gateway is the last address in the range, which is slightly more eye crossing than with IPV4, but ends in FF:FF:FF:FF. However, in EL8 (and in Debian-derived distros to my knowledge), ifup doesn't add a default gateway if it can't ping the address, and it can't ping the address because it needs a default gateway. This also applies to IPV4 in OVH's network, which is a pain for automatic provisioning.
The network-scripts system is remarkably flexible, and wrangling it is a dying art, but the key here is the ifup-post script. This runs after ifcfg-<device> and goes through a sequence of the other scripts in /etc/sysconfig/network-scripts- you can see which ones if you look at the code.
One of these is route6-<device> and this is where you add your default route so it's added after the initial network config. In OVH images it is disabled, and perhaps confusingly, includes a list of static routes generated by cloud-init. Rename that one, and using your editor of choice (vim, you animals), open a new file vi route6-eno1 (for example) and enter your gateway and default route like this:
2001:xxxx:xxxx:xxff:00ff:00ff:00ff:00ff dev eno1
default via 2001:xxxx:xxxx:xxff:00ff:00ff:00ff:00ff dev eno1Enable the script by making it executable chmod +x route6-eno1.
and restart the network with systemctl restart network.
and you should now be able to ping the IPV6 host of your choice (let's face it, your local google.com gateway).
On microblogging
On his blog, Terence Eden wrote about the Time I Invented Twitter and it's an interesting proposition which would have just been slightly ahead of its time in 2003. Microblogging probably needed Internet enabled phones to take off, like social networking in general.
Douglas Adams was a very intelligent man who liked his gadgets - Macintoshes, synths and left-handed Casio guitars appeared in the first Dirk Gently novel, Dirk Gently's Holistic Detective Agency, and, with The Book in The Hitchhiker's Guide to the Galaxy he, at the very least, created the idea of a portable device that knew everything.
When the commercial Internet started, Adams got involved in games, co-founding The Digital Village, developers of the game Starship Titanic, who then started h2g2, a collaborative dictionary project described as the Earth edition of the Hitchhikers' Guide to the Galaxy.
In his article My Vision for H2G2, written in 2000, he wrote this:
...when you write in something as simple as 'The coffee here is lousy!' the Guide will know exactly what to do with that information and where to put it. And if you see, a few seconds later, a note which says 'Yes, but the cheesecake is good' it might be worth looking round the other tables to see who you've just made contact with.
That resonated with me, and I filed it away. At the time I had a kind of mobile Internet setup, with a Palm V and a Nokia 8210 that communicated by infrared, that I could actually browse the web on, and with a bit of coaxing even use telnet on. Given this experience with the mobile web, I got tipped off about a trial project that involved the Handspring Visor, a PalmOS device that improved on Palm's hardware by adding an expansion slot, that among other things, had a GSM phone module and a TCP/IP stack (my memory says it was actually Trumpet Winsock ported to PalmOS, but my memory sometimes makes things up). For £99 and couple of feedback sessions, I had a connected handheld device.
At the feedback session, they walked me and other trial users through what they had in mind, and from what I can remember, it was fairly grey stuff. Online banking seemed to feature quite a lot, and it turned out that the service was rather limited by the data provider and a relatively high cost for data at the time. It might have even been limited in availability from 8am to 8pm or something like that. Given the lack of ideas coming from the company, I quoted DNA's idea above, which seemed to me to be a logical progression, and it seemed that the idea had never occurred to the company.
The project was abandoned. I heard later that the running cost was unpopular - data pricing was on top of airtime, which wasn't cheap back then, but also that the applications they were aiming at just didn't catch the imagination. I held onto the Visor for a while but it more or less died when PalmOS died, and by the first generation of genuinely smart handheld devices were starting to appear.
To embrace that in 2003 was to look forward from blogging, which, let's not forget, was huge then, to something much more personal, but also that would as Adams imagined, hopefully add to the sum total of global knowledge.
Disabling a Wacom laptop touchscreen in an Arch Linux based OS
(Without disabling everything else)
My main laptop is a Lenovo X1 Yoga 2nd Gen running Manjaro KDE Plasma. Lovely machine and does what I want it to do. As a Yoga device, it has a touchscreen, which I don't use a lot and have often thought of disabling.
The other day, the laptop had the wrong kind of drop, which has cracked the touchscreen in the corner. This hasn't affected the display at all but has messed up the touchscreen input so that it keeps getting random signals that trigger events, which was sufficiently intrusive to need to turn off the touchscreen
The first thing I found was this from the Manjaro Forum. Tl;dr, disable the module that powers the touchscreen with sudo modprobe -r usbhid and make it permanent by creating a blacklist at /etc/modprobe.d/blacklist/wacom that contains the following:
blacklist wacom
blacklist usbhidand restart.
This worked fine when just on the laptop, but I have a multi-monitor desktop setup that has a USB mouse and keyboard, and when I came to start work this morning, neither worked. Enabling usbhid in the blacklist just brought the spurious touches back.
There was going to be something that creates rules to selectively allow USB devices, and that something is USBGuard.
The Arch Wiki documents it well, but basically install with pacman -Sy usbguard (or your software manager of choice), and create your ruleset as root with usbguard generate-policy > /etc/usbguard/rules.conf.
This lists all your connected devices as allowed, including the touchscreen:
allow id 056a:50b6 serial "" name "Pen and multitouch sensor" hash "B1HYEaAtN9VpnKbIK5GQeZFfg3XN7EAAeQUvTx5zIhk=" parent-hash "jEP/6WzviqdJ5VSeTUY8PatCNBKeaREvo2OqdplND/o=" via-port "1-10" with-interface { 03:00:00 03:00:00 } with-connect-type "not used"To disable it, change allow to block to stop it being processed, or reject to stop the device being loaded at all. At the moment I have it set to block.
Start USBGuard with systemctl start usbguard and enable it on boot with systemctl enable usbguard.
This stopped the touchscreen responding but kept the USB keyboard and mouse working. I haven't tested it across a reboot yet but I can't see why it won't continue to work.
On defederation
The Apple TV series Mythic Quest had an episode in which the game found it had an extreme right wing problem. Their solution was to corral the right wingers in a server where they could shout at each other and fight as much as they wanted without bothering other players.
The Fediverse, running, as it does, largely on free software, came about in part due to Twitter and other platforms' unwillingness or inability to deal with an extreme right wing problem. However, as it's free and open source software, the bad actors were also free to create their own instances and interact. The response was filtering, blocking and defederation.
If you run a Mastodon, or other Fediverse social media instance, even if it's for yourself, it's one of the most powerful tools you have. You can filter hashtags, users and whole instances both personally and globally.
Lists have developed over the years along with the tools to apply them, but they have often been personal efforts. The hashtag #fediblock can also be used, but it's something of a blunt instrument and is too easily hijacked for personal opinions and even feuds.
The current attempt to deal with this in an effective way is the Oliphant.social blocklist files, created through a consensus of ten of the most active fediverse instances This produces a collection of blocklists that can be applied as an administrator sees fit. Follow the instructions there to download, maintain and apply them. At the moment I apply the Tier 0 blocklist.
In addition, Ro Iskinda's The Bad Space collects the most commonly reported bad actors and new ones that appear, and can be used with an API to check if you encounter a doubtful user or instance.