Preamble ramble

There aren't a lot of real world examples of using BTRFS around online. Some of the most useful advice I found was in r/BTRFS on Reddit but it's hard to actually find user experiences beyond the theroretical.
This is sort of one, but the tl;dr is have good backups and learn from your experiences. The practical stuff starts at 'The Practical Stuff'.

I've had a home disk array for a long time for various reasons. As much as anything I have a home music streaming system as I prefer to buy and own music, even if it is as a download, and over the years I have digitised CDs, originally into iTunes but currently using Funkwhale.
The current disk array is a Terramaster D5-300, a five disk JBOD array that has Windows based software RAID support, but, being Just A Bunch of Disks, can be set up with other on-disk RAID architectures, so, as a Linux user, there were a couple of immediate choices: ZFS, or BTRFS.
ZFS is an enterprise level filesystem that was originally developed by Sun (RIP) but was ported to Linux when Sun's BSD based OS Solaris was open sourced before the Oracle takeover and was forked as OpenZFS. ZFS itself is a great filesystem, but at the time (2016) OpenZFS development had slowed and it's not really a good fit for the kind of low powered machine that goes with a home setup (I think I set it up on an Acer Revo at first).
BTRFS, on the other hand, is in the Linux kernel, so if something runs the kernel, it should work, and just needs the btrfs-tools package to manage it, so it can be configured and moved around if necessary.

At the beginning of this month, owing to a full fibre upgrade and a declining powerline network that meant my home office wasn't enjoying the benefits of the kind of connectivity that was unimaginable when I first plugged in a modem, I had ethernet run around the house by Tom at WestworldIT, which required the study to look less of a tip, facilitiating a much needed clean but also the shutdown of the server for a while, and as is often the case in these situations, checking the disks with smartctl showed that a couple of the aging second hand 3TB disks in it failed their tests. I had a pair of 4TB disks so went looking for some more slightly less aging second hand ones and found some via eBay at the Edinburgh Remakery.
As the array disks were holding up, I backed them up to a spare 4TB and started the replacement.

The Practical Stuff

Note: there are mistakes here. If you know anything about BTRFS you will probably be shouting at me.
The array was originally set up as RAID10 with metadata striped 1c3, so on three disks.
I replaced one of the failing disks with the spare and ran btrfs balance /mount. I should have possibly done btrfs replace here but both seem to be valid at this point. However, balance is a rich command and works best with filters, which are well described in the manual page for btrfs-balance, so running it as is is going to take some time.
When the new (second-hand) 4TB disks arrived the next day, I left them to warm up and let the balance complete.
The disks were a variety of 'desktop' (does anyone have a 4TB drive in a desktop these days - I suppose they do) and NAS disks, all around six years old. They all checked out with smartctl, so I started to replace the older 3TB disks using btrfs replace start <id> /dev/sdx /mount.
Each disk took about 24 hours, for around 2.5TB of data. The first two worked perfectly. Progress can be monitored with btrfs replace status /mount.
I did two disks and resized them, but the third appeared to be showing problems when trying to mount the volume. It appeared it might be the disk itself but smartctl didn't show anything. I swapped it out for the other spare 4TB that I used to back up (copying the data onto one of the 3TB disks and a bit of juggling).
However, here was my mistake: I picked out the last disk I replaced for the backup, and that must have been the one that had metadata on it, so I was now down to two metadata disks and the replace had got stuck and the volume would only mount read only, indicating that it was time to get data off and start again. You are welcome to shout now, as I did.
As I have backups I decided not to fight and just recreated the volume and started restoring and it looks like it's going to be a day or so of rsyncs.

Lessons learned

  1. Maintain backups
  2. Watch what state your metadata is in between swapping disks - I'm not 100% sure that this was the issue but it's what I suspect, and balancing metadata would probably have avoided it.
  3. Have patience - it's a slow process and rightly so. Cloning disks appears to be an alternative but I'm not sure how that would work with a RAID scheme like 1c3.
  4. Maintain backups.

As a note the current storage host is a 8GB Raspberry Pi 4, which is excellent as a server but less so in doing maintenance work, so that gets handed off to a laptop.

I set up offworld.fedisonic.cloud when it was apparent that Twitter was going to start circling the drain pretty quickly. I had an idea of running a Mastodon instance for anyone in my circle who wanted it, but with my usual overambition I started to think that it could be a business of some kind.
I first looked at Mastodon when it was announced back in 2016 or so, when it was becoming apparent when Twitter was getting gamed, and put up an instance on a VM, but the discovery process wasn't there and it was hard to find people and I took it down again. I made a mental note to watch ActivityPub and developments around it. I joined mastodon.social (and chaos.social, which is the opposite) when the toxicity in Twitter was starting to become the norm but didn't find it compelling, but still kept an eye on what was going on.
When That Guy squandered enough money to end world hunger on being That Guy on Twitter every day, enough was enough. I had a couple of underused servers so threw up a instance, invited people to see what the fuss was about and imported my followers as everyone was doing the same.
Now it's just short of a year later and I've closed that server. I'm running one for myself. set up in a better way, and of the people I invited, one has stayed on Mastodon.
Over the last year I've realised a few things:

  • Mastodon is not the Fediverse. It's shorthand for the network because it is the most popular platform, and we like to have an easy handle for these platforms.
  • Mastodon the application is technically simple to set up. I am a sysadmin of several million lifetimes' experience to be sure but it is made of common components and the documentation is well written, and if you can set up WordPress or Drupal it's not too much of a challenge.
  • Mastodon the service is challenging to maintain. The abillty to block instances and indeed users at the server level is powerful, but a year ago it relied on a hashtag which of which it quickly became apparent was open to abuse and indeed straightforward politics.
    Now, common blocklists have emerged and operators can choose what to use, but with any kind of success comes abuse, and it's obvious that public instances have bigger operating overheads than scaling their hardware.
  • Mastodon the software is resource hungry. At scale it is probably very efficient. With two (two!) active users it's overkill and things like scheduled deletes of busy follows could grind my server to a halt if that server was in the same timezone (if that makes sense). As a system that's designed with privacy in mind it's also difficult to troubleshoot as there are no logs by default (they can be switched on of course) so the main source of information is the job queues.
  • Technically, use cloud storage rather than on server storage, or at least don't use the same disk device. Cache as well, also not on the same disk device. It creates a lot of I/O.
  • Legal implications are getting more onerous across the world, and while they're a grey area for small instances, in the UK at least, we're not far from operators having to take a lot more responsibility for their users.

Running a small server with and for other people is a committment. Doing it for money would be something else again. I'm glad that there are businesses setting that up
On the actual social side of social media (remember that?), my experience of the Fediverse has been generally good. My feed seems to be largely cat pictures (especially on , but finding good people has been relatively easy. I'm not a big socialiser at all but I've long liked how the Internet works in that respect, and as ActivityPub is basically marshalling RSS feeds, we're back where were before we were rudely interrupted, just doing it in a way that makes more sense now.
I am going to try a couple of other platforms as I don't think Mastodon is the best option for a single user fediverse instance, or even two or three users. and work out how to aggregate different applications, like WordPress for example, so that's the next year sorted.

Pixelfed is a federated photo sharing service that is an alternative to Instagram and uses ActivityPub to share posts across other services such as Mastodon, Pleroma and indeed WordPress. It is one of the more mature Fediverse apps and has been in continuous development for several years.

As with Mastodon and other apps that can share images and other larger media files, it supports saving attachments to S3-compatible cloud storage. However, this isn't well documented so far, so here's a few notes on getting it to work, specifically with iDrive E2, a low cost E2 storage provider.

Pixelfed is a Laravel application and its user config is stored in a .env file in the application root. The S3 section looks like this:

## S3 Configuration (Post-Installer)
PF_ENABLE_CLOUD=true
FILESYSTEM_CLOUD=s3
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_DEFAULT_REGION=
AWS_BUCKET=
AWS_URL=
AWS_ENDPOINT=
AWS_USE_PATH_STYLE_ENDPOINT=false

IDrive E2 appears to be based on Minio but the priniciple is broadly similar for all S3 compatible services:

  1. Create a bucket
  2. Create credentials for that bucket
  3. Configure the application with those credentials

I think they're fairly self-explanatory in that the service will usually present them in the same way. The default region can be a bit of odd one but the naming seems to be arbitrary - I'm using iDrive's London region, which is labelled LDN, so go with what the service gives you.
AWS_URL is as it says, a standard URL, so it's https:// AWS_ENDPOINT / AWS_BUCKET. AWS_ENDPOINT is an important thing though - IDrive has the option of public and private endpoints with different hostnames. I'm not entirely clear on how a private endpoint works in this context - I would think it's a location that needs to be accessed with a shared key - but as you're publishing images online that's not going to be possible, and in testing I couldn't get Pixelfed to write to a private endpoint. In addition they're going to need to be public on an open server, so following the principle of least privilege, public and read-only works.
I had to comment AWS_USE_PATH_STYLE_ENDPOINT out as setting it to false didn't work for some reason. There's probably something in the code that assumes it's going to be an AWS URL or similar.

The Horizon dashboard was very useful for debugging. Again, this is a Laravel tool and part of the admin suite. I'm not that familiar with Laravel and but it basically appears to be the application manager. It writes to a log but that doesn't say much, and all the useful information is in the dashboard.

If you want to share images in the Fediverse, Pixelfed is the application you need. Its author, Daniel Supernault has been steadily improving it for years, and he's also currently working on a secure and federated messaging system intended as a drop-in replacement for direct messages. Cloud storage reduces your hosting costs as it's usually cheaper in bulk than VPS, and it's portable, so now you can replace Instagram with something under your control.

This is a bit of a niche one, not least because network-scripts is supposed to be on its way out as a configuration method, despite cPanel still not supporting NetworkManager properly. No problem people, it's only been around for ten years.
This may also be specific to OVH's network, but I haven't found a resource that covers it, so here are my notes.
Dedicated servers at OVH are allocated a /64 IPV6 range. The gateway is the last address in the range, which is slightly more eye crossing than with IPV4, but ends in FF:FF:FF:FF. However, in EL8 (and in Debian-derived distros to my knowledge), ifup doesn't add a default gateway if it can't ping the address, and it can't ping the address because it needs a default gateway. This also applies to IPV4 in OVH's network, which is a pain for automatic provisioning.
The network-scripts system is remarkably flexible, and wrangling it is a dying art, but the key here is the ifup-post script. This runs after ifcfg-<device> and goes through a sequence of the other scripts in /etc/sysconfig/network-scripts- you can see which ones if you look at the code.
One of these is route6-<device> and this is where you add your default route so it's added after the initial network config. In OVH images it is disabled, and perhaps confusingly, includes a list of static routes generated by cloud-init. Rename that one, and using your editor of choice (vim, you animals), open a new file vi route6-eno1 (for example) and enter your gateway and default route like this:

2001:xxxx:xxxx:xxff:00ff:00ff:00ff:00ff dev eno1
default via 2001:xxxx:xxxx:xxff:00ff:00ff:00ff:00ff dev eno1

Enable the script by making it executable chmod +x route6-eno1.
and restart the network with systemctl restart network.
and you should now be able to ping the IPV6 host of your choice (let's face it, your local google.com gateway).

On his blog, Terence Eden wrote about the Time I Invented Twitter and it's an interesting proposition which would have just been slightly ahead of its time in 2003. Microblogging probably needed Internet enabled phones to take off, like social networking in general.

Douglas Adams was a very intelligent man who liked his gadgets - Macintoshes, synths and left-handed Casio guitars appeared in the first Dirk Gently novel, Dirk Gently's Holistic Detective Agency, and, with The Book in The Hitchhiker's Guide to the Galaxy he, at the very least, created the idea of a portable device that knew everything.
When the commercial Internet started, Adams got involved in games, co-founding The Digital Village, developers of the game Starship Titanic, who then started h2g2, a collaborative dictionary project described as the Earth edition of the Hitchhikers' Guide to the Galaxy.
In his article My Vision for H2G2, written in 2000, he wrote this:

...when you write in something as simple as 'The coffee here is lousy!' the Guide will know exactly what to do with that information and where to put it. And if you see, a few seconds later, a note which says 'Yes, but the cheesecake is good' it might be worth looking round the other tables to see who you've just made contact with.

That resonated with me, and I filed it away. At the time I had a kind of mobile Internet setup, with a Palm V and a Nokia 8210 that communicated by infrared, that I could actually browse the web on, and with a bit of coaxing even use telnet on. Given this experience with the mobile web, I got tipped off about a trial project that involved the Handspring Visor, a PalmOS device that improved on Palm's hardware by adding an expansion slot, that among other things, had a GSM phone module and a TCP/IP stack (my memory says it was actually Trumpet Winsock ported to PalmOS, but my memory sometimes makes things up). For £99 and couple of feedback sessions, I had a connected handheld device.

At the feedback session, they walked me and other trial users through what they had in mind, and from what I can remember, it was fairly grey stuff. Online banking seemed to feature quite a lot, and it turned out that the service was rather limited by the data provider and a relatively high cost for data at the time. It might have even been limited in availability from 8am to 8pm or something like that. Given the lack of ideas coming from the company, I quoted DNA's idea above, which seemed to me to be a logical progression, and it seemed that the idea had never occurred to the company.

The project was abandoned. I heard later that the running cost was unpopular - data pricing was on top of airtime, which wasn't cheap back then, but also that the applications they were aiming at just didn't catch the imagination. I held onto the Visor for a while but it more or less died when PalmOS died, and by the first generation of genuinely smart handheld devices were starting to appear.

To embrace that in 2003 was to look forward from blogging, which, let's not forget, was huge then, to something much more personal, but also that would as Adams imagined, hopefully add to the sum total of global knowledge.

(Without disabling everything else)

My main laptop is a Lenovo X1 Yoga 2nd Gen running Manjaro KDE Plasma. Lovely machine and does what I want it to do. As a Yoga device, it has a touchscreen, which I don't use a lot and have often thought of disabling.

The other day, the laptop had the wrong kind of drop, which has cracked the touchscreen in the corner. This hasn't affected the display at all but has messed up the touchscreen input so that it keeps getting random signals that trigger events, which was sufficiently intrusive to need to turn off the touchscreen

The first thing I found was this from the Manjaro Forum. Tl;dr, disable the module that powers the touchscreen with sudo modprobe -r usbhid and make it permanent by creating a blacklist at /etc/modprobe.d/blacklist/wacom that contains the following:

blacklist wacom
blacklist usbhid

and restart.
This worked fine when just on the laptop, but I have a multi-monitor desktop setup that has a USB mouse and keyboard, and when I came to start work this morning, neither worked. Enabling usbhid in the blacklist just brought the spurious touches back.
There was going to be something that creates rules to selectively allow USB devices, and that something is USBGuard.
The Arch Wiki documents it well, but basically install with pacman -Sy usbguard (or your software manager of choice), and create your ruleset as root with usbguard generate-policy > /etc/usbguard/rules.conf.
This lists all your connected devices as allowed, including the touchscreen:

allow id 056a:50b6 serial "" name "Pen and multitouch sensor" hash "B1HYEaAtN9VpnKbIK5GQeZFfg3XN7EAAeQUvTx5zIhk=" parent-hash "jEP/6WzviqdJ5VSeTUY8PatCNBKeaREvo2OqdplND/o=" via-port "1-10" with-interface { 03:00:00 03:00:00 } with-connect-type "not used"

To disable it, change allow to block to stop it being processed, or reject to stop the device being loaded at all. At the moment I have it set to block.
Start USBGuard with systemctl start usbguard and enable it on boot with systemctl enable usbguard.
This stopped the touchscreen responding but kept the USB keyboard and mouse working. I haven't tested it across a reboot yet but I can't see why it won't continue to work.

The Apple TV series Mythic Quest had an episode in which the game found it had an extreme right wing problem. Their solution was to corral the right wingers in a server where they could shout at each other and fight as much as they wanted without bothering other players.

The Fediverse, running, as it does, largely on free software, came about in part due to Twitter and other platforms' unwillingness or inability to deal with an extreme right wing problem. However, as it's free and open source software, the bad actors were also free to create their own instances and interact. The response was filtering, blocking and defederation.

If you run a Mastodon, or other Fediverse social media instance, even if it's for yourself, it's one of the most powerful tools you have. You can filter hashtags, users and whole instances both personally and globally.

Lists have developed over the years along with the tools to apply them, but they have often been personal efforts. The hashtag can also be used, but it's something of a blunt instrument and is too easily hijacked for personal opinions and even feuds.

The current attempt to deal with this in an effective way is the Oliphant.social blocklist files, created through a consensus of ten of the most active fediverse instances This produces a collection of blocklists that can be applied as an administrator sees fit. Follow the instructions there to download, maintain and apply them. At the moment I apply the Tier 0 blocklist.

In addition, Ro Iskinda's The Bad Space collects the most commonly reported bad actors and new ones that appear, and can be used with an API to check if you encounter a doubtful user or instance.

I got my first Android phone, a Moto Droid, in 2010. It's wasn't my first smartphone as that had been a quest with varying degrees of success for years (I still miss the Nokia E series keyboard phones a bit).

Among the things installed on it was Facebook. Even then I was suitably suspicious so wanted to remove it, except I couldn't without rooting as it was a system app, and this was the awful wrapper-around-the-website version. However, being a system app, it still had full access to your phone. Which was nice.

However, by 2010 it had been being installed for a few years, if not as native app, as phone provider bloatware, and I think this was a major contributor to Facebook's takeup. You took your phone out of its box, went through the welcome screens and there it was. You could sign up and you could speak your brains to your heart's content.

I think this was as important a change in Internet culture as Microsoft putting an icon on the Windows 95 desktop labelled 'The Internet'. No, wait, come back.

Large scale social networks were not a new thing when Facebook went global. Friends Reunited created networks based on school connections in the late 90s. Six Degrees, Livejournal, Friendster, Myspace, Orkut all came and went to varying degrees. Google tried and couldn't get it to work, even when betting the farm on it. In that context, Facebook survived and persisted by learning continually, starting with putting the icon on your desktop/phone screen.

It annoys me that I have to keep a Facebook presence, for a few friends and that the parents of my daughter's school use it for messaging, but it and WhatsApp are the lowest common denominators for messaging, but I've handed that off to a bridge - I'm testing Beeper, but could fall back to Matrix bridges (same thing, self hosted) if it doesn't work out.

Of course, this is little compared to having a whole device under your control, especially when Google has now seemingly decided that it doesn't believe in the open Internet any more.

I'm not going to say 'this is what you should do' on this blog at any time, but I am going to say 'this is what I'm going to do'. I've been online for nearly 30 years, nothing compared to some people, and one of the compelling things about the Internet for me is interoperability. Walled gardens always become overgrown, and when someone decides to effectively turn one inside out, then it's time to do some weeding (yes, I'm prone to that sort of hyperbole, sorry).

I have a virtual OPNSense router as the gateway to a production network based on Proxmox servers (more about this another day). All VMs and containers get an unrouted IP address in the 10.x.x.x range and I have a couple of IP ranges for public access, generally but not always behind Haproxy. This doesn\'t seem to be documented properly elsewhere. ## A quick summary: 1. Assign IP range(s) as Virtual IPs 2. Create a One to One NAT between an external IP from a virtual IP range and an internal address 3. Create a WAN rule for the ports you want open to the internal address. ### 1. Assign IP ranges as Virtual IPs Our servers are with OVH and connected by their vRack service, which is remarkably flexible - connected servers can use an unrouted IP network, which is shared between them automatically, and for routed IP addresses, you rent a range (minimum is a /30, for reasons I\'ll explain shortly) and allocate it to the vRack, and in the simplest usage they are allocated to a physical interface on a server and routed to the ranges\' gateway, which is, somewhat counter-intuitively, the second to last address in the range, which is why the smallest range possible is a /30: | Range | CIDR | Gateway | Usable addresses | |-------|------|---------|------------------| | 192.168.10.20 | /30 | 192.168.1.22 | 1 | | 192.168.10.16 | /29 | 192.168.1.22 | 5 | | 192.168.10.16 | /28 | 192.168.1.29 | 12 | For the purposes of this article, I\'ll add a /29 as I have a routed one that\'s about to be reallocated from my account. a) Go to Interfaces > Virtual IPs > Settings b) Press \'+\' c) For the mode in this context, you will use Proxy ARP as IP Alias is for a single IP - the \'Single IP\' option is disabled. d) Select WAN for the interface e) Select \'Network\' for Type f) Enter the network range in the Address field without the CIDR range and select that from the dropdown. You don\'t define a gateway at this point, which also seems a bit counterintuitive but that comes at allocation time. g) Set a description and save. h) Apply changes on the main page. ### 2. Enable Reflection in the Firewall settings UPDATE 19/12/2022: I\'ve been trying to work out why I couldn\'t search other ActivityPub services from my Mastodon instance and found that on my local network at least, I couldn\'t find the address of my blog, so this is the fix for being able to access services internally. a) Go to Firewall > Settings > Advanced b) In the Network Address Translation section, check the Reflection options that you use. I went with all three just to be sure. c) Save. ### 2. Add a One to One NAT a) Go to Firewall > NAT > One-to-One b) Press \'+\' c) For \'Interface\' select \'WAN\' d) For Type selected \'BINAT\' so outbound traffic is rewritten as well e) For \'External network\' select an IP from your allocation and give it the CIDR range /32 f) For \'Source\' select \'Single host or Network\' and enter the IP address of the destination server/VM/container, again with a CIDR range of /32. This is also somewhat non-intuitive at first as the pfSense instructions say the CIDR range should be the same for both addresses and in this case we have a /29 for external traffic and a /22 for internal traffic. g) Set \'Destination\' to \'any\' h) Add a description i) Set NAT reflection to \'Use system default\' or as required. The default is set in Firewall:Settings:Advanced and is Disabled. j) Save k) Apply changes on the main page ### 3. Create firewall rules a) Go to Firewall > Rules > WAN b) Press \'+\' c) Action is \'Pass\' d) Interface is WAN e) Direction is in f) TCP/IP version is generally TCP g) Source is \'any\' h) Destination port range will usually a single port for a protocol such as HTTPS i) Destination is \'Single host or Network\' and the value is the IP address of the backend device with the correct CIDR range for your network, so in this case it\'s 10.10.x.x/22 j) Optionally set a description, but it\'s advisable as it\'s useful for documentation. k) Save l) Apply changes on the main page. Revision 2

This took a few days to work out and I finally found it buried in an Opnsense forum post, so here it is in a slightly easier to find way.

Objective

To forward HTTP(S) traffic to a virtual machine running Nginx (or any other HTTP proxy or load balancer) through a gateway running OPNsense.

Solution

1. Change the Web GUI port

Even if the Web GUI is inaccessible from the public interface of an Opnsense VM, it's still listening on it on both port 80 and port 443, so change the port.
In System > Settings > Administration > Web GUI, assuming you have set the protocol to HTTPS, change TCP Port to something other than 443:
Opnsense Web GUI TCP Port

Press Save and the configuration will reload. You may need to click on the link in the notification bar for it to complete.

2. Create aliases for the HTTP(S) service

This was the bit I didn't get. I was trying to define NAT ports and IPs manually. More about aliases in OPNSense
Assume your internal network is 192.168.1.0/24 and your web server is at 192.168.1.10. The WAN address is the address is the IP address of your gateway.
In Firewall > Aliases > View

  1. Click the plus button to add a new alias.
  2. Enter the following:
    • Enabled: check
    • Type: Hosts
    • Name: Reverse Proxy (or whatever, this is the identifier)
    • Content: 192.168.1.10
    • Statistics (as required)
    • Description: The reverse proxy host
  3. Click Save
  4. Click the plus button to add a new alias again.
  5. Enter the following:
    • Enabled: check
    • Type: Port(s)
    • Name: Reverse Proxy Ports
    • Content: 80, 443 (each port is a separate item)
    • Statistics: (as required)
    • Description: Reverse proxy ports
  6. Click Save and Apply Settings.

3. Create a NAT rule

In Firewall > NAT > Port Forward

  1. Click the Plus button to add a new Port Forward rule
  2. Enter the following:
    • Interface: WAN
    • TCP/IP version: IPv4
    • Protocol: TCP
    • Destination: WAN Address
    • Destination port range: from Reverse Proxy Ports to Reverse Proxy Ports (this will be in the dropdown under Aliases)
    • Redirect Target IP: Reverse Proxy (from the dropdown)
    • Redirect Target Ports: Reverse Proxy Ports (from the dropdown)
    • Pool Options: Default
    • NAT Reflection: Enable
    • Filter rule association: Rule
  3. Click Save and Apply settings.
    This also creates a rule in Filewall > Rules > WAN for the aliases.

If your web server/load balancer VM is up and you at least have a default configuration available, you should get a page.

H/T to huticip in the Opnsense forums.