Posted on
by

Preamble ramble

There aren't a lot of real world examples of using BTRFS around online. Some of the most useful advice I found was in r/BTRFS on Reddit but it's hard to actually find user experiences beyond the theroretical.
This is sort of one, but the tl;dr is have good backups and learn from your experiences. The practical stuff starts at 'The Practical Stuff'.

I've had a home disk array for a long time for various reasons. As much as anything I have a home music streaming system as I prefer to buy and own music, even if it is as a download, and over the years I have digitised CDs, originally into iTunes but currently using Funkwhale.
The current disk array is a Terramaster D5-300, a five disk JBOD array that has Windows based software RAID support, but, being Just A Bunch of Disks, can be set up with other on-disk RAID architectures, so, as a Linux user, there were a couple of immediate choices: ZFS, or BTRFS.
ZFS is an enterprise level filesystem that was originally developed by Sun (RIP) but was ported to Linux when Sun's BSD based OS Solaris was open sourced before the Oracle takeover and was forked as OpenZFS. ZFS itself is a great filesystem, but at the time (2016) OpenZFS development had slowed and it's not really a good fit for the kind of low powered machine that goes with a home setup (I think I set it up on an Acer Revo at first).
BTRFS, on the other hand, is in the Linux kernel, so if something runs the kernel, it should work, and just needs the btrfs-tools package to manage it, so it can be configured and moved around if necessary.

At the beginning of this month, owing to a full fibre upgrade and a declining powerline network that meant my home office wasn't enjoying the benefits of the kind of connectivity that was unimaginable when I first plugged in a modem, I had ethernet run around the house by Tom at WestworldIT, which required the study to look less of a tip, facilitiating a much needed clean but also the shutdown of the server for a while, and as is often the case in these situations, checking the disks with smartctl showed that a couple of the aging second hand 3TB disks in it failed their tests. I had a pair of 4TB disks so went looking for some more slightly less aging second hand ones and found some via eBay at the Edinburgh Remakery.
As the array disks were holding up, I backed them up to a spare 4TB and started the replacement.

The Practical Stuff

Note: there are mistakes here. If you know anything about BTRFS you will probably be shouting at me.
The array was originally set up as RAID10 with metadata striped 1c3, so on three disks.
I replaced one of the failing disks with the spare and ran btrfs balance /mount. I should have possibly done btrfs replace here but both seem to be valid at this point. However, balance is a rich command and works best with filters, which are well described in the manual page for btrfs-balance, so running it as is is going to take some time.
When the new (second-hand) 4TB disks arrived the next day, I left them to warm up and let the balance complete.
The disks were a variety of 'desktop' (does anyone have a 4TB drive in a desktop these days - I suppose they do) and NAS disks, all around six years old. They all checked out with smartctl, so I started to replace the older 3TB disks using btrfs replace start <id> /dev/sdx /mount.
Each disk took about 24 hours, for around 2.5TB of data. The first two worked perfectly. Progress can be monitored with btrfs replace status /mount.
I did two disks and resized them, but the third appeared to be showing problems when trying to mount the volume. It appeared it might be the disk itself but smartctl didn't show anything. I swapped it out for the other spare 4TB that I used to back up (copying the data onto one of the 3TB disks and a bit of juggling).
However, here was my mistake: I picked out the last disk I replaced for the backup, and that must have been the one that had metadata on it, so I was now down to two metadata disks and the replace had got stuck and the volume would only mount read only, indicating that it was time to get data off and start again. You are welcome to shout now, as I did.
As I have backups I decided not to fight and just recreated the volume and started restoring and it looks like it's going to be a day or so of rsyncs.

Lessons learned

  1. Maintain backups
  2. Watch what state your metadata is in between swapping disks - I'm not 100% sure that this was the issue but it's what I suspect, and balancing metadata would probably have avoided it.
  3. Have patience - it's a slow process and rightly so. Cloning disks appears to be an alternative but I'm not sure how that would work with a RAID scheme like 1c3.
  4. Maintain backups.

As a note the current storage host is a 8GB Raspberry Pi 4, which is excellent as a server but less so in doing maintenance work, so that gets handed off to a laptop.

Posted on
by

I set up offworld.fedisonic.cloud when it was apparent that Twitter was going to start circling the drain pretty quickly. I had an idea of running a Mastodon instance for anyone in my circle who wanted it, but with my usual overambition I started to think that it could be a business of some kind.
I first looked at Mastodon when it was announced back in 2016 or so, when it was becoming apparent when Twitter was getting gamed, and put up an instance on a VM, but the discovery process wasn't there and it was hard to find people and I took it down again. I made a mental note to watch ActivityPub and developments around it. I joined mastodon.social (and chaos.social, which is the opposite) when the toxicity in Twitter was starting to become the norm but didn't find it compelling, but still kept an eye on what was going on.
When That Guy squandered enough money to end world hunger on being That Guy on Twitter every day, enough was enough. I had a couple of underused servers so threw up a instance, invited people to see what the fuss was about and imported my followers as everyone was doing the same.
Now it's just short of a year later and I've closed that server. I'm running one for myself. set up in a better way, and of the people I invited, one has stayed on Mastodon.
Over the last year I've realised a few things:

  • Mastodon is not the Fediverse. It's shorthand for the network because it is the most popular platform, and we like to have an easy handle for these platforms.
  • Mastodon the application is technically simple to set up. I am a sysadmin of several million lifetimes' experience to be sure but it is made of common components and the documentation is well written, and if you can set up WordPress or Drupal it's not too much of a challenge.
  • Mastodon the service is challenging to maintain. The abillty to block instances and indeed users at the server level is powerful, but a year ago it relied on a hashtag which of which it quickly became apparent was open to abuse and indeed straightforward politics.
    Now, common blocklists have emerged and operators can choose what to use, but with any kind of success comes abuse, and it's obvious that public instances have bigger operating overheads than scaling their hardware.
  • Mastodon the software is resource hungry. At scale it is probably very efficient. With two (two!) active users it's overkill and things like scheduled deletes of busy follows could grind my server to a halt if that server was in the same timezone (if that makes sense). As a system that's designed with privacy in mind it's also difficult to troubleshoot as there are no logs by default (they can be switched on of course) so the main source of information is the job queues.
  • Technically, use cloud storage rather than on server storage, or at least don't use the same disk device. Cache as well, also not on the same disk device. It creates a lot of I/O.
  • Legal implications are getting more onerous across the world, and while they're a grey area for small instances, in the UK at least, we're not far from operators having to take a lot more responsibility for their users.

Running a small server with and for other people is a committment. Doing it for money would be something else again. I'm glad that there are businesses setting that up
On the actual social side of social media (remember that?), my experience of the Fediverse has been generally good. My feed seems to be largely cat pictures (especially on , but finding good people has been relatively easy. I'm not a big socialiser at all but I've long liked how the Internet works in that respect, and as ActivityPub is basically marshalling RSS feeds, we're back where were before we were rudely interrupted, just doing it in a way that makes more sense now.
I am going to try a couple of other platforms as I don't think Mastodon is the best option for a single user fediverse instance, or even two or three users. and work out how to aggregate different applications, like WordPress for example, so that's the next year sorted.

Posted on
by

A touch of : I worked at Easynet in the early days of the UK commercial Internet, and we packaged browser in our welcome pack as it was initially free for non-commercial purposes and we stated we were providing it as an option for browsing the web rather than supported software.
The pricing model changed in February 1995 and the company had a visit from a sales person at Netscape's UK office, who presented them with a bill for the retail price of the number of copies of the browser they had distributed.
A deal was negotiated based on that (it might have been zero, I never found out the details) and the pricing policy changed within a year, quite possibly because they'd had the same response in many ISPs.
Netscape's position as most popular web browser lasted as long as it took for Microsoft to license the same code base (Mosaic, licensed from Spyglass, who had in turn licensed the code from the University of Illinois) and improve on it and include it with Windows. Their other corporate products, Web Server, Identity Server, etc which were pretty good at the end of the 90s, also got passed around various companies and survive as open source versions that are arguably more successful than their commercial antecedents.
That could have been Mark Andreesen's legacy, but it was all born out of a want to get rich, and as he's got older, he's gone the same way as so many of the early tech bros, and like many of them, not actually doing anything useful as a means.
This piece puts it in a much better way.

Posted on
by

Pixelfed is a federated photo sharing service that is an alternative to Instagram and uses ActivityPub to share posts across other services such as Mastodon, Pleroma and indeed WordPress. It is one of the more mature Fediverse apps and has been in continuous development for several years.

As with Mastodon and other apps that can share images and other larger media files, it supports saving attachments to S3-compatible cloud storage. However, this isn't well documented so far, so here's a few notes on getting it to work, specifically with iDrive E2, a low cost E2 storage provider.

Pixelfed is a Laravel application and its user config is stored in a .env file in the application root. The S3 section looks like this:

## S3 Configuration (Post-Installer)
PF_ENABLE_CLOUD=true
FILESYSTEM_CLOUD=s3
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_DEFAULT_REGION=
AWS_BUCKET=
AWS_URL=
AWS_ENDPOINT=
AWS_USE_PATH_STYLE_ENDPOINT=false

IDrive E2 appears to be based on Minio but the priniciple is broadly similar for all S3 compatible services:

  1. Create a bucket
  2. Create credentials for that bucket
  3. Configure the application with those credentials

I think they're fairly self-explanatory in that the service will usually present them in the same way. The default region can be a bit of odd one but the naming seems to be arbitrary - I'm using iDrive's London region, which is labelled LDN, so go with what the service gives you.
AWS_URL is as it says, a standard URL, so it's https:// AWS_ENDPOINT / AWS_BUCKET. AWS_ENDPOINT is an important thing though - IDrive has the option of public and private endpoints with different hostnames. I'm not entirely clear on how a private endpoint works in this context - I would think it's a location that needs to be accessed with a shared key - but as you're publishing images online that's not going to be possible, and in testing I couldn't get Pixelfed to write to a private endpoint. In addition they're going to need to be public on an open server, so following the principle of least privilege, public and read-only works.
I had to comment AWS_USE_PATH_STYLE_ENDPOINT out as setting it to false didn't work for some reason. There's probably something in the code that assumes it's going to be an AWS URL or similar.

The Horizon dashboard was very useful for debugging. Again, this is a Laravel tool and part of the admin suite. I'm not that familiar with Laravel and but it basically appears to be the application manager. It writes to a log but that doesn't say much, and all the useful information is in the dashboard.

If you want to share images in the Fediverse, Pixelfed is the application you need. Its author, Daniel Supernault has been steadily improving it for years, and he's also currently working on a secure and federated messaging system intended as a drop-in replacement for direct messages. Cloud storage reduces your hosting costs as it's usually cheaper in bulk than VPS, and it's portable, so now you can replace Instagram with something under your control.

Posted on
by

This is a bit of a niche one, not least because network-scripts is supposed to be on its way out as a configuration method, despite cPanel still not supporting NetworkManager properly. No problem people, it's only been around for ten years.
This may also be specific to OVH's network, but I haven't found a resource that covers it, so here are my notes.
Dedicated servers at OVH are allocated a /64 IPV6 range. The gateway is the last address in the range, which is slightly more eye crossing than with IPV4, but ends in FF:FF:FF:FF. However, in EL8 (and in Debian-derived distros to my knowledge), ifup doesn't add a default gateway if it can't ping the address, and it can't ping the address because it needs a default gateway. This also applies to IPV4 in OVH's network, which is a pain for automatic provisioning.
The network-scripts system is remarkably flexible, and wrangling it is a dying art, but the key here is the ifup-post script. This runs after ifcfg-<device> and goes through a sequence of the other scripts in /etc/sysconfig/network-scripts- you can see which ones if you look at the code.
One of these is route6-<device> and this is where you add your default route so it's added after the initial network config. In OVH images it is disabled, and perhaps confusingly, includes a list of static routes generated by cloud-init. Rename that one, and using your editor of choice (vim, you animals), open a new file vi route6-eno1 (for example) and enter your gateway and default route like this:

2001:xxxx:xxxx:xxff:00ff:00ff:00ff:00ff dev eno1
default via 2001:xxxx:xxxx:xxff:00ff:00ff:00ff:00ff dev eno1

Enable the script by making it executable chmod +x route6-eno1.
and restart the network with systemctl restart network.
and you should now be able to ping the IPV6 host of your choice (let's face it, your local google.com gateway).

Posted on
by

On his blog, Terence Eden wrote about the Time I Invented Twitter and it's an interesting proposition which would have just been slightly ahead of its time in 2003. Microblogging probably needed Internet enabled phones to take off, like social networking in general.

Douglas Adams was a very intelligent man who liked his gadgets - Macintoshes, synths and left-handed Casio guitars appeared in the first Dirk Gently novel, Dirk Gently's Holistic Detective Agency, and, with The Book in The Hitchhiker's Guide to the Galaxy he, at the very least, created the idea of a portable device that knew everything.
When the commercial Internet started, Adams got involved in games, co-founding The Digital Village, developers of the game Starship Titanic, who then started h2g2, a collaborative dictionary project described as the Earth edition of the Hitchhikers' Guide to the Galaxy.
In his article My Vision for H2G2, written in 2000, he wrote this:

...when you write in something as simple as 'The coffee here is lousy!' the Guide will know exactly what to do with that information and where to put it. And if you see, a few seconds later, a note which says 'Yes, but the cheesecake is good' it might be worth looking round the other tables to see who you've just made contact with.

That resonated with me, and I filed it away. At the time I had a kind of mobile Internet setup, with a Palm V and a Nokia 8210 that communicated by infrared, that I could actually browse the web on, and with a bit of coaxing even use telnet on. Given this experience with the mobile web, I got tipped off about a trial project that involved the Handspring Visor, a PalmOS device that improved on Palm's hardware by adding an expansion slot, that among other things, had a GSM phone module and a TCP/IP stack (my memory says it was actually Trumpet Winsock ported to PalmOS, but my memory sometimes makes things up). For £99 and couple of feedback sessions, I had a connected handheld device.

At the feedback session, they walked me and other trial users through what they had in mind, and from what I can remember, it was fairly grey stuff. Online banking seemed to feature quite a lot, and it turned out that the service was rather limited by the data provider and a relatively high cost for data at the time. It might have even been limited in availability from 8am to 8pm or something like that. Given the lack of ideas coming from the company, I quoted DNA's idea above, which seemed to me to be a logical progression, and it seemed that the idea had never occurred to the company.

The project was abandoned. I heard later that the running cost was unpopular - data pricing was on top of airtime, which wasn't cheap back then, but also that the applications they were aiming at just didn't catch the imagination. I held onto the Visor for a while but it more or less died when PalmOS died, and by the first generation of genuinely smart handheld devices were starting to appear.

To embrace that in 2003 was to look forward from blogging, which, let's not forget, was huge then, to something much more personal, but also that would as Adams imagined, hopefully add to the sum total of global knowledge.

Posted on
by

(Without disabling everything else)

My main laptop is a Lenovo X1 Yoga 2nd Gen running Manjaro KDE Plasma. Lovely machine and does what I want it to do. As a Yoga device, it has a touchscreen, which I don't use a lot and have often thought of disabling.

The other day, the laptop had the wrong kind of drop, which has cracked the touchscreen in the corner. This hasn't affected the display at all but has messed up the touchscreen input so that it keeps getting random signals that trigger events, which was sufficiently intrusive to need to turn off the touchscreen

The first thing I found was this from the Manjaro Forum. Tl;dr, disable the module that powers the touchscreen with sudo modprobe -r usbhid and make it permanent by creating a blacklist at /etc/modprobe.d/blacklist/wacom that contains the following:

blacklist wacom
blacklist usbhid

and restart.
This worked fine when just on the laptop, but I have a multi-monitor desktop setup that has a USB mouse and keyboard, and when I came to start work this morning, neither worked. Enabling usbhid in the blacklist just brought the spurious touches back.
There was going to be something that creates rules to selectively allow USB devices, and that something is USBGuard.
The Arch Wiki documents it well, but basically install with pacman -Sy usbguard (or your software manager of choice), and create your ruleset as root with usbguard generate-policy > /etc/usbguard/rules.conf.
This lists all your connected devices as allowed, including the touchscreen:

allow id 056a:50b6 serial "" name "Pen and multitouch sensor" hash "B1HYEaAtN9VpnKbIK5GQeZFfg3XN7EAAeQUvTx5zIhk=" parent-hash "jEP/6WzviqdJ5VSeTUY8PatCNBKeaREvo2OqdplND/o=" via-port "1-10" with-interface { 03:00:00 03:00:00 } with-connect-type "not used"

To disable it, change allow to block to stop it being processed, or reject to stop the device being loaded at all. At the moment I have it set to block.
Start USBGuard with systemctl start usbguard and enable it on boot with systemctl enable usbguard.
This stopped the touchscreen responding but kept the USB keyboard and mouse working. I haven't tested it across a reboot yet but I can't see why it won't continue to work.

Posted on
by

The Apple TV series Mythic Quest had an episode in which the game found it had an extreme right wing problem. Their solution was to corral the right wingers in a server where they could shout at each other and fight as much as they wanted without bothering other players.

The Fediverse, running, as it does, largely on free software, came about in part due to Twitter and other platforms' unwillingness or inability to deal with an extreme right wing problem. However, as it's free and open source software, the bad actors were also free to create their own instances and interact. The response was filtering, blocking and defederation.

If you run a Mastodon, or other Fediverse social media instance, even if it's for yourself, it's one of the most powerful tools you have. You can filter hashtags, users and whole instances both personally and globally.

Lists have developed over the years along with the tools to apply them, but they have often been personal efforts. The hashtag can also be used, but it's something of a blunt instrument and is too easily hijacked for personal opinions and even feuds.

The current attempt to deal with this in an effective way is the Oliphant.social blocklist files, created through a consensus of ten of the most active fediverse instances This produces a collection of blocklists that can be applied as an administrator sees fit. Follow the instructions there to download, maintain and apply them. At the moment I apply the Tier 0 blocklist.

In addition, Ro Iskinda's The Bad Space collects the most commonly reported bad actors and new ones that appear, and can be used with an API to check if you encounter a doubtful user or instance.

Posted on
by

I got my first Android phone, a Moto Droid, in 2010. It's wasn't my first smartphone as that had been a quest with varying degrees of success for years (I still miss the Nokia E series keyboard phones a bit).

Among the things installed on it was Facebook. Even then I was suitably suspicious so wanted to remove it, except I couldn't without rooting as it was a system app, and this was the awful wrapper-around-the-website version. However, being a system app, it still had full access to your phone. Which was nice.

However, by 2010 it had been being installed for a few years, if not as native app, as phone provider bloatware, and I think this was a major contributor to Facebook's takeup. You took your phone out of its box, went through the welcome screens and there it was. You could sign up and you could speak your brains to your heart's content.

I think this was as important a change in Internet culture as Microsoft putting an icon on the Windows 95 desktop labelled 'The Internet'. No, wait, come back.

Large scale social networks were not a new thing when Facebook went global. Friends Reunited created networks based on school connections in the late 90s. Six Degrees, Livejournal, Friendster, Myspace, Orkut all came and went to varying degrees. Google tried and couldn't get it to work, even when betting the farm on it. In that context, Facebook survived and persisted by learning continually, starting with putting the icon on your desktop/phone screen.

It annoys me that I have to keep a Facebook presence, for a few friends and that the parents of my daughter's school use it for messaging, but it and WhatsApp are the lowest common denominators for messaging, but I've handed that off to a bridge - I'm testing Beeper, but could fall back to Matrix bridges (same thing, self hosted) if it doesn't work out.

Of course, this is little compared to having a whole device under your control, especially when Google has now seemingly decided that it doesn't believe in the open Internet any more.

I'm not going to say 'this is what you should do' on this blog at any time, but I am going to say 'this is what I'm going to do'. I've been online for nearly 30 years, nothing compared to some people, and one of the compelling things about the Internet for me is interoperability. Walled gardens always become overgrown, and when someone decides to effectively turn one inside out, then it's time to do some weeding (yes, I'm prone to that sort of hyperbole, sorry).

Posted on
by

I deal with a number of clients in the legal profession, and this morning one of them forwarded me an email which he thought was suspicious. It appeared to be sent from a Rob Moore at a company called Progressive Property, and the body of the mail was this:

Good Afternoon,

I hope you are having a good weekend

I am instructed to deal with your firm for the purchase of a property, we are real estate company acting on behalf my clients .

The Executors appointed were her husband (predeceased) and another firm of solicitors who have renounced. The beneficiaries are the children, Tania Beaumont and Catherine Ried. They are applying for the Grant of Probate as residuary beneficiaries and therefore acting as Executors.

Please can you contact them to give a quote. They are happy to keep this probate sale with us.

Let me know when informations you need, they are looking to market the property around £3.25m.

Please can you copy in Tania’s husband into correspondence, his name is Peter Beaumont and he is very helpful.

A couple of GMail addresses and a plausible looking footer, except there was no From: address (not necessary of course, but generally expected), and a footer which included the logo of Progressive Property and the email rob@progresssiveproperty.com, and a legal disclaimer footer of the type that every legal firm has, except again, on a quick search the company it purported to be from didn't match anything else and the address didn't match the company at all

Yes, three s's. On additional readings the body of the mail looks increasingly like a 419 scam every time you read it, and the domain name returns Cloudflare records and a Microsoft Exchange login page.

It occurred to me to look up Progressive Property and of course there's a successful UK company called and Rob Moore is one of the founders. Obviously they're in no way related to this scam but it was increasingly understandable how much trust this could create, and it was clearly aimed at a particular business sector.