Poor user experience - or DevOps and definitely no SRE.

 In my continuation to find examples of where software systems have not moved beyond the SDLC of the 1980s and 90s, or have started doing DevOps, but haven't realised that now we're in to SRE, and if you think SRE is DevOps, then there's a problem.

DevOps = Automation, implement the new features, make sure it's reliable and stable, make sure "we" can fix an issue when it occurs.

Right, you probably think that includes SRE because you're thinking about the client/stakeholder, but how do they (the client) know if the problem is them or you?

So many organisations don't provide enough information to a client when they have an issue, either through an update, or an outage of some kind.

SRE requires our understanding of the user, not just in how they use the application, but making sure they don't have to be technical. It's not down to the user to uninstall and re-install the application and then have to log back in - that to a non-technical user is the biggest "bug bear" you could ask them to do. We're now in the 21st century and if you're a large organisation with lots of clients, should be able to think more about your customers experience. Yes, SRE has some nice principles for determining whether the user is experiencing a degradation in performance, or whether developers should be focusing on reliability rather than new features. BUT we should still be thinking, are we putting too much on the end user and assuming that they are comfortable in re-installing, or want keep putting in their password every week?

These are areas that we should be thinking about if we want to make our software more exceptional than the competition. If you can provide the experience to users that saves them logging in every 5 minutes, or having to re-install manually, then you're moving into a whole new area of customer satisfaction.

Today's poor practice;

Southern Water's application that 1000s of sea and river swimmers rely on at this time of year due to the amount of sewage discharges they do, even in non-extreme weather conditions is out for the entire day - see the picture attached.

https://riversandseaswatch.southernwater.co.uk/release-history?BathingSite=FOLKESTONE&Status=Genuine

Yesterday Tennis TV had performed an update without checking the client applications, causing the App to just quit with no warning messages - this in turn causing the user to become frustrated and panic.

Don't slip back to the 1970 and 1980s method of customer support and hide behind the Internet and ChatBots (in the 70s and 80s it was the phone), and not provide adequate information to the users. Today there are many practices and tooling that we should not be seeing this type of service as users.

Step in to the 21st century and start using DevOps environments to test your updates and upgrades for both application and infrastructure before deploying to LIVE. Ensure that your pre-Live environments have at least 1 of every resource so that there is NOTHING NEW when you deploy to LIVE.

TESTING is not a last minute options, it's a before you code to ensure you met requirements, and that you don't write code to suit the tests (1980s and 90s).

Provide meaningful error messages to your users, don't fail SILENT!

Lost photos and videos

A friend of mine was migrating his Emby system to a new all singing all dancing server that he built with Unraid, since that was what he was using originally.

During his migration to the new server and setting up the new disks with Emby, all of the photos and videos disappeared from one of the disks, leaving him with lots of photos and videos being lost.

Luckily before doing anything drastic he called me and asked if I'd come over to take a look.  So I did.

Conveniently he hadn't used his largest disk, meaning we could do a recovery, and being Linux based file systems meant this was easy to do, but would take a long while to perform the recovery.

Unraid was running from USB stick in memory.  This meant that for each recovery run the software would need to be installed again; minor issue.

The main element for helping out was to provide my friend with the knowledge to be able to do this himself.  So showing him how to;

1. Identify the Linux disk name from Unraid, where Unraid calls them Disk 1, Disk 2, etc, but clicking the disk provides the information for the /dev location - which I told him he needed to perform the recovery.

2. Install the software on Unraid, as he would need to do this for each recovery.

3. Perform the recovery.

4. Sift through 1000s of files as they may not have the correct name after recovery, especially if the directory information is unavailable.

Having found the relevant disk name for the data that had disappeared, I then stopped Unraid to free up the disks to mount using the normal Linux commands, and ensure no other processes were affecting the disks.

2 disks were required for the task.

• The disk with the missing data
• The disk that would be the recovery location

The recovery location disk was mounted to /mnt/recovery.

The application to perform the recovery is called testdisk which contains 2 methods of recovery. The more advanced testdisk command and the easier photorec command.  I opted for photorec since it did most of the grunt work for you, leaving you to supply the disk to recover and the location to recover too.  TestDisk requires more information and knowledge of the file systems you were recovering.

Having supplied photorec with the device path of the disk to recover (unmounted) and the location to recover to (mounted drive : /mnt/recovery), it went off and started the recover, and within seconds had discovered several hundred photos and videos, stating a recovery time of 24 hours, which changed as it worked through the disk.

After a day or 2 the task completed and my friend informed me that he had to sift through the 1000s of files, some of which he had no idea they were on there, and others being the files he needed.  This he was working through for quite a few days.  He has the other 4 disks to work through.

Kodi on Ubuntu 24.04

Recently I decided to move back to a slightly larger media entertainment system, having use the Raspberry Pi 4 for a couple of years with 12TB of external USB storage.

The new system from mini-itx.com, came in a tidy little chassis which housed the AMD Ryzen 7 processor, 16Gb RAM, 1 x 2GB NVMe, 1 x 4TB NVMe and 2 x 4TB SSD, and Intel AX210NGW wireless card which works perfectly with the AMD CPU.

Installing Ubuntu 24.04 is as always nice and straight forward, installing the bare minimum. Kodi is now part of the Ubuntu repo, so also straight forward.  The Ubuntu installation with Kodi supports a Microsoft MCE style I.R. remote reasonably well, unlike other systems.

Once the media libraries were in place I have a Python script that allows me to set or unset watched flag in bulk.

My favourite skin is the Aeon NOX: Silvo.  Once I'd got everything as required then came the serious element of watching.

Another feature was to make use of Kodi's UPNP feature to enable LG TV's media player and Roku media player to stream media from the Kodi system and provide the same convenient method of browsing the files, without having to deal with files missing because the library couldn't understand the file, which is the issue you get with systems such as Emby, Jellyfin and others.  So Kodi works better when sharing media to other devices, and easier to find what you want.

After a while it became apparent that Kodi wad temperamental in shutting down using the menu.  Searching the internet provided various solutions;

- polkit fixes due to permissions required to allow the user to shutdown the system without requiring sudo.

- kernel setting through Grub to deal with BIOS power drivers.

There were some others, but after trying these, nithing worked.  Even trying to use Mint, only made things worse as the MCE controller didn't work as well and it still didn't shutdown.

The solution to the shutdown problem, after reverting back to Ubuntu, turned out to be the most obvious, but the least obvious, since it was not posted anywhere.  The solution:

Use the default skin Kodi is installed with.  This fixed all the problems.

The next problem, was that of a disappearing and temperamental UPNP service.  This was due to their being configuration in multiple locations;

- upnpsettings.xml

But there are others.  Not in the database files.

I've yet to look for where else the UPNP service is configured, but a fresh .kodi directory and copying your original database directory back in and configuring UPNP service resolved the issue.

Finally, I like to see what's coming, and IMDB Trailers is a must.  This is a fix that is well documented on the internet, which requires you to set the cacerts as Kodi and its plugins tend to have the wrong certs.  The following command fixes this issue;

ln -s /etc/ssl/cert.pem /usr/share/kodi/system/certs/cacert.pem.

KODI watched Episodes

The issue

Although Kodi is one of the better media centres out there, and I've been using it for some while ever since mediaportal went down hill, there have been some annoying changes that the developers make in removing key features from the user interface that are essential to the way people watch stored programs, such as;

How do I remove all the ticks for a watched TV series or program?

Kodi when it was XBMC and with the Confluence skin used to have a very useful option that if you right clicked the TV Series folder, you could unwatch the entire series.  When rebadged as Kodi they decided that this feature wasn't something people used - typical of developers who don't use something themselves, or talk to their community enough.

Having dug through some Kodi docs, and helpful people on line that have tried to solve this scenario, and without having to go through each item one by one to untick them I'm on the quest to write a script, and possibly a plug-in that will solve this long lost useful item of the media centre (yes I'm British so I will spell centre in this way, for the rest of you translate to center, until the UK and USA become a continent now that Brexit is in full swing and Europe in continuing to do what they've always done).  Tell you what though it would be a more impressive tunnel :-)

Sorry, I digress on issues that are of no interest at this moment in time.

The Tools

To achieve the desired result we need to now dig in to the Kodi database which is stored in the users home directory that you run Kodi as;

$HOME/.kodi/userdata/Database

From here we will need to work with the MyVideo??.db file, in my case MyVideo99.db

These files are sqlite3 database files so the following will be of use;

• sqlitebrowser
• A graphical tool for easily viewing the schema
• sqlite3
• CLI tool for working with these database files
• Handy if you prefer to write Shell scripts to control the DB schema and data
• sqlite3 python library
• import sqlite3

The Table

The table required is called files.

Why you should NOT run Docker containers as root

Why a Docker container should never run as root!

Most organisations ensure that you need to have sudo access to run the Docker commands on your systems, but even if you're using Kubernetes or OpenShift, or if you have added users to the docker group you run the risk of being compromised.

As a simple demonstration try out the following steps that would expose your /etc/shadow file to a docker container ran by an ordinary user;

1. Make sure your ordinary user is added to the docker group:

       sudo usermod -G docker steve

2. If you are logged in as that user you will need to log out and back in for the group to take affect

3. Check the user can run the docker command:

       docker ps

    You should see the docker ps header and any running containers on that system

4. Now let's cat the /etc/shadow file using a container from Docker Hub:

       docker run -it --rm -v /etc:/hack python:3.12 cat /hack/shadow

5. You'll notice that what you will be looking at is your hosts shadow file and not the shadow file in the container since that is in /etc/shadow.

If you're not doing the following to secure your environment then you will be at risk of people being able to crack the root password on your systems.

1. Create and use only your own organisations private Docker registry

2. Block access to docker.io, public.ecr.aws, quay.io, and any other public repositories you know of.  Access to these should only be allowed by those who will be building the base container images for your organisations

3. Any base images downloaded from the registries in 2 should be built as new images that run with a specific user ID greater than 1000.

4. If developers need to install software onto containers then this should be performed in the developer environments only, or through CI pipelines which perform the root level installs and then add the user to the end of the build.

Root running containers should only be used where software needs to be installed that the basic user cannot, which is why it is important to have the necessary base images available for developers so that they only need worry about their code and libraries.

Disable DNF Dragora on Fedora

Synopsis

Being a person that likes to manage updates on my own schedule, rather than being reminded, and also not liking processes running that don't need to be, I searched around to find how to disable the DNF Dragora application.

This application is a Fedora GUI application and the background task is not a systemd task, but an autostart task.

Locations

Autostart directories are located in 1 of 2 places:

• /etc/xdg/autostart
• $HOME/.config/autostart

The first is system wide since it is under /etc, where as the other is personal to you and things that you have decided you want to be running once you've logged into the GUI.

You should always start by checking your home version first and renaming the file if you're not sure if you want it stopped, or deleting it if you want it permanently gone.  In most cases you can also deal with start up applications through gnome-tweaks.

Finding DNF Dragora

Having check my home directory $HOME/.config/autostart for dnfdragora as follows:

grep -ri drag ~/.config/autostart

I found no files containing the dnfdragora command.

Locating other autostart folders with:

sudo find / -name autostart

We find the /etc/xdg/autostart folder.

Using the following grep command we find the following file:

grep -ri drag /etc/xdg/autostart

/etc/xdg/autostart/org.mageia.dnfdragora-updater.desktop

Disabling DNF Dragora

To disable it (rather than removing it, if you change your mind) you simply do the following:

cd /etc/xdg/autostart

mv org.mageia.dnfdragora-updater.desktop org.mageia.dnfdragora-updater.desktop.old

Simply making sure that .desktop is not the last part of the file name will prevent the file being seen.

Log out and back in and DNF Dragora should no longer be there.

How to simply build a Jenkins server and Agent

Having been working on another DevOps Academy it was surprising with the research that the students did on how to build a Jenkins server with Agent, how complicated most people made it.

This example is based on using AWS with 2 EC2 instances, but would work on-prem and other clouds.

Steps to build

1. Create 2 Ubuntu instances both Medium

• 1 is Controller
• 1 is the Agent
• Settings
• t2.medium
• 20GB disk
• Ubuntu image
• Select or create a security group (e.g. jenkinssg) that has the following inbound ports
• 8080
• 22
• 8200

2. Install Jenkins on the controller

• ssh onto the Jenkins controller
• sudo apt update # It's Ubunutu after all
• sudo apt -y install openjdk-11-jdk # Install Java, Jenkins needs it
• wget -q -O - https://pkg.jenkins.io/debian/jenkins.io.key | sudo apt-key add -
• sudo sh -c 'echo deb http://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'
• sudo apt update
• sudo apt install jenkins
• sudo cat /var/lib/jenkins/secrets/initialAdminPassword   # Get the login password

3. Web browser

• Point your web browser at http://yourPublicIPforInstance:8080
• Paste in the text from the file
• Click Continue button
• Click Install suggested plugins
• Fill in the form to create the main Jenkins user
• Username: admin
• Password: secret123
• Confirm password: secret123
• Fullname: Administrator
• E-mail address: root@nowhere.com
• Click Save and Continue button
• Change the IP address to the Private IP address of your Jenkins instance
• Leave the http:// and the :8080/
• Click Save and Finish button
• Click Start using Jenkins button

4. Configuring Jenkins to see the new node to be the agent

• After doing step 3 you should be logged in as Administrator
• If not log in using admin and the password you set
• Click Manage Jenkins in the left menu
• Click Manage Nodes and Clients
• Click New Node in the new menu
• Set the Node name to Worker1
• Select Permanent Agent
• Click Create button
• New screen
• Number of executors: 2
• Remote root directory: /home/ubuntu
• Labels: all
• Leave all other as default
• Click Save
• The worker node will show as not connected
• Click on Worker1 link
• You'll notice an error message
• JNLP agent port is disabled and agents cannot connect this way. Go to security configuration screen and change it.
• Click the link Go to security configuration screen and change it.
• Scroll down to Agents
• Select Fixed
• Set the Port to 8200 (This is already allowed by jenkinssg)
• Scroll to the bottom and click the Save button
• Click Manage Nodes and Clients
• Click Worker1
• Right click the blue agent.jar link and Copy link address

5. Now ssh on to your Worker/Agent instance

• sudo apt update
• sudo apt -y install openjdk-11-jdk
• wget http://52.213.211.75:8080/jnlpJars/agent.jar
• Where the http:// link is pasted from the Copy link address
• On the Jenkins web page copy the 2nd box echo line
• Paste this line into the terminal of the Worker/Agent ssh session
• On the Jenkins web page copy the 2nd box java -jar line
• In the terminal of the Worker/Agent ssh session
• Type the word   nohup
• Then paste the java -jar line after this
• Type a space and then   &    at the end of the line
• e.g.
• nohup java -jar agent.jar -jnlpUrl http://172.31.16.36:8080/computer/Worker1/jenkins-agent.jnlp -secret @secret-file -workDir "/home/ubuntu" &

6. Back to the Jenkins web site

• Click Back to List if you are in the Agent Worker1 screen
• or
• Click Dashboards top left of the page
• Click Manage Jenkins
• Click Manage Nodes and Clouds
• Note your agent is now connected

NOTE:

This configuration doesn't create a service to start the agent on reboot, but is purely an example to get it running.  To make the agent a proper service you would need to create the appropriate file in /etc/systemd/system (call it jenkins-agent.service) or for older systems the service script in /etc/init.d.