Category: Linux

Posted on

Moving a Linux install to a new disk

Recently I had to move my Linux install from one drive to another, as I was experiencing some issues with a WD SN550 nvme drive causing some short random freezes of the GUI with IO intensive tasks. Since I also have a Samsung nvme drive installed, I decided to see if the problem persists on the other drive.

But… Having a fully configured and customized Linux install is a pain in the behind to redo. I did not want to clone, because I made a mistake during install the previous time and it installed in legacy MBR mode, so I wanted to do a proper install using UEFI mode. But preferably not having to re-do all the setup and customizations.

And I didn’t have to. Apt to the rescue.

$ apt-mark showauto > pkgs_auto.lst
$ apt-mark showmanual > pkgs_manual.lst

This will generate a list of .deb packages installed on the system. The first one with all the automatically installed packages, the second one all the manually installed packages from the commandline.

I also made a backup of /etc/apt/sources.list.d and /etc/apt/trusted.gpg.d. The first directory contains all the repositories I had in use on the original install, and the second directory holds all the GPG keys that go with these repositories. Important!

First install the system on the new drive, and make sure all updates are installed. You don’t need to install or setup anything but the base system. Now you can continue with the back ups and files you created earlier.

After I moved the two directories above to their respective place on the new install, and of course doing an sudo apt update && sudo apt upgrade to make sure all packages are still up to date, I loaded up the list of packages I created earlier.

$ sudo apt install $(cat pkgs_auto.lst)
$ sudo apt install $(cat pkgs_manual.lst)

This will create some errors due to packages that cannot be installed like this, or packages that were installed from a .deb file and aren’t located in any repository. Clean up those entries, and try again and let it run.

When it is finished, copy your /home/<user> from your old drive to your new drive and when you reboot and log back in as your user, everything should be just as it was before.

Success!

Posted on

Linux and a little rant

Allow me to start with the rant bit first. I took my first steps in the world of Linux around 1993, 1994. Back then, Linux was still pretty new and very few distributions existed. Debian just spawned into life. RedHat wouldn’t come to life until mid 1995. There were no big Internet forums or user communities. Most people using Linux were computer engineers that knew how to build a Linux system from scratch and didn’t take newbie kid questions all too well. N00b questions were typically quickly and swiftly dealt with by a grump “RTFM” and you were left to do a lot of reading and figuring it out yourself. None to hold your hand and do the hard work for you. Annoying at times, but when you did figure it out it gave you a real sense of accomplishment. This is how I learned my way around Linux.

Fast forward to today, and you have hundreds of distributions. Some are touted as ‘beginner distros’, others as ‘advanced distros’. Let me shoot that down immediately. There are only user-friendly distros and user-unfriendly distros. Some extremely user-unfriendly. Functionally, they can all to the same. There is not some magical advanced functionality in these so-called ‘advanced distros’ not available in the ‘beginner distros’. The only difference between the two is that the ‘beginner’ distros are as easy to initially set up and run as your typical Windows or MacOS install and don’t offer too many customization choices to confuse you and provide you with a default set of applications you may or may not use, to get you started, while the ‘advanced’ distros pretty much require you to be a masochist and think and decide about every step. Once they are up and running, they work pretty much the same and can do the same. There is literally nothing one of these ‘advanced’ distros can do you cannot also do on one of these ‘beginner’ distros.

This gets me to the users… Some of these ‘advanced’ distro users feel they are sooo smart. They consider everybody that uses a ‘beginner’ distro to be a n00b that should be pitied as they cannot begin to grasp the advancedness of their knowledge and their choice of distribution. This holds particularly true to a significant portion of the Arch Linux community. They think they are so smart for using a distribution that only installs a basic system and dumps you to the command-line so you can use their package manager to install the stuff you actually want on your system. Don’t get me wrong, it is a very valid philosophy that avoids cluttering up your disk with GB’s of stuff you’re never going to be looking at. But it is not advanced.

Back in the day, you had to download the sources for just about everything because if you wanted something that wasn’t installed by default, it probably didn’t have a binary available for your system. You’d have go through the documentation, find and fix all the dependencies yourself (often also by compiling and installing the right versions, in order) and then compile your application from source, hoping it actually would compile without errors you’d then have to debug and fix before trying again. And when it was finally installed, it probably didn’t work until you build your configuration files manually.

Telling a package manager which software you want to install only for the package manager to download it for you from a central repository, fix any dependencies automatically and install your selected package so it works is not anything advanced. It’s just manual work. It’s typing dumb commands on the command prompt. It does not make you some Linux Guru. It doesn’t even give you any usable extra knowledge. If you enjoy doing things that way, all the more power to you. But don’t be some cocky arrogant SOB that belittles others for not wanting to do that. It doesn’t make your install better, just leaner, it doesn’t make you smarter and behaving like that only makes you a prick.

There are a lot of people that have heard enough about Linux to be curious and wanting to try it, that are permanently put-off by the arrogance and belittlement of the Linux community. People should try to remember they too had to learn at some point and realize there are no stupid questions, only stupid answers. More competition means more and better choices for us as users, but for that to happen new users that are willing to learn something new should be encouraged, not belittled for being new and put down until they give up to never return.

If you want to try Linux, just do it. Pick a distro you like and stick with it. Don’t be fooled into believing the nonsense about beginner and advanced distributions, thinking at some point you need to upgrade to get a more advanced version. They can all be customized to look like whatever you want, they can all have the same functionality, there is not a single one that is better than the rest. There is only personal preference of the users using them. That’s not to say it cannot be fun to do some distro hopping to find the one that really suits you. Just as long as you remember its preference, not functionality that makes the difference.

Posted on

Monitoring your network or homelab using Zabbix and Grafana

For the longest time, I have been monitoring my network and homelab using Observium. This worked and does work very well. Observium is very good at what it does. However, there are a couple of things that do not work so well for me using Observium:

  • Observium does not let me add applications to monitor very easily or at all
  • Observium is limited to what can be offered through SNMP
  • Observium is not open source and as such it cannot be modified or changed to your needs
  • Observium is not an application I have come across in my professional life, so knowing how it works does not help me professionally.

That last bit is obviously not a necessity, however I do feel it’s always a nice thing to be able to apply things you have learned in your homelab into your professional environment.

After lots of investigations and trial & error, I have settled on using Zabbix for my monitoring needs. Zabbix is open source product that is used a lot in corporate environments and it is very flexible and extensible. Obviously you could add or change code as it is open source, but you don’t really need to. As Zabbix is template driven, its functionality can be extended by adding templates and there is a plethora of templates available for Zabbix, both on the Zabbix site itself (Zabbix Share) as well as places like GitHub. Also, Zabbix can use an agent installed on the system to collect the data you want to monitor, or you can use SNMP if you can’t or don’t want to install an agent on a device (for instance a network router).

The one thing I do not like about Zabbix is that the historic view is not easy to get to, nor as pretty displayed in a dashboard-like view as it is in Observium. However, that is not a blocker as we can use Grafana, another open source tool that is used quite a bit in Corporate Land to create dashboards and display relevant historic data.

Installing Zabbix

Installing Zabbix is no more complicated than installing most other software on Linux. I installed Zabbix on my Raspberry Pi 2B and the short version is this:

a. Install the Zabbix repository:
# wget https://repo.zabbix.com/zabbix/5.4/raspbian/pool/main/z/zabbix-release/zabbix-release_5.4-1+debian10_all.deb
# dpkg -i zabbix-release_5.4-1+debian10_all.deb
# apt update

b. Install the Zabbix server, frontend and agent on the server machine
# apt install zabbix-server-mysql zabbix-frontend-php zabbix-apache-conf zabbix-sql-scripts zabbix-agent

c. Create the database for Zabbix
# mysql -uroot -p
password
mysql> create database zabbix character set utf8 collate utf8_bin;
mysql> create user zabbix@localhost identified by 'password';
mysql> grant all privileges on zabbix.* to zabbix@localhost;
mysql> quit;
And import the database schema
# zcat /usr/share/doc/zabbix-sql-scripts/mysql/create.sql.gz | mysql -uzabbix -p zabbix
If your database is not residing on your Zabbix server, you can add -h <server> to the above to make sure you are connecting to the remote database server. Also, when creating the user and you are using a remote database, make sure the Zabbix user is allowed to connect over the network to the database.

d. Configure the Zabbix server to use the database you have created by filling out the relevant fields in /etc/zabbix/zabbix_server.conf

e. Start your brand-new Zabbix server:
# systemctl restart zabbix-server zabbix-agent apache2
# systemctl enable zabbix-server zabbix-agent apache2

f. You are done. You can now login to your server on http://<server>:3000 and log in with user Admin and password zabbix and configure everything else using the web frontend.

Installing Grafana

Installing Grafana is equally simple.
a. Install the pre-requisites and add the repository key
sudo apt-get install -y apt-transport-https
sudo apt-get install -y software-properties-common wget
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -

b. Add the repository
echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

c. Install Grafana
apt update
apt install grafana

d. Then it is just a matter of starting the Grafana server and making sure it starts at boot.
systemctl daemon-reload
systemctl start grafana-server
systemctl status grafana-server
systemctl enable grafan-server

That’s it! You can now log in using the default username/password of admin/admin.

Note: if you want to apply specific configurations, like for instance database (mysql, postgres, sqlite3) beyond the default, you should refer the Grafana manuals as that would be a bit beyond the scope of this page.

Tying things together…

First you will need to install the Zabbix app into Grafana. We can do this using the commandline interface for Grafana:

grafana-cli plugins install alexanderzobnin-zabbix-app

After you do this, you can configure the Zabbix datasource. Go to Configuration -> Data sources and click “add data source”. Scroll down to the bottom, and you will see one called ‘Zabbix’.

Fill in the details of your Zabbix installation, and make sure you add the api_jsonrpc.php to the end of your URL. Check ‘With credentials’ under auth and under Zabbix API details add your username and password. Click “Save & test” and if all is ok, it will give you a green checkmark while saying data source updated.

You are now ready to add a dashboard to your Grafana and start monitoring your Zabbix data. I use this dashboard from Paulo Paim. You can add it by going to Dashboards -> Manage and then clicking the import button. In the box saying import from grafana.com, type the ID of the dashboard. In this case 5363 and click load.

That’s it!

Links:
Zabbix manual
Grafana documentation
Zabbix templates and add-ons
Grafana dashboards and plugins

Proudly powered by WordPress