The Bourne Again SHell, or BASH for short, is a powerful language that can be used for low-level linux system administration operations. While modern systems engineers have a strong liking to more robust solutions like Redhat’s Ansible, for consumer facing projects I find that readability is key. Now, sure ansible is readable, but it’s unapproachable. Which is why this guide is going to get you up to speed with everything you need to know about the unix shell.

File Structure

Directories & Navigation

Introduction

When you login to a machine via ssh, you’re essentially telling it “Hi I’m bpetch and I want to access my shell.” When doing this, you’ll get into the machine and if you’re not familiar with Linux, you’ll wonder “what did I just get myself into.” Once you’re logged in, you’ll likely see nothing but a disclaimer and bpetch@host ~ $. This means that you’re in the shell and in your HOME directory. The HOME directory is where your files should live on the drive.

Essentially, if you were to login and run pwd it will likely return something like /home/username — this is the absolute path of where you are currently, the shell is shortening this to ~, and you can also reference it as $HOME.

Navigation

To navigate through directories, there is a command called cd. Essentially, if your user has permission to go to the location, it will put your current shell session into that location. An example might look like cd /home/bpetch which would take me to my home directory. You could also do cd $HOME and it would do the same thing if the environment variable is set.

Writing Files

In addition to cd, there is an application called touch and another called echo. Each of these has a purpose. If you touch a file, it will create a new file on the disk. So lets say I’m at $HOME and wrote touch newfile, it will create a new file called new file. This will be empty. Lets say that you wanted to add to the file, you can use echo to send text directly into the file. echo "hello world" >> newfile will put the text “hello world” into the file.

Editing Files

There are also a number of editors. For someone new to the shell, I’d almost always recommend learning nano. It’s a very simple text editor with some pretty good features. nano allows you to use arrow keys, type normally, and quickly match words by doing ctl-w. In addition, the save feature, accessed by doing ctl-x, pressing y, then entering the filename will allow for easy modification of files. You can also create new directories by typing mkdir foldername and it will create a new folder called foldername. If you are needing to create a list of directories, you can do mkdir -p folders/i/need and it will create folders/i/need on the filesystem.

Removing Files

Removing files is quite similar to touch, essentially if you’ve created a directory, you’ll need to use rm -r foldername this will remove the folder and anything under it. If you’re removing a single file, you can run rm filename and it will delete the file. Note that there is no recycling bin like you might expect on a macOS or Windows system. If you delete it, it’s gone.

In addition, there is the -f flag, which means that it will forcefully remove files. Never run this on / or any other high-level directory. It can and will blow up your files.

Installing Applications

Lets say that you require a few applications to get your software stack up and running– maybe it’s something like curl, or wget, tools to download files from the internet– Or maybe it’s a webserver like nginx… If you’re using Debian or Ubuntu, the apt package manager is going to be your best friend. (Please, do not use snap).

The APT package manager is extremely robust. It pulls from repositories to install applications that are supported by the distribution that you are using. Do note that adding repositories can create unintended damage to systems if you don’t fully understand what’s going on. We’re only going to use things included in the default repositories for today.

So lets install curl— First we’re going to need to update the package cache, do do this, we’re going to run the following command, followed by the second one to actually complete the installation.

sudo apt-get update 
sudo apt-get install curl -y

Congratulations! You’ve installed your first package. You can now download files off the internet using curl!

An example of a curl command would look like this:

curl -L "https://google.com" -o "$HOME/google.html"

You should now have a file called google.html located at $HOME.

Permissions & File Ownership

Introduction

These are really something that you don’t want to mess up — trust me, I’ve been there. For the most part, filesystem permissions are not something that need to be fiddled with, although if you do, chmod 777 is not the solution.

Ownership

Lets first understand what a user and a group is. Essentially your username, take bpetch for example, is created. This user, as the first user on the system will take on what is called a user id, which in my case is 1000, along with a group that belongs to that user, which also has the same id 1000. User and Group IDs can differ, and a user can be a member of multiple groups.

So lets say that a file is owned by root, for nginx (a common webserver) as an example. The root user is essentially a superuser on the system. It can access the entire filesystem and make modifications to almost any part of it — ok but why does this matter? The root user can also grant other users access to specific resources via file ownership. File ownership means that the user and group ids we talked about earlier “own” the file, meaning that it gets full access.

So inside a folder owned by bpetch, you’d expect that if root was to write a file to that location it would be owned by bpetch right? If you thought so, you’re mistaken. The user will be owned by root, and will not be writable by the bpetch user (although with the default 644 permissions on the file, it will be readable).

Octal Permissions

Octal permissions work on a range of numbers from 0-7. There are often 3 digits to this value. The first being user permissions, the second being group permissions, and the last being other permissions (someone in neither the user or group). You can gain a bit of an understanding about how the permissions are allocated here: https://chmod-calculator.com/ — Essentially, a value of 4 allows the digit to read a file, a digit of 2 allows for writing of a file, and a digit of 1 allows for execution of a file.

Putting it all together

The chmod unix command allows the permissions to be set on a file using both octal and +rwx -rwx notations.

So lets say that both bpetch and jerry want to access the same file. You can’t give both ownership (with the exemption of using a more fancy acl). What we do instead is create a group for both users. The -R flag recursively owns files and recursively modifies permissions.

groupadd storage
mkdir -p /mnt/storage
chown bpetch:storage /mnt/storage
chmod 770 /mnt/storage

# adds the user bpetch and jerry to the storage group
usermod -a -G storage bpetch 
usermod -a -G storage jerry

Now, both users can read and write to storage.

Testing & Tuning Network Throughput

Introduction

So… you’ve been given a machine and want to see how the speeds are, just head to speedtest.net right? Nope. Think again. While speedtest-cli does exist and is free, the better, less proprietary solution that actually allows you to measure speed from one host to another is called iperf3. iperf3 allows both a server and a client to be run using the command and will report back the true speeds during the test. You can find some public iperf3 servers here. If you’re running a Debian based OS like Ubuntu, or Debian itself, you can run sudo apt-get update && sudo apt-get install iperf3 to get the application installed.

TEsting

So what are the usages? The following command will try to connect you to that server and will run the test. You can also supply -P and a number (lets say 4, -P4) and it will spin up that many threads (simultaneous connections) to the test server.

iperf3 -c someserver.com -P4

Tuning Network Throughput

If you aren’t finding satisfactory results, but your line speed is guaranteed, you could try what is called “host tuning” where you modify the networking stack at the kernel level. While everyone kind-of reaches their own conclusions on the best parameters, I’d be sure to checkout https://fasterdata.es.net/host-tuning/linux/ — they’ve got a number of amazing resources that will help you get the most out of your connection. For just about anything I’d recommend looking at their config for

“For a host with a 10G NIC optimized for network paths up to 200ms RTT, and for friendliness to single and parallel stream tools, or a 40G NIC up on paths up to 50ms RTT:”

https://fasterdata.es.net/host-tuning/linux/

# allow testing with buffers up to 128MB
net.core.rmem_max = 134217728 
net.core.wmem_max = 134217728 
# increase Linux autotuning TCP buffer limit to 64MB
net.ipv4.tcp_rmem = 4096 87380 67108864
net.ipv4.tcp_wmem = 4096 65536 67108864
# recommended default congestion control is bbr 
net.ipv4.tcp_congestion_control = bbr
# recommended for hosts with jumbo frames enabled
net.ipv4.tcp_mtu_probing = 1
# recommended to enable 'fair queueing'
net.core.default_qdisc = fq

These settings usually do more than enough to get you up and moving, although if you’re still not satisfied, maybe checkout the following config: https://klaver.it/linux/sysctl.conf. And also do some reading at https://wiki.archlinux.org/title/improving_performance and https://wiki.archlinux.org/title/Sysctl#Improving_performance

Stay tuned for a Docker Tutorial.