Recursive Grep

In this bash tutorial we will check out recursive grep usage examples and how it can be extremely beneficial in some use cases.

Table of Contents

  • Basics of grep -R
    1. Excluding Directories
    2. How to remove “Permission denied” messages
    3. Ignoring case sensitivity
    4. You can combine grep with regex
  • Benefits of recursive grep

grep -R, Recursive Query Function

grep -R can be used for recursive text search across many files in a directory. It can be extremely useful and can be applied to many different application cases.

1. Excluding Directories

Sometimes specific folders can cloud the query results when using grep recursively. For example you might have specific folders full of the stuff you are searching for but you might find to look for your query everywhere else under the same structure.
A typical occurrence is when you search under the root folder and proc directory becomes a nuisance in the results. This folder includes process IDs and cpu and kernel specific information which is often irrelevant for search results. Here is an example for excluding proc directory when using grep recursively but you can use it for any directory you like.
grep -R --eclude-dir=/proc "import requests" /

grep recursively looks for “import requests” under / (root) while excluding /proc directory.

2. How to remove "Permission denied" messages

grep -R  "import" /

Sometimes when you might have to use grep without sudo. Or some system directories might be unreachable even for the admin. In those cases you might get results with hundreds of lines similar to below which will be quite hard to read.

Check out an excerpt of the results from the bash script above.

/usr/share/doc/libjpeg-turbo/TJExample.java:import java.io.*;
grep: /proc/net/ip_tables_names: Permission denied
grep: /proc/net/ip_tables_matches: Permission denied
grep: /proc/net/ip_tables_targets: Permission denied
grep: /proc/sys/fs/binfmt_misc/register: Permission denied
grep: /proc/sys/kernel/cad_pid: Permission denied
grep: /proc/sys/kernel/usermodehelper/bset: Permission denied
grep: /proc/sys/kernel/usermodehelper/inheritable: Permission denied
grep: /proc/sys/net/core/bpf_jit_harden: Permission denied
grep: /proc/sys/net/core/bpf_jit_kallsyms: Permission denied
grep: /proc/sys/net/core/bpf_jit_limit: Permission denied
grep: /proc/sys/net/ipv4/route/flush: Permission denied
grep: /proc/sys/net/ipv4/tcp_fastopen_key: Permission denied
grep: /proc/sys/net/ipv6/conf/all/stable_secret: Permission denied
grep: /proc/sys/net/ipv6/conf/default/stable_secret: Permission denied
grep: /proc/sys/net/ipv6/conf/lo/stable_secret: Permission denied
grep: /proc/sys/net/ipv6/conf/wlp58s0/stable_secret: Permission denied
grep: /proc/sys/net/ipv6/route/flush: Permission denied
grep: /proc/sys/vm/compact_memory: Permission denied
grep: /proc/sys/vm/drop_caches: Permission denied
grep: /proc/sys/vm/mmap_rnd_bits: Permission denied
grep: /proc/sys/vm/mmap_rnd_compat_bits: Permission denied
grep: /proc/sys/vm/stat_refresh: Permission denied
grep: /proc/tty/driver: Permission denied
grep: /proc/kmsg: Permission denied
grep: /proc/mtrr: Operation not permitted
grep: /proc/kcore: Permission deniedgrep: /proc/slabinfo: Permission denied
grep: /proc/kpagecount: Permission denied
grep: /proc/kpageflags: Permission denied
grep: /proc/timer_list: Permission denied
grep: /proc/kpagecgroup: Permission denied
grep: /proc/vmallocinfo: Permission denied
grep: /proc/pagetypeinfo: Permission denied
grep: /proc/sysrq-trigger: Permission denied
Permission denied output can be excluded by adding 2> /dev/null to the query. Here is an example. 
grep -R  "import requests" /home/usa 2> /dev/null
/usr/share/doc/libjpeg-turbo/TJExample.java:import java.io.*;

2. Ignoring case sensitivity

grep -R  --ignore-case "IMPORT" /

You can also disable case sensitivity when using grep as above.

3. You can combine grep with regex

Regex is beyond the scope of this tutorial but you can combine grep to create very specifically tailored queries. This can be useful in server administration, cloud migration, cloud management, devops operations and similar industry-level tasks.

grep -R  --ignore-case "IMPORT" /

Benefits of Recursive Grep

Using grep recursively can be incredibly useful. Here are a few examples.

  • You can find files without knowing the file names. Imagine you recovered some data but file names are a mess. Let’s say you have 100K files with weird names. You could use grep -R for recursion and find specific files with specific content. For example you can search for a query like import requests or another Python code and you will get Python. files.
  • You can find specific text strings across many text files. You can search for an Einstein quote or a price information without opening each text file.
  • Applying a similar logic you can filter html files or you can query images. Most images include EXIF information which can be used to find specific images as well.

grep -R will treat each file as strings and go through every single file in the specified folder and output query results.

References

[1] Grep Command: GNU Grep Manual

Launching desktop environments with startx

You can have total control over launching your operating system’s graphic server in Linux.

Most desktop environments use a graphic server called either X (or Xorg) or Wayland. X is approximately 40 years old and it is the legacy graphic server that serves its purpose really well. X still dominates the market.

Wayland on the other hand is a modern approach that’s been gaining traction and it also has lots of support and is being used by quite a few desktop environments in Linux such as KDE.

I find it very useful to be able to manually launch and quit desktop environments in Linux and shut down (or never launch) the graphic server when I want to.

This Linux tutorial  will explain how to take control over launching the Xorg graphic server in almost any Linux distro and the reasoning behind it.

Table of Contents

  • X Graphic Server Terminology
  • Launching Desktop Environments with startx
    1. Gnome .xinitrc
    2. XFCE .xinitrc
    3. Cinnamon .xinitrc
    4. KDE Plasma .xinitrc
    5. Qtile .xinitrc
  • Handling the Login Manager
  • Benefits of Manually Launching Graphic Environment in Linux
  • Troubleshooting startx & .xinitrc

X Graphic Server Terminology

Here is a breakdown of the most commonly used desktop environments and the codes you can use to start them with startx which will initiate X graphic server/client through Xinit (Server Launcher Program) and Xorg (Graphic Server).

Xorg is synonymous with X server or sometimes just X.

You can create .xinitrc under user’s home and add just a one line code such as:

Xinit : Program which launches Xorg graphic server.

Xorg : X Graphic Server

startx : Command used to launch graphic applications which defaults to .xinitrc file.

.xinitrc : File under the home directory that instructs startx command

startplasma-x11 start. : an example .xinitrc entry. See below for specific entries for each Desktop Environment or Window Manager.

Screenshot of Endeavour OS based on Arch Linux and KDE Plasma Desktop

1. Launching desktop environments with startx

Here is a breakdown of the most commonly used desktop environments and the codes you can use to start them with startx which will initiate X graphic server/client through Xinit (Server Launcher Program) and Xorg (Graphic Server).

Xorg is synonymous with X server or sometimes just X.

You can create .xinitrc under user’s home and add just a one line code such as:

startplasma-x11 start.

So the code goes inside .xinitrc, and the directory structure would be:

/home/userxyz/.xinitrc

Make sure .xinitrc file has a dot before it as shown. Xinit and Xorg directly looks for this file under home directory. (dot makes the file hidden which can be seen by ls -la command)

Here are the specific codes to include in the .xinitrc file for each desktop environment.

There are also a few points down below which might be helpful for troubleshooting or avoiding pitfalls if you are doing this kind of stuff for the first time. You can also read about the benefits of manually launching desktop environments or at least my reasoning for it. If you have comments feel free to drop it at the bottom.

Please review “Disabling the Login Manager” down below before starting to use the startx command.

Gnome .xinitrc

You can instruct startx either via exec in the beginning or start in the end.

gnome-session start

or

exec gnome-session

XFCE .xinitrc

startxfce4 start

or

exec startxfce4

Cinnamon .xinitrc

cinnamon-session start

or

exec cinnamon-session

KDE Plasma .xinitrx

startplasma-x11 start

or

exec startplasma-x11

Qtile .xinitrc

qtile start

or

exec qtile

Now, almost a year later, we have Google’s Imagen published which makes Dall-e 2 results look primitive. Imagen’s image outcomes are so impressive that, it looks beyond the artistic capabilities of human designers and illustrators. Of course art and design are subjective terms but every single Imagen image published is truly mind-blowingly accurate, well-designed and photorealistic.

Starting desktop environment with startx

2. Handling the Login Manager

If you are building a Linux system from the ground up then you might not have a Login Manager but if you are trying to modify an existing distro such as Endeavour OS, Mint OS, Manjaro OS, Fedora, Cent OS, Ubuntu, OpenSUSE etc. then you will want to disable to existing login manager.

Starting and Stopping the Login Manager

You can also immediately start and stop the SDDM Login Manager service using the commands below. (Stopping the login manager on the go might be tricky sometimes but disabling fixes that problem for the next boot.)

  • For starting SDDM:
    • systemctl start sddm
  • For stopping SDDM:
    • systemctl stop sddm

Enabling and Disabling the Login Manager

First you need to find out which Login Manager your system is using if there is one. Login Managers (or Display Managers or DMs) are graphical login screens that can be useful in some cases. I personally don’t find them to be very useful. They take resources and complicate things and running Desktop environments from tty is perfectly minimalistic and aesthetically pleasing anyway.

Here are some commands to list login managers, find them, enable/disable them and start/stop them…

You can list the services enabled at systemd level via systemctl by using either of the following commands.

First one is the native way of filtering enabled services with systemctl’s own flag while second method makes use of grep command for a similar output.

systemctl list-unit-files --state=enabled

systemctl list-unit-files | grep enabled

SDDM Login Manager

So disabling and enabling login manager at startup level is quite easy with systemd. Here are the example commands for SDDM login manager.

For disabling SDDM:

systemctl disable sddm

For enabling SDDM:

systemctl enable sddm

Make sure you are comfortable with working with the CLI (command line interface) before disabling the login manager first because you might not have the login manager which triggers the graphic interface of the desktop manager automatically behind the scenes.

Login Managers are quite persistent meaning even if you stop them, they might auto-launch and get in the way when you are trying to manually launch your graphic environment (through desktop environments, windows managers or sometimes specific individual apps.)

If you disable your login manager and reboot your Linux system, you will just get a TTY command line without graphics server activated. Normally you will get TTY1.

You can usually navigate between multiple virtual terminals using the Ctrl+Alt F2 (or F3 for TTY3, F1 for back to TTY1, F4 for TTY4 and so on.)

Light DM (LDM) Login Manager

You can enable / disable Light DM as below:

For disabling Light Display Manager:

systemctl disable lightdm

For enabling Light Display Manager:

systemctl enable lightdm

3. Exiting Xorg Graphic Server and Desktop Environments

After launching the desktop environment from the tty command line, you might want to go back to the tty environment and/or leave your computer on without the Xorg graphic server.

Termination of Xorg and desktop environments can be conveniently achieved when there is no Login Manager such as SDDM running. That’s because SDDM can be persistent and auto launch itself which can make it difficult to kill without disrupting the system.

To kill the desktop environment just execute the appropriate one for your setup from the following commands.

Commands to kill various Desktop Environments

So disabling and enabling login manager at startup level is quite easy with systemd. Here are the example commands for SDDM login manager.

For killing Gnome session:

killall gnome-session

For killing XFCE session:

killall startxfce4

For killing Cinnamon session:

killall cinnamon-session

For killing KDE Plasma session:

killall startplasma-x11

For killing Qtile session:

killall qtile

It’s also perfectly possible to map the commands above to a key combination to kill the desktop environment. One combination that works perfectly for me is Ctrl+Meta+Q. It’s usually a free combo and has the Q (quit) in it so it’s easy to remember. It’s also not easy to accidentally hit it.

Benefits of Manually Launching Graphic Environment in Linux

It’s starting to feel like we are there. AI’s real-world successes are being increasingly felt. Every new applied-AI milestone leaves your mouth open. It’s hard not to imagine the societal impacts. Millions of humans who derive lots of self-worth and satisfaction from their work suddenly becoming unemployed or even worse irrelevant.

  • Achieving focus: Being immersed in a graphic environment has its pros and cons. Having the graphic environment can be counterproductive when you need to focus on certain tasks. For some tasks that can be done in the command line, I find it helpful to not having a desktop environment launched at all. There is something special about not having the graphic server launched at all versus having the GUI programs available in the environment. That being said of course sometimes the graphic environment can be very useful and even inevitable. Some tasks I find myself performing through the CLI are:
    • Writing code
    • Writing scripts
    • Playing music (cmus)
    • Taking notes
    • Writing articles
    • Reading notes/books
  • Increased Ram & CPU resources: In most computers, the most resource hungry components are often the graphic environments. Graphic server communicates with each input and each output at pixel level which understandably creates a burden on RAM and CPU resources.
  • Increased battery life: Manually launching X graphic server when needed can optimize battery life enormously. I forgot my Dell XPS on for one week and the battery was still 90%+. Since there’s no graphic server launched the necessary processes to keep the OS running are at absolute minimal. X graphic server comes with many support libraries as well which can bloat the system.
  • Having more control: Being able to launch and terminate the graphic server and all the desktop environment and other related programs in a very practical way gives you more power over using your computer’s operating system and even its hardware. It’s also helpful to understand the inner workings of the operating system’s graphic components.

graphical.target vs multi-user.target

systemctl get-default

systemctl set-default graphical.target

systemctl set-default multi-user.target

Troubleshooting startx and .xinitrc

1- startx is not meant to be run as root

2- .xinitrc is not under home directory.

3- .xinitrc has incorrect entries.

start startplasma-x11 instead of startplasma-x11 start

entries after using exec

forgetting & character at the end of the line when there is multiple entries.

4- Having multiple .xinitrc in multiple locations which will confuse Xorg server or make it ignore intended .xinitrc.

5- User doesn’t have access to video group

6- Not having necessary Xorg packages installed.

7- Not having necessary graphic firmware installed.

References

[1] X Graphic Server Documentation: https://www.x.org/wiki/UserDocumentation/
[2] Arch Xorg Documentation: https://wiki.archlinux.org/title/xorg
[3] Debian’s Xinitrc Documentation: https://wiki.debian.org/Xinitrc

Installing AUR Packages with Pacman

We will compile and install software packages from source code.

Content Table:

  • AUR vs Arch Packages
  • Installing packages from source
    • Step 1: Cloning the source code
    • Step 2: Compiling the package
    • Step 3: Installing the package
  • Post Installation: Updating & removing external packages
  • Troubleshooting makepkg
Software packages can be comopiled from source code in Arch Linux.

AUR - Arch User Repository

AUR or Arch User Repository is a repo which holds PKGBUILDs or source code of various computer programs.

AUR can be used to find source code ready to be compile and make packages. You will realize while AUR lists these packages, source codes themselves are hosted on Github.

Let’s get to it.

AUR vs Arch Packages

87K packages can be found on AUR as of December 2022 while there are about 13K packages on Arch Packages.

Arch packages can be installed directly using the pacman package manager.

sudo pacman -S <packagename>

Installing packages through AUR requires a couple of extra steps.

In this mini-tutorial we will learn how to install packages directly from source code using 2 methods:

  • makepkg
  • pacman

1. Cloning the source code

git clone https://aur.archlinux.org/librewolf.git

Make sure git clone isn’t run as root, a common mistake…

git clone will create a folder with the necessary program files along with the PCKGBUILD (a script used to build packages). You can then navigate to this folder and continue with the instructions in the following steps from inside this folder. 
cd -newpackagefolder-

2. Compiling the package

We will use makepkg to compile source code. Makepkg is a convenient package build utility. You can read more about it here.
We can compile a package using makepkg -s.
-s flag: Stands for syncing and syncs the dependencies of the package being compiled so you don’t have to compile them one by one.
makepkg -s

You can also use the following command to continue with installation of the package built from source code.

makepkg -si

If you’ve used makepkg -si, you don’t need to follow the 3rd step to install the package using pacman.

3. Installing the package

I prefer this method because I am used to using pacman to install packages.

pacman -U <.zst file> 
pacman -U <.zst file>

Post Installation (Updating or Removing Packages)

You can still manage packages installed through compiling source code or other external sources to an extent.

You can’t update them automatically since they were compiled and they aren’t included in the pacman’s arch packages repo. But you can use pacman for related operations below:

  • remove manually installed packages
    • pacman -R <packagename>
  • get information about manually installed packages
    • pacman -Qi <packagename>

If you find out your package’s version is becoming outdated. You can update it using the same compiling and installation steps explained above and the package will be updated.

Troubleshooting makepkg

Here are a few of the most common makepkg errors and their solutions. It’s usually a simple fix to solve makepkg errors.

1- Pacman failed to install missing dependencies

When makepkg -s is used, dependencies will be installed given they exist in the pacman repository. Sometimes this is not the case and dependencies can’t be satisfied. In which case you’ll get an error similar to below:
==> ERROR: 'pacman' failed to install missing dependencies.
==> Missing dependencies: -> ocaml-ocamlsdl -> ocaml-graphics -> lablgtk2
==> ERROR: Could not resolve all dependencies.
In that case dependencies also need to be compiled manually. You can find them on AUR and repeat the steps above. Once you have the dependencies installed you can continue with makepkg -s to compile the main package you intended to install.

2- You do not have write permission for the directory $BUILDDIR

This is usually caused by wrong directory privileges and particularly package directory being owned by the root.

If you run sudo git clone you might get write error.

Solution is to just remove the directory and rerun git clone without sudo or change directory ownership to the user. Simple as that.

==> ERROR: You do not have write permission for the directory $BUILDDIR (/tmp/librewolf-bin).
Aborting...

If you run git clone as root you would be required to run makepkg as root as well. The problem is running makepkg as root is not allowed. You can see the specific explanation for that specific error in the following section.

3- Running makepkg as root is not allowed as it can cause permanent damage

Makepkg should never be run as root and the system won’t allow you to do so. If you attempt running makepkg as root you will get the following error.

==> ERROR: Running makepkg as root is not allowed as it can cause permanent, catastrophic damage to your system.

The rationale behind this restriction is that if you compile a source code acquired from external resources as root then you would completely open the computer and the linux os to intentional and unintentional risks as these packages are created  by users and added to the AUR (or another repo or Github) repo.

Resources

[1] Pacman Package Manager Documentation: https://wiki.archlinux.org/title/pacman

[2] AUR (Arch User Repository): https://aur.archlinux.org/packages

[3] Makepkg package build utility documentation: https://wiki.archlinux.org/title/makepkg