:– title: “Bash Commands” output: html_document: toc: yes toc_float: true —

knitr::opts_chunk$set(eval=FALSE)

Introduction

This page is the continuation of my blog post on GNU-Linux bash shell commands.

Linux is the kernel of the operating system on top of which other programs are built. A detailed list of GNU core utilities is available under the command :

info coreutils 

The page below also contains many commands from external tools that are not part of the GNU project.

Find

Find files

Find files in subdirectories of the current directory (Quotes are requited to prevent shell command expansion).

find . -name "*.pdf" 

find . -mtime 0 # modified in the last 24 hours

Find files in the whole system

locate filename

Get information about a file

Determine file type and encoding. The -i option shows the mime type strings.

file filename
file -i filename

list a directory

ls
ls -R # list subdirectories recursively
ls -lh # sizes in human readable format

Display the last modification date of a file

date -r <filename>
ls -l <filename>

More file information

stat <filename>

Search and filter text files

Search with Grep

grep "text" file.txt

Recursive search (-r) in all files in the directory for a given pattern, case insensitive (-i)

grep -ri "pattern" .

Give 1 line of context before and 2 lines of context after:

grep -B 1 -A 2 pattern README.md

Search for lines containing “Done” or “encountered” only in specific log files recursively in a directory. Also displays the line count at the end.

grep -r Done --include runner.log | tee /dev/tty |wc -l
grep -r encountered --include runner.log | tee /dev/tty |wc -l

Note: How to pipe stdout while keeping it on screen

Show all csv files mentioned in a source code (supposing the csv file name contain only alphanumeric characters). Use Extended regular expression (-E) to search for alphanumeric characters, only show the matching result (-o), not the full line and show the line number (-n) where the string occurs:

grep -Eon "[[:alnum:]]+.csv" file.txt

Count occurrences of a word in a file

grep -roh word filename.txt  | wc -w

Awk tutorial, for example filter a large file for lines that have a third field (product code) starting with 44, keep the header line:

awk -F, '$3 ~ /^44/||NR==1' nc201501.dat|less

Regexp match beginning of and end of line with ^ and $.

Follow the end of a log file as it is written

tail -f

See tab and end of line characters in a text file

cat -te filename |less 

Count the number of lines in a text file

Count the number of lines in a file

wc -l filename.txt 

Count the number of lines in all files, and give the total line count.

find . -name '*.py' | xargs wc -l # python code
find . -name '*.R' | xargs wc -l # R code

Read

Play Midi files

Install a software synthesizer to play midi files. VideoLan Wiki

If the FluidSynth codec is not shown in VLC’s preferences, you have to install it as well as sound fonts. E.g. on Ubuntu 14.04 and derivatives it is in the vlc-plugin-fluidsynth package, while the fluid-soundfont-gs and fluid-soundfont-gm packages install some sound

Edit

Edit Image files

Resize a picture to 2000 px wide using ImageMagick:

convert picture1.jpg -resize 2000 picture2.jpg 

Batch processing, resize all jpg images and place the resized images in a new folder

mkdir newfolder
mogrify -resize 2000 -path ./newfolder *.jpg

Edit PDF files

Commands based on the poppler library for PDF manipulation. Search a text pattern in all PDF files present in a directory:

pdfgrep pattern *.pdf

Merge multiple PDF into one:

pdfunite in-1.pdf in-2.pdf in-n.pdf out.pdf

Alternatively, pdftk can be used to merge PDF files

pdftk input1.pdf input2.pdf cat output output.pdf

Extract pages from a pdf

pdftk full-pdf.pdf cat 12-15 output outfile_p12-15.pdf

Split a pdf in two

mutool poster -x 2 cbmflow.pdf output.pdf
# if necessary, install mutool with 
apt install mupdf-tools

Replace strings in text files

Replace strings

first="I love Suzy and Mary"
second="Sara"
first=${first/Suzy/$second}

Replace strings with sed

sed -i  's/pattern/replacement/g' bli.txt 

sed -i  's/^.*\://g' input_file.txt # edit file in place
grep EMAIL input_file.txt |sed  's/^.*\://g' > output_file.txt 

Add a line above and below a line that contains a particular string. This was necessary for a Latex to Lyx document conversion. The Latex file had problematic characters in href links. Conversion worked fine without the href links, to keep the links

I quoted them in \begin{verbatim} statements. sed add # before a line containing a pattern sed performing several substitutions

# Test output to stdout placing # before and after a matching line
sed -e /href/s/^/\\n#\\n/ -e /href/s/$/\\n#\\n/ clipboard_3_to_delete.tex| grep -C2 href 
 # Modify the file in place

Remove duplicated lines from a file

awk '!a[$0]++' input.txt 

Refactor

A one liner to replace strings with perl in a git repository. The examples below replace all occurrences of the first string with the second string, for all files under git version control.

git grep -lz 'first_string'| xargs -0 perl -i'' -pE "s/first_string/second_string/g"
git grep -lz 'readcsvfromgauss'| xargs -0 perl -i'' -pE "s/readcsvfromgauss/readcsvfromgauss0/g"
git grep -lz 'RMySQL::'| xargs -0 perl -i'' -pE "s/RMySQL::/RMariaDB::/g"

Alias to refactor:

refactor() {
    echo "Replacing $1 by $2 in all files in this git repository."
    git grep -lz $1| xargs -0 perl -i'' -pE "s/$1/$2/g"
}

Usage, for example replace ‘word’ with ‘sword’:

refactor word sword
git diff

Organise

Compress one file

Compress a file using gzip or 7z compression

gzip <filename>
7zr a -si<filename> <filename>.7z 

Decompress a single file from several archive formats:

bzip2 -d <filename.bz2>
gunzip <filename.gz>
7zr e <filename.7z> # extract all files in the same directory 
7zr x <filename.7z> # keep directory structure (see also multiple files below)

Convert a 7z file to gz using a pipe similar to this answer.

7zr e -so <filename.7z> | gzip -c > <filename.gz>

Compress multiple files

How do I compress a whole directory?

tar -zcvf archive-name.tar.gz directory-name 

Where

-z: Compress archive using gzip program
-c: Create archive
-v: Verbose i.e display progress while creating archive
-f: Archive File name

Decompress an archive containing multiple files Keep full path of files within the archive To extract content from the archive in the current directory

tar -zxvf archive-name.tar.gz 
7zr x <filename.7z>  

List content of an archive

7zr l <filename.7z>

Move and copy files

Copy a file

cp file1 file2

Recursive copy some files while keeping the directory structure

find . -name '*.csv' -exec cp --parents \{\} /target \;

Rename files

For example to rename all upper-case .JPG extension into lower-case .jpg extension.

rename 's/\.JPG$/\.jpg/' *.JPG

Convert

Pandoc

Pandoc demos contains a lot of example commands to convert text files from one format to another.

Convert an odf document to markdown

pandoc filename.odt -o filename.md 

Convert a Microsoft Word document to markdown

pandoc filename.docx -o filename.md 

Markdown has two different header styles Use ATX style headers in markdown output. The default is to use setext-style headers for levels 1-2, and then ATX headers.

pandoc --atx-headers filename.docx -o filename.md 

Download

Download with Curl

Load a file from a URL, using the file name from the URL

curl -O <url_including_a_file_name>

Download videos and audio

Install youtube-dl using pip:

 sudo pip install --upgrade youtube_dl

You may also need to install the ffmpeg package:

sudo apt-get install ffmpeg

Download a video from youtube :

youtube-dl video_url

Download only the audio in an OGG Vorbis format

youtube-dl --extract-audio --audio-format vorbis video_url

Download only the audio in an .mp3 format

youtube-dl --extract-audio --audio-format mp3 video_url

Create bash aliases to download various audio formats from youtube

alias youtubevorbis="youtube-dl --extract-audio --audio-format vorbis"
alias youtubemp3="youtube-dl --extract-audio --audio-format mp3"

Users

Manager users and groups

Check your user id

id 
echo $UID
echo $USER

What group do you belong to as a user

groups

Add a new user

useradd username

Set a password for the new user

passwd username 

Delete a user

userdel username

Temporary log in as a given user

su username

Add a user to the super users

adduser username sudo

That user needs to re-log into the shell for the change to take effect.

Show all users

getent passwd

Show all groups

getent group 

Change file permission

Change user and group of a file

chown groupname:username filename

Change user and group of all files in a folder, recursively

# In one command
sudo chown -R paul:paul scenarios
# In two commands
sudo chown -R paul scenarios
sudo chgrp -R paul scenarios

Change file permission

chmod a=rwx filename
chmod 777 filename # all permissions to every one

Change file permissions recursively:

chmod 755 directoryname # all permissions to the user, only read and execute for other users
sudo chmod -R 770 scenarios

nice illustration of read, write execute permission codes on Linux.

Chmod instructions can be given with characters or numbers, chmod 777 or chmod a=rwx is a question of preference.

Some prefer 755 over 777 because giving write access to group and other users could be a security risk. 755 leaves read and execute rights to groups and other users. 755 is visible as "rwxr-xr-x" in ls -l. 
The default for document files on Debian seems to be chmod 644, visible as "-rw-r--r--" in ls -l.

Alias

alias ll="ls -lh" 

Based on how can i sort du-h output by size

alias du='du -hd1 | sort -h -r'

You can place those commands in your ~/.bashrc to create a permanent alias. bashrc:

“You may want to put all your additions into a separate file like ~/.bash_aliases, instead of adding them here directly.”

Profile .bash_profile and .bashrc

These are places where a user can turn of the system BEEP :

setterm -blength 0

.bash_profile is executed on login shell, when you login in another tty or when you access a system through ssh. .bashrc is executed on non-login shells when you open a terminal window in Gnome.

Debian Dotfiles

"Now, since bash is being invoked as a login shell (with name "-bash", a special ancient hack), it reads /etc/profile first. Then it looks in your home directory for .bash_profile, and if it finds it, it reads that."

[...] "You may have noted that .bashrc is not being read in this situation. You should therefore always have command source ~/.bashrc at the end of your .bash_profile in order to force it to be read by a login shell.  "

In .bashrc a user can set environment variables, define alias (see above).

System

OS release

less /etc/os-release 

Disk usage

du -hd1  # sub-folder size at depth=1 
du -hs * # sub-folder size summary

Display available space on drives

df -h

Display available RAM memory

less /proc/meminfo

Install a program

sudo apt-get install 

System name

uname -a

file /sbin/init

hostname -f 

Start and quit a super user session

su
exit 

Last time the system was started

last reboot 
last

Show environment variables

printenv 
printenv LC_TIME

Partition disks

Format a USB drive as a FAT 32 partition.

sudo fdisk -l 
sudo umount /dev/mmcblk0p1 
sudo mkdosfs -F 32 -I /dev/mmcblk0

format a USB drive as a FAT 32 partition smaller than the drive size

sudo mkdosfs -I -F 32 /dev/mmcblk0 $((2*1024*1024))

I used the GUI tool Gparted. To see if a partition is primary, use the command: sudo parted /dev/mmcblk0 # then: print

This will print the type (primary or logical) of the partition.

Keyboard

bash french blogger recommended a simple shell command to change keyboard layout :

sudo loadkeys fr

fr-keyboard on Debian wiki for a more permanent system configuration and use in GUI apps. Switching between keyboads can then be done with:

setxkbmap de
setxkbmap fr

Information about the system Debian-Admin Finding out basic information about your Hardware:

cat /proc/meminfo
cat /proc/cpuinfo

Find your Debian version

cat /etc/debian_version
lsb_release -a

Shortcuts Keyboard shortcuts for bash for example :

  • Ctrl+A to go to the beginning of a line
  • Ctrl+E to go to the end of a line

Documentation

Creating and running bash scripts
Unofficial Ubuntu starter guide.

Locale

Debian wiki page on Locale settings

Display locale settings

locale

To install new locales, run :

sudo dpkg-reconfigure locales

SSH forwards locales to the server. The server will try to use the given locale, but it might not have them installed. Locale variables have no effect in remote shell

How do I add a locale to a Ubuntu Server? Check which locales are supported:

locale -a

add the locales you want (for example ru):

sudo locale-gen ru_RU
sudo locale-gen ru_RU.UTF-8

run this update command

sudo update-locale 

Partitioning and formatting drives

Manipulate the disk partition table See forum post July 21st, 2012, 09:08 AM

sudo fdisk /dev/sdb
d # delete
w # write table to disk
sudo fdisk /dev/sdb
n # create a new partition
# yes to all default choices
w # write table to disk

Create a MS-DOS file system for USB drives

mkfs.vfat -n 'usb_drive_name' -I /dev/sdb1

You can also use the GUI gparted.

Monitoring

Display Linux processes

top

Interactive process viewer

htop

Run a command every 2 seconds, for example to list a folder as new files are being written to it:

watch ls

Follow the end of a log file, as it is written

tail filename.log

Find out which program is listening to which port

sudo netstat -pna | grep <port_number>

Job handling

List

jobs

Bring a job to the foreground

fg job_number

Run a job in the background. A command followed by an & will run in the background.

Stop a job

CTRL ^ Z

Move a stopped job to the background

bg

How to manage foreground and background processes “By default, the bg command operates on the most recently stopped process. If you’ve stopped multiple “

Quit a job

 CTRL ^ C

Kill a malfunctioning program:

kill process_id
kill -9 process_it # -9 forces the kill

Find a program id with:

ps aux

Kill a graphical program, by clicking on it:

 xkill

Bash Programming

Variables

Variables

message = "Hello World"
echo $message

Wild cards

Search for a pattern in python and R files:

grep -n <pattern> *{py,R}

Authentification and security

Generate SSH keys and add them to the agent

Generate a new key

cd ~/.ssh
ssh-keygen <id_rsa_keyname>

Temporarily add a key to ssh

ssh-add ~/.ssh/<id_rsa_keyname>

List keys

ssh-add -l

After a restart of the ssh agent

killall ssh-agent
eval `ssh-agent`

This key will not be available anymore to permanently add a key to the ssh config file (SO) Edit your ~/.ssh/config file:

IdentityFile ~/.ssh/<id_rsa_keyname>

If the key was generated with a password,and you don’t exactly remember the password, you can test a few password you might remember with:

ssh-keygen -y -f <private_key_file_name>

Note if you copy keys from one machine to another, you might encounter this error: ssh “permissions are too open” error Change the permission so only the current user has access to the file:

chmod 600 id_rsa

Check a key’s fingerprint

On github, ssh key fingerprints are displayed in md5format. To display the fingerprint in md5format, enter:

ssh-keygen -l -E md5 -f ~/.ssh/id_rsa.pub

How to generate an SSH key fingerprint

Secure shell and secure copy

log into a remote machine

ssh user@remote_machine

Copy a local file to a file on the remote machine

scp local_file_name user@remote_machine:path_to_file/file_name

Copy a file from the remote machine to a local file

scp user@remote_machine:path_to_file/file_name  local_file_name

Copy a full directory (dmouraty) from the remote machine:

 scp -rp user@dest:/path destdirectory

GPG signing keys

Generate a pgp key

gpg --gen-key

To submit your key to a server, export the key

gpg -a --export name@emailserver.com > public_key_sender.asc 

My keys are visible on pgp.mit.edu search=rougieux

Networking

Show the IP address

ip addr show

Show the state of network interfaces

ip link

Check open ports on a system

nmap -Pn <hostname or IP address>

Terminal multiplexers

Superuser.com tmux or screen

Screen

I used screen to detach and reattach bash sessions on remote hosts. This enabled me to run long algorithms on the remote host. I could disconnect in the evening from the ssh session and reconnect the following morning to check if the algorithm had finished running. The advantage of screen is that it is old and available on many systems.

Tmux

Tmux is under more active development than Screen.

Sessions

To start and name a session:

tmux new -s session_name

To detach from a session:

tmux detach (CTRL + b + d)

To reattach to a session:

tmux attach -t session_name

To list current sessions:

tmux list-sessions

To switch to another session:

tmux switch -t session_name

To delete a session:

tmux kill-session -t targetSession

Windows (tabs)

Create a new windows

tmux new-window (CTRL + b + c)

Move to a window

tmux select-window -t :0-9 (CTRL + b + 0-9)

Rename the current window

tmux rename-window (CTRL + b + ,)

Panes (splits)

Following commands should be prefixed by CTRL + b. Vertical split

%

Horizontal split

"

Kill a pane

x

Display pane numbers

q

How to switch directly to a pane in tmux?:

“You can jump directly to a pane by typing pane’s index while it is showed by display panes command.”

Movements

To scroll inside a pane press

CTRL + b + Page up down

tmux configuration

This answer explains how to create a tmux source-file. The configuration file tells tmux to create windows and panes and to start programs in each one of them. I started to create a tmux source file to start my favorite programs: A particular setup I use forces me to start these GUI programs from a command line:

neww
split-window -v 'lyx'  
split-window -v 'jabref'  
split-window -h 'rstudio' 
new-window

Start tmux with this configuration file:

tmux new-session -s work "tmux source-file ~/.tmux/work"      

Make it a bash alias, by adding the following line to ~/.bashrc:

alias tmuxw='tmux new-session -s work "tmux source-file ~/.tmux/work"'

Reddit discussion on vim splits and tmux tabs

VPN

A VPN was useful when: * SMS interface didn’t work on orange.fr. I could access it through a VPN server based in Paris. * FTP download site based in Canada didn’t work from Italy. A colleague could access it through a VPN server based in Canada. * SSH access to remote servers didn’t work on company wifi. I could access the machines through the VPN. VPN access also removes localisation of site searches on google and google maps.

Example session,from a company wifi system which is probably blocking SSH traffic. The SSH connection breaks:

paul@debianpaul:~/repos$ ssh cbm1 
ssh_exchange_identification: Connection closed by remote host

I start the VPN:

paul@debianpaul:~/repos$ vpn connect
Connecting to France - Paris - 1...     100.0%                       
Connected to France - Paris - 1

I try to reconnect to that server through SSH:

paul@debianpaul:~/repos$ ssh cbm1 
The authenticity of host '[.......eu-west-1.compute.amazonaws.com] ([__IP__])' can't be established.
ECDSA key fingerprint is SHA256:PTwO1p7Q3XqPE4Gs+KH37XqH88ePDYh5bNqKXyVpGAE.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[......eu-west-1.compute.amazonaws.com],[....]' (ECDSA) to the list of known hosts.

Now the connection works.