knitr::opts_chunk$set(eval=FALSE)

Introduction

This page describes tools available at the bash command line, starting with the GNU core utilities. It also contains many commands from external tools that are not part of the GNU project.

Reference documents

Linux is the kernel of the operating system on top of which other programs operate. List of reference documents describing tools used at the bash command line:

  • A detailed list of GNU core utilities is available under the command info coreutils, also visible on the web at GNU Core Utilities manual.

This page is the continuation of my blog post on GNU-Linux bash shell commands.

Command history

To search within previous commands, press CTRL+R and start typing words contained in the previous command.

If you accidentally press CTRL+S the terminal will hang Press CTRL+Q to regain control. See also Ctrl-s hang terminal emulator? which provides a workaround: entering stty -ixon in .bashrc.

Convert

This section is about converting from one document format into another. For example converting a text file into a pdf file and vice-versa.

Audio

Convert from Wav to mp3 seen in this gist

for i in *.WAV; do test -f "${i%.WAV}.mp3" || lame -b 320 -h "${i}" "${i%.WAV}.mp3"; done

  • Convert audio from ogg to mp3

    ffmpeg -i “file name with spaces”.{ogg,mp3} # Convert all files from ogg to mp3 for i in *.ogg; do ffmpeg -i “\({i%.ogg}".{ogg,mp3}; done for i in *.ogg; do echo "\){i%.ogg}”; done

Cut an audio file at the given start time in seconds

ffmpeg -ss 132 -i input.mp3 output.mp3
ffmpeg -ss 621 -i 'Sharon Shannon Band-Tnbqwx3KRXU.ogg' 'Sharon Shannon Band.ogg'

ffmpeg also has a -to argument to “Stop writing the output or reading the input at position.”

Clipboard

Pipe to from the clipboard in bash script

For example copy the readme file to the clipboard used by CTRL+C CTRL+V:

xclip -selection c README.md

Pdf to images

How to convert pdf to images

cd ~/downloads
convert -density 150 input.pdf -quality 90 output.png

Images to pdf

Convert one or more images to pdf

img2pdf img.jpg -o output.pdf

Convert a directory of JPG images into a printable A4 pdf:

img2pdf --output out.pdf --pagesize A4 --border 0cm:0cm *.JPG

Place the images in a landscape A5 document for printing 2 pages on one page. The ^T argument is to flip to landscape:

img2pdf --output out.pdf --pagesize A5^T --border 0cm:0cm *.jpg

The man page of img2pdf explain the fit option with the following example:

“On each A4 page, fit images into a 10 cm times 15 cm rectangle but keep the original image size if the image is smaller than that.”

img2pdf --output out.pdf -S A4 --imgsize 10cmx15cm --fit shrink *.jpg
# I have tried
img2pdf --output out.pdf --pagesize A4 --border 0cm:0cm --imgsize 10cmx15cm --fit shrink *.jpg

The help page man img3pd explains the arguments

Convert a directory of JPEG images into a PDF with printable A4 pages in landscape mode. On each page, the photo takes the maximum amount of space while preserving its aspect ratio and a print border of 2 cm on the top and bottom and 2.5 cm on the left and right hand side.

img2pdf --output out.pdf --pagesize A4^T --border 2cm:2.5cm *.jpg

Appending ^T (a caret/circumflex followed by the letter T) turns the paper size from portrait into land‐ scape.

I used the following instruction to create an A4 page with 2 images one jpg and one png

img2pdf image1.jpg image2.png -o output.pdf --pagesize A4^T --border 1cm
img2pdf image1.jpg image2.png -o output.pdf --pagesize A4 --border 1cm
img2pdf *.jpg -o output.pdf --pagesize A4 --border 1cm

Interestingly when I asked for landscape mode, only the second image appeared in landscape. When I asked for portrait mode, only the second image appeared in portrait mode. The first jpg image contained exif orientation metadata (it was rotated vertically). Opening the first image in Gimp, rotating it and exporting it under a new name (without the exif metadata) solves the issue. Both pages are portrait A4.

Convert all jpg images in the current directory to a pdf

img2pdf $(find . -iname '*.jpg' | sort -V) -o ./document.pdf

If there is an alpha layer in the image, it will not be added to the pdf. The Warning message suggests using imagemagick to remove the alpha layer.

WARNING:root:Image contains transparency which cannot be retained in PDF.
WARNING:root:img2pdf will not perform a lossy operation.
WARNING:root:You can remove the alpha channel using imagemagick:
WARNING:root:  $ convert input.png -background white -alpha remove -alpha off output.png
ERROR:root:error: Refusing to work on images with alpha channel

Images from a zip folder to a pdf document

Unzip the archive containing jpeg images into a temporary folder, reduce the image sizes to 2000 pixel wide and generate a pdf.

mkdir -p /tmp/bli/newfolder
cd /tmp/bli/
unzip ~/downloads/fwdbungenfrfrderunterrichtfrdienchstenstunden.zip
mogrify -resize 2000 -path ./newfolder *.jpeg
cd newfolder/
img2pdf *.jpeg -o output.pdf --pagesize A4 --border 1cm
echo "The pdf document containing all images is located at: /tmp/bli/output.pdf"

Images to other formats

See also the resize section.

Convert a png image to a jpgeg image

convert canto_dei_sanfedisti.png canto_dei_sanfedisti.jpg

Convert a png image with an alpha layer to an image without alpha layer

convert input.png -background white -alpha remove -alpha off output.png

Convert from png to jpg and change size

mogrify -resize 2000 -path ./newfolder -format jpg *.png

Only change format

mogrify  -path ./newfolder -format jpg *.png

See also more image modification commands in the edit image files section below.

Pandoc convert documents between various formats

See also the markdown page. It contains more examples and formatting instructions such as how to use a bibliography, change the font size or use a two column layout.

Convert a markdown document to pdf

pandoc filname.md -o filename.pdf

Convert a markdown document to pptx

pandoc filname.md -o filename.pptx

Pdf to text with pdfminer

(pdfminer](https://github.com/pdfminer/pdfminer.six) Convert all pdf documents to text files in the current folder:

for pdffile in *.pdf; do
txtfile=${pdffile%.pdf}.txt
echo "converting $pdffile to $txtfile"
pdf2txt.py $pdffile > $txtfile
done

See also a utility to extract tables from pdf documents.

Pdf to tables with tabula

Extract tables from a pdf with tabula

cd /home/paul/Downloads/tabula-jar-1.2.1/tabula
java -Dfile.encoding=utf-8 -Xms256M -Xmx1024M -jar tabula.jar

Then visit http://127.0.0.1:8080/

Powerpoint to Markdown

Using the python package https://github.com/ssine/pptx2md

pptx2md file_name.pptx

To install the package the first time:

pip install pptx2md

ASCII art

Convert text to large ascii art

figlet

Count

Number of files

Count the number of files in the current folder

ls |wc -l

Recursively count the number of files in a directory, limiting the maximum depth

find -maxdepth 1 -type f | wc -l

Counting files with a specific extension

find -name '*.csv' |wc -l
find -name '*.ogg' |wc -l

Recursively counting files

find DIR_NAME -type f | wc -l

Number of lines in a text file

Count the number of lines in a file

wc -l filename.txt

Count the number of lines in all files, and give the total line count.

find . -name '*.py' | xargs wc -l # python code
find . -name '*.R' | xargs wc -l # R code

Documentation

Read the documentation of a bash tool

man ls

Open man pages in vim

man ls | vi -

DOS

Here are some DOS commands because sometimes you need to work on windows.

Remove directories recursively

rmdir /S

Show environmental variables

echo %Path%

Dot files

Alias

alias ll="ls -lh"

Based on how can i sort du-h output by size

alias du='du -hd1 | sort -h -r'

You can place those commands in your ~/.bashrc to create a permanent alias. bashrc:

“You may want to put all your additions into a separate file like ~/.bash_aliases, instead of adding them here directly.”

Alias from other persons:

https://github.com/statox/dotfiles/blob/master/bash_aliases

Reload bash aliases:

source ~/.bashrc

.bash_profile, .bashrc and .bash_aliases

  • I have chosen to define all aliases and environment variables in .bash_aliases

  • I keep .bashrc for other stuff

.bash_profile is executed on login shell, when you login in another tty or when you access a system through ssh. .bashrc is executed on non-login shells when you open a terminal window in Gnome.

My .bash_profile loads .bash_aliases as follows:

if [ -f ~/.bash_aliases ]; then
    . ~/.bash_aliases
fi

Debian Dotfiles

"Now, since bash is being invoked as a login shell (with name "-bash", a special ancient hack), it reads /etc/profile first. Then it looks in your home directory for .bash_profile, and if it finds it, it reads that."

[...] "You may have noted that .bashrc is not being read in this situation. You should therefore always have command source ~/.bashrc at the end of your .bash_profile in order to force it to be read by a login shell.  "

In .bashrc a user can set environment variables, define alias (see above).

Turn off system beep

Enter the following in .bash_profile or .bashrc to turn of the system BEEP :

setterm -blength 0

Download

Download with Curl

Curl man page:

“curl is a tool to transfer data from or to a server, using one of the supported protocols (DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, TELNET and TFTP). The command is designed to work without user interaction.”

The “remote-name” argument or capital “-O” loads a file from a URL, using the file name from the URL as explained in the curl man page.

” -O, –remote-name Write output to a local file named like the remote file we get. (Only the file part of the remote file is used, the path is cut off.)”

curl -O <url_including_a_file_name>

Download videos and audio

Install youtube-dl using pip:

 sudo pip install --upgrade youtube_dl

Install from the compressed tar.gz version of a repository that doesn’t require git to be installed on your laptop:

pip install --force-reinstall https://github.com/ytdl-org/youtube-dl/archive/refs/heads/master.tar.gz

You may also need to install the ffmpeg package:

sudo apt-get install ffmpeg

Download a video from youtube :

youtube-dl video_url

Download in a small format below 480 pixel height:

youtube-dl  -f 'bestvideo[height<=480]+bestaudio/best[height<=480]' https://www.youtube.com/watch?v=YAtLTLiqNwg

The repetition of the hight in the audio part is needed, otherwise, there will be no sound.

Download only the audio in an OGG Vorbis format

youtube-dl --extract-audio --audio-format vorbis video_url

Download only the audio in an .mp3 format

youtube-dl --extract-audio --audio-format mp3 video_url

Create bash aliases to download various audio formats from youtube

alias youtubevorbis="youtube-dl --extract-audio --audio-format vorbis"
alias youtubemp3="youtube-dl --extract-audio --audio-format mp3"

If you encounter some Errors 403 forbidden, removing the cache might help

error 403 with youtube-dl

youtube-dl --rm-cache-dir

You can also download multiple URLs from a batch file:

-a, –batch-file FILE

File containing URLs to download (‘-’ for stdin), one URL per line. Lines starting with ‘#’, ‘;’ or ‘]’ are considered as comments and ignored.

Lower resolution

See the man page “FORMAT SELECTION”. Download in a lower resolution (copied from example in man youtube-dl)

youtube-dl -f 'bestvideo[height<=480]+bestaudio/best[height<=480]' --extract-audio --audio-format vorbis --sleep-interval 30 -i --batch-file /tmp/dl.md

Video format selection

youtube-dl -f 'bestvideo[height<=480]+bestaudio/best[height<=480]' <URL>

Download only audio (using a low resolution for the video which is dropped)

youtube-dl -f 'bestaudio/best[height<=480]' --extract-audio --audio-format vorbis --sleep-interval 30 -i --batch-file /tmp/dl.md

Parse html files for URLS

How to use grep and cut in script to obtain website URLs from an HTML file Nice examples such as this one with decomposition

grep -Eoi '<a [^>]+>' source.html |
grep -Eo 'href="[^\"]+"' | 
grep -Eo '(http|https)://[^/"]+'

and this one which extracts all top level domain present in an html page

cat urls.html | grep -Eo "(http|https)://[a-zA-Z0-9./?=_-]*" | sort -u

This is how I extracted all links from an html file:

grep -Eo '(http|https)://[^"]+' Musicwelistento.html | sed -e  's/https:\/\/www.google.com\/url?q=//g' > song_links.txt

Edit

Edit Image files

Resize or reduce image size

Resize a picture to 2000 px wide using ImageMagick:

convert picture1.jpg -resize 2000 picture2.jpg

Resize a picture to 2000 px wide and also convert it to gray scale:

convert picture1.jpg -resize 2000 -type Grayscale picture2.jpg

Batch processing, resize all jpg images and place the resized images in a new folder

mkdir small
mogrify -resize 2000 -path ./small *.jpg
mogrify -resize 2000 -path ./small *.jpeg
mogrify -resize 2000 -path ./small/ *.JPG
mogrify -resize 2000 -type Grayscale -path ./small *.jpg
# Convert iphone HEIC pictures to jpg in smaller format
mogrify -resize 2000 -path ./small -format jpg *.HEIC

Convert

Convert to different color scheme, other file format or rotate images. For example to convert from png to jpg and change size

mogrify -resize 2000 -path ./newfolder -format jpg *.png

Convert to black and white (tricky to get the threshold right):

convert picture1.jpg -threshold 40% picture2.jpg

Other option to convert to black and white good black and white bimodal images:

convert -monochrome in.png out.png

Note this doesn’t result in smaller images. To get smaller files, set the -quality parameter to a value between 1 and 100.

Rotate

Batch processing, rotate all images clockwise or counter clockwise

mogrify -rotate 90 ../small/ *.JPG
mogrify -rotate 270 ../small/ *.JPG

Error: “cache resources exhausted” can appear in conversion of large images.

sudo find / -name "policy.xml"

SO explanation Edit the file as super user

sudo vim /etc/ImageMagick-6/policy.xml

Change

to

A4 portrait Scan emulation

Combine some of the commands above to create a scanned A4 portrait pdf document from pictures taken from a camera. First reduce the size of the pictures to 2000 pixel wide and rotate them 90 degrees, then combine all images in a pdf. Use a higher compression for smaller image sizes. You might need to change the rotation from -90 to 90 depending on how the camera was held.

mkdir small
mogrify -resize 2000 -quality 40 -rotate -90 -path ./small/ *.JPG
cd small/
img2pdf --output out.pdf --pagesize A4 --border 0cm:0cm *.JPG

Remove the -quality argument to keep default higher quality images.

mogrify -resize 2000 -rotate 90 -path ./small/ *.JPG

Use only the quality argument without rotation

mogrify -quality 40 -path ./small/ *.JPG

Edit music files

Mp3 tags from file names tagpy

Crop or trim music files

Cut or crop a music file, to start only from the 6th second

ffmpeg -i hungarian_dance.ogg -ss 6 -c copy file-2.ogg

Stack exchange question trim audio file using start and stop times

ffmpeg -i file.mkv -ss 20 -to 40 -c copy file-2.mkv
ffmpeg -i file.mkv -ss 00:00:20 -to 00:00:40 -c copy file-2.mkv

Edit PDF files

Combine PDF

Commands based on the poppler library for PDF manipulation. Merge multiple PDF into one:

pdfunite in-1.pdf in-2.pdf in-n.pdf out.pdf

Alternatively, pdftk can be used to merge PDF files

pdftk input1.pdf input2.pdf cat output output.pdf

Extract pages from a pdf

pdftk full-pdf.pdf cat 12-15 output outfile_p12-15.pdf

Also works with several distant groups of pages

pdftk Accordeon-facile-Volume-3.pdf cat 10-11 14-17 30 output la_java_bleue.pdf

Rotate PDF

Stack Exchange Linux and Unix answer:

pdftk input.pdf cat 1-endwest output output.pdf
pdftk input.pdf cat 1-endwest output output.pdf

Reduce pdf size (lower image resolution)

Resize all images in input.pdf for screen viewing screen (low resolution), ebook (slightly higher resolution) or printing printer (higher resolution):

gs -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dPDFSETTINGS=/screen -dNOPAUSE -dQUIET -dBATCH -sOutputFile=output.pdf input.pdf
# Preferred setting ebook:
gs -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dPDFSETTINGS=/ebook -dNOPAUSE -dQUIET -dBATCH -sOutputFile=output.pdf input.pdf
gs -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dPDFSETTINGS=/printer -dNOPAUSE -dQUIET -dBATCH -sOutputFile=output.pdf input.pdf

Example size reduction on a pdf with 4 figures. The input pdf file had an unusually large size (12Mb). File size of the output files with comments:

  • 289K output_screen.pdf One of the images, a map, is really pixelised. Bar charts look fine.

  • 401K output_ebook.pdf the map looks better on screen at this resolution.

  • 625K output_printer.pdf it is hard to notice the difference between the map at this resolution and the map from the ebook format. Probably because the original map in the paper was pixelized anyway.

Based on this quick assessment, ebook seems to be a good compromise in terms of file size and image resolution.

Split PDF

Split a pdf in two

mutool poster -x 2 cbmflow.pdf output.pdf
# if necessary, install mutool with 
apt install mupdf-tools

Replace strings in text files

Replace strings

first="I love Suzy and Mary"
second="Sara"
first=${first/Suzy/$second}

Replace strings with sed

sed -i  's/pattern/replacement/g' bli.txt 

sed -i  's/^.*\://g' input_file.txt # edit file in place
grep EMAIL input_file.txt |sed  's/^.*\://g' > output_file.txt

Add a line above and below a line that contains a particular string. This was necessary for a Latex to Lyx document conversion. The Latex file had problematic characters in href links. Conversion worked fine without the href links, to keep the links

I quoted them in \begin{verbatim} statements. sed add # before a line containing a pattern sed performing several substitutions

# Test output to stdout placing # before and after a matching line
sed -e /href/s/^/\\n#\\n/ -e /href/s/$/\\n#\\n/ clipboard_3_to_delete.tex| grep -C2 href

Modify the file in place

Both of these work

cd /tmp
cat "Export Quantity" >> file.md

bli () {
perl -i'' -pi -e "s/$1/$2/g" file.md
}
bli "Export Quantity" "export_quantity"

bla () {
perl -i'' -pE "s/$1/$2/g" file.md
}
bla "Export Quantity" "export_quantity"

Remove duplicated lines from a file

awk '!a[$0]++' input.txt

Select an editor

To choose your preferred text editor, used for example by crontab -e, run:

select-editor

Events

Calendar

Reddit CLI calendar applications

Cron

In order to use two differet /etc/hosts depending upon the time, I added the following to /etc/hosts_worktime:

0.0.0.1 www.youtube.com
0.0.0.1 youtube.com

I then created a cron job that copies that files to /etc/hosts at 9am. Start the crontab editor:

crontab -e

Add these lines:

0  9 * * 1-5 ln -f /etc/hosts_worktime /etc/hosts
0 17 * * 1-5 ln -f /etc/hosts_playtime /etc/hosts

I initially used cp instead of ln -f, but it’s better to make a hard link so any subsequent modification of /etc/hosts will actually be kept in one of the file, and not overwritten by the cp. I noticed that youtube.com was still available for a little while after I blocked it. I’m not sure it’s necessary or if you simply need to wait a little bit for the change to the hosts file to take effect. Anyway here is an instruction to restart networking and clear the DNS cache:

sudo /etc/init.d/networking restart

To do lists

Taskwarrior

To add a small unit of work in the Kitchen project:

task add project:Kitchen Select floor tiles

Environment variables

Environment variables are the way the tell processes where resources are located.

For example

export PYTHONPATH="$HOME/repos/biotrade/":$PYTHONPATH

Tells python where the biotrade package is located.

PostGreSQL utilities createuser and createdb can be told which port to use by specifying the following environment variable beforehand:

export PGPORT=5433

The following variables can be used to specify a user and password inside a postgresql docker container

POSTGRES_USER=user_name
POSTGRES_PASS=password

To display an environment variable in bash

echo $VAR_NAME

To remove an environment variable (without the $ sign) in bash

unset VAR_NAME

Find

Find files

Find files in subdirectories of the current directory (Quotes are requited to prevent shell command expansion). Find files of a specific extension:

find -name "*.pdf"

Like -name, but the match is case insensitive

find -iname "*phd*"

Find files of a specific modification time:

find . -mtime 0 # modified in the last 24 hours

Find files not having a specific extension:

find . -type f -not -name '*.csv'
# Alternative syntax
find . -type f ! \( -name '*.csv' \)

Limit the depth of the search

find . -maxdepth 1 -name "*.pdf"

Find files in the whole system

locate filename

Show the beginning of many files

To show the beginning of many files in a hierarchy:

head $(find -name *.csv)

Get information about a file

Determine file type and encoding. The -i option shows the mime type strings.

file filename
file -i filename

list a directory

ls
ls -R # list subdirectories recursively
ls -lh # sizes in human readable format

Display the last modification date of a file

date -r <filename>
ls -l <filename>

More file information

stat <filename>

Grep search and filter text files

Search with Grep

grep "text" file.txt

Recursive search (-r) in all files in the directory for a given pattern, case insensitive (-i)

grep -ri "pattern" .

Recursive search only for a given file extension https://stackoverflow.com/questions/12516937/how-can-i-grep-recursively-but-only-in-files-with-certain-extensions

grep -r -i --include=\*.txt 'searchterm' ./

Give 1 line of context before and 2 lines of context after:

grep -B 1 -A 2 pattern README.md

Search for lines containing “Done” or “encountered” only in specific log files recursively in a directory. Also displays the line count at the end.

grep -r Done --include runner.log | tee /dev/tty |wc -l
grep -r encountered --include runner.log | tee /dev/tty |wc -l

Note: How to pipe stdout while keeping it on screen

Show all csv files mentioned in a source code (supposing the csv file name contain only alphanumeric characters). Use Extended regular expression (-E) to search for alphanumeric characters, only show the matching result (-o), not the full line and show the line number (-n) where the string occurs:

grep -Eon "[[:alnum:]]+.csv" file.txt

Count occurrences of a word in a file

grep -roh word filename.txt  | wc -w

Regexp match beginning of and end of line with ^ and $.

Search and filter tables

Awk tutorial, for example filter a large file for lines that have a third field (product code) starting with 44, keep the header line:

awk -F, '$3 ~ /^44/||NR==1' nc201501.dat|less

Search the lines that contain “paper” in the forth column of a file

awk "/paper/{print $4}" /tmp/file_name.csv

Can be used in vim to filter the file in place (seen in a blog post called Vim Kōans)

:%!awk "/paper/{print $4}"

Search in pdf files

Search a text pattern in all PDF files present in a directory:

pdfgrep pattern *.pdf

Look at lines of a text file with less

Less and vim can handle very large text files (several Gb). Go to a specific line using less

<line number>g

To display line numbers in less

-N

Go to a specific line using vim

:<line number>

To display line numbers in vim

:set nu

Tail

Follow the end of a log file as it is written

tail -f

See tabs and end of line characters

See tab and end of line characters in a text file

cat -te filename |less

List files

List all files in a directory

ls

List a file with its full directory path

realpath README.md
ls -ld $PWD
ls -ld ~/downloads/*

To include dot files, pipe the output of ls -a to realpath:

ls -a | xargs realpath

To list subdirectories recursively:

ls -aR | xargs realpath

In case of you have spaces in file names, xargs recomments using the -o option to prevent file names from being processed incorrectly, this works best with the output of find -print0 and it starts to look a lot more complex than other answers:

find -print0 |xargs -0 realpath

The above is an answer I posted on Stackoverflow. See also a Unix and Linux stackexchange question on how to list all files in a directory with absolute path.

Tree

List contents of a directories in a tree like format

tree
tree -d # List directories only, no files

Ignore files that contain compiled python bytecode with the -I option

tree -I "*.pyc"

Show only python file matching the "*.py" pattern with the -P option

tree -P "*.py"

Networking

Show IP addresses

Show the IP address

ip addr show

Show the state of network interfaces

ip link

Check ports

Check open ports on a system

nmap -Pn <hostname or IP address>

MAC Address

Show the MAC address of each connection

ip link

Show the address of a particular interface, in this case wlo1:

ip link show wlo1

MAC address spoofing.

Show domain name and host name

Show the domain name

domainname

Show the host name

hostname

Open files and folders in the GUI

If bash is running inside a Graphical desktop environment, you can open the current folder with

xdg-open .

And open any file with the preferred application with for example

xdg-open filename.pdf
xdg-open filename.mp4

I created an alias in .bash_aliases

alias open="xdg-open"

There is already an open command on some systems. But creating an alias for it should be safe according to an answer to this SO question because “once you alias it, it’s valid only for your user”.

The alias makes it possible to open files with the simple command

open .

Open many files

Open many files with xdg-open

find -iname '*.pdf' -print0 | xargs -0 -n 1 xdg-open

These simpler version of the same command have issues with files that have spaces or quotes in their names

ls *.pdf| xargs -n 1 open
ls *.pdf |xargs -L 1 open

Move, copy, rename, compress files

Copy a file

cp file1 file2

To copy a file from /tmp/file.md to /tmp/file2.md

cp /tmp/file.md /tmp/file2.md

Rsync

Copy a folder and all it’s content to a remote server:

rsync -zav ~/repos/forobs/tradetools/ edartrade:repos/tradetools/
rsync -zav ~/repos/forobs/tradetools edartrade:repos

Note the slash at the end of the first repo is very important, other wise it crease a sub repos with the same name inside edartrade on the remote machine. More explanations for the second command on the section “slash or no slash”.

Rsync man page:

Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon. […] It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Rsync is widely used for backups and mirroring and as an improved copy command for everyday use.

The most important option is -a:

-a, --archive archive mode; equals -rlptgoD (no -H,-A,-X)

The option -a does all of the following:

-r, --recursive             recurse into directories
-l, --links                 copy symlinks as symlinks
-p, --perms                 preserve permissions
-t, --times                 preserve modification times
-g, --group                 preserve group
-o, --owner                 preserve owner (super-user only)
    --devices               preserve device files (super-user only)
    --specials              preserve special files
-D                          same as --devices --specials

Other options I have used:

-v, --verbose increase verbosity, use -vv for more verbose.
-z, --compress compress file data during the transfer

To sync only some file extensions

“So if we only want *.sh files we have to exclude all files --exclude="*", include all directories --include="*/" and include all *.sh files --include="*.sh".”

Sync only .mdb files that are the output of a model

rsync -zav --include="*/" --include="*.mdb" --exclude="*"  /home/paul/rp/cbmcfs3_data/scenarios/static_demand/ edartrade:static_demand/

Slash or no slash

SO slashes and the rsync command quoting the rsync man page:

“A trailing slash on the source changes this behavior to avoid creating an additional directory level at the destination. You can think of a trailing / on a source as meaning”copy the contents of this directory” as opposed to “copy the directory by name”, but in both cases the attributes of the containing directory are transferred to the containing directory on the destination. In other words, each of the follow‐ ing commands copies the files in the same way, including their setting of the attributes of /dest/foo: ”

rsync -av /src/foo /dest
rsync -av /src/foo/ /dest/foo

Flatten a file hierarchy

Copy all files in the folder hierarchy to a flat folder at the root. solution from askubuntu

find . -type f -print0 | xargs -0 -I file mv --backup=numbered file .

I also follow with another command

rename 's/((?:\..+)?)\.~(\d+)~$/_$2$1/' *.~*~

to keep the numbering of duplicated files before the extension.

Compress multiple files

7z archive

Compress

7zr a -si<filename> <filename>.7z

Extract to current directory

7zr x <filename.7z> # keep directory structure (see also multiple files below)
7zr e <filename.7z> # extract all files in the same directory

List content of an archive

7zr l <filename.7z>

Convert a 7z file to gz using a pipe similar to this answer.

7zr e -so <filename.7z> | gzip -c > <filename.gz>

Bzip archive

Uncompress bzip file

bzip2 -d <filename.bz2>

Gzip archive

Compress a file using gzip compression

gzip <filename>

Decompress a file

gunzip <filename.gz>

Tar gzip archive

How do I compress a whole directory?

tar -zcvf archive-name.tar.gz directory-name

Where

-z: Compress archive using gzip program
-c: Create archive
-v: Verbose i.e display progress while creating archive
-f: Archive File name

Askubuntu to extract a .tgz file with tar you need to use,

tar -xvzf /path/to/file.tgz
tar -xvzf /path/to/file.tgz -C /path/where/to/extract/ 
# Extract in a folder that has the archive name
tar --extract -f /path/to/file.tgz --one-top-leve

Decompress an archive containing multiple files Keep full path of files within the archive To extract content from the archive in the current directory

tar -zxvf archive-name.tar.gz

Zip archive

Create the zip archive of a directory

zip dir.zip -r dir

Recursive copy of some files

Recursive copy some files while keeping the directory structure

find . -name '*.csv' -exec cp --parents \{\} /target \;

Move files from various sub folders to the same folder see the section on a flatten a file hierarchy for a description.

cd ~/Pictures/scan

Rename files

For example to rename all upper-case .JPG extension into lower-case .jpg extension.

rename 's/\.JPG$/\.jpg/' *.JPG

Program bash scripts

Loop over file names

Display all .ogg files in a loop

for i in *.ogg; do echo "${i%.ogg}"; done

Makefile

I use GNU make and the following simple make file to generate all pages in this website:

render:
     Rscript -e "rmarkdown::render_site()"

Build an R package

The makefile below builds the FAOSTAT R package. I modified it from knitr’s makefile.

# Usage:
# build the package's tarball, check and remove the ../FAOSTAT.Rcheck folder:
# make
# build the package's tarball and check (keep the ../FAOSTAT.Rcheck folder):
# make check
# generate the documentation:
# make docs
#
# This makefile was copied and apdapted from  yihui / knitr 
# https://github.com/yihui/knitr/blob/master/Makefile

# prepare the package for release
PKGNAME := $(shell sed -n "s/Package: *\([^ ]*\)/\1/p" DESCRIPTION)
PKGVERS := $(shell sed -n "s/Version: *\([^ ]*\)/\1/p" DESCRIPTION)
PKGSRC  := $(shell basename `pwd`)

all: docs check clean

deps:
    tlmgr install pgf preview xcolor;\
    Rscript -e 'if (!require("Rd2roxygen")) install.packages("Rd2roxygen", repos="http://cran.rstudio.com")'

docs:
    R -q -e "devtools::document(roclets = c('rd', 'collate', 'namespace', 'vignette'))"

build:
    cd ..;\
    R CMD build $(PKGSRC)

install: build
    cd ..;\
    R CMD INSTALL $(PKGNAME)_$(PKGVERS).tar.gz

check: build
    cd ..;\
    R CMD check $(PKGNAME)_$(PKGVERS).tar.gz --as-cran

clean:
    cd ..;\
    $(RM) -r $(PKGNAME).Rcheck/

Compile a latex article

This make file compiles a latex article to pdf and generate a zip file containing all documents necessary to generate the article and another zip file containing the figures.

# Usage:
# make        # Generate the pdf and zip files for the submission. 
# make render # Render the pdf, keep all automatically generated files for Latex diagnostics.
# make clean  # Keep a copy of the pdf one level above and remove automatically generated files.
# make zip    # Generate the zip archive of the article and figures for the submission process.
#
all: render clean zip

render:
     latexmk -pdf -bibtex -use-make template.tex

clean:
     cp template.pdf ../manuscript.pdf || echo "Template.pdf not there."
     git clean -xf

zip: # Remove the zip archive otherwise deleted files will remain in the archive and it will be simply updated
     rm ../manuscript.zip ../figures.zip || echo "No existing zip archive to delete."
     zip ../manuscript.zip -r *
     zip ../figures.zip -r figures/*
     echo "Verify that the pdf and zip archives have been updated by checking their update time."
     ls -lh ../*.zip
     ls -lh ../*.pdf

Convert documents with pandoc

Generic makefile for pandoc

Pipe

Pipe the output of ls to grep

ls | grep ".md$"

When the command doesn’t read from stdin by default, use xargs

echo README.md | xargs ls -lah
ls -a | xargs realpath

Based on an explanation of xargs and pipes. When piped to cat, this command simply displays the text “README.md”

echo README.md | cat

Using xargs, the output of echo is interpreted as a file name

echo README.md | xargs cat

Try catch

Is there a try catch command in bash

There is no try/catch in bash; however, one can achieve similar behavior using && or ||.

Using ||: if command1 fails then command2 runs as follows

command1 || command2

Similarly, using &&, command2 will run if command1 is successful

Variables

Variables

message = "Hello World"
echo $message

Wild cards

Search for a pattern in python and R files:

grep -n <pattern> *{py,R}

Read

Hexadecimal dump

Show the last 3 lines of a file as a hexadecimal dump

xxd -s -0x30 Trade_DetailedTradeMatrix_E_All_Data_\(Normalized\).csv

Play Midi files

Install a software synthesizer to play midi files. VideoLan Wiki

If the FluidSynth codec is not shown in VLC’s preferences, you have to install it as well as sound fonts. E.g. on Ubuntu 14.04 and derivatives it is in the vlc-plugin-fluidsynth package, while the fluid-soundfont-gs and fluid-soundfont-gm packages install some sound

Installation instruction:

sudo apt install vlc-plugin-fluidsynth fluid-soundfont-gs fluid-soundfont-gm

Play music with cmus

Cmus tutorial:

man cmus-tutorial

Cmus has several view modes. When starting cmus for the first time press 5 to go to the file-browser panel, choose a folder and press “a” to add it to the cmus library. Then press “2” to go to the simple library view or press “1” to go to the artist/album view.

Here is the list of view modes:

1 view tree
2 view sorted
3 view playlist
4 view queue
5 view browser
6 view filters
7 view settings
Common Context Remote Command Description
b player-next
C Continue at end of track
c player-pause
q quit -i
s shuffle
t toggle show_remaining_time
v player-stop
x player-play
z player-prev
e append tracks to the play queue
E prepend tracks to the play queue

Commands can be sent through a command line with cmus-remote, for example to pause a running instance of cmus (and restart it):

cmus-remote -C player-pause

I use this in the pomodoro gnome app to pause music at breaks.

Volume control

Common Context Description
= :vol +10%
+ :vol +10%
- :vol -10%
[ :vol +1% +0%
] :vol +0% +1%
{ :vol -1% -0%
} :vol -0% -1%

Jump to currently playing song i. Follow currently playing song f see issue 206

There is a feedback of all the currently active options at the bottom right of the cums screen.

Playlist

Create a playlist file, in bash

find -name '*.ogg'> playlist.m3u

Open a playlist in the playlist view (4), in the cmus command pane

:load playlist.m3u

Add a song from the library view (2) to the playlist view (4)

y

Delete a song from the playlist view (4)

:suppr

Save the modified playlist file

:save

You can also create a playlist within cmus with

:pl-create playlist_name

Export the playlist with

:pl-export playlist_enfants

Cmus and other audio programs

Some programs such as Skype or MS teams (based on Skype and chromium) turn off the volume of cmus when entering into a phone call. They sometimes forget to put it back on. You can turn the volume back on into Gnome’s sound configuration menu, under the applications tab.

Read CSV in bash

Read CSV in bash

Read text files

With less or vim (see the vim page).

Head and tail

Display the first 2 lines of a text file

head -2 file.txt

Display the last 2 lines of a text file

tail -2 file.txt

Record a terminal session

Record a terminal session with tools such as:

  • asciinema

  • ttygif

Refactor

A one liner to replace strings with perl in a git repository. The examples below replace all occurrences of the first string with the second string, for all files under git version control.

git grep -lz 'first_string'| xargs -0 perl -i'' -pE "s/first_string/second_string/g"
git grep -lz 'readcsvfromgauss'| xargs -0 perl -i'' -pE "s/readcsvfromgauss/readcsvfromgauss0/g"
git grep -lz 'RMySQL::'| xargs -0 perl -i'' -pE "s/RMySQL::/RMariaDB::/g"

Alias to refactor, use single quotes for the patterns:

refactor() {
    echo "Replacing $1 by $2 in all files in this git repository."
    git grep -lz "$1"| xargs -0 perl -i'' -pE "s/$1/$2/g"
}

When there are slashes in the patterns, use another separator

refactor() {
    echo "Replacing $1 by $2 in all files in this git repository."
    git grep -lz "$1"| xargs -0 perl -i'' -pE "s|$1|$2|g"
}

Usage, for example replace ‘word’ with ‘sword’:

refactor word sword
git diff

Security and authentication

Generate passwords

apg is an automated password generator. According to the Debian maintainers, it is not updated upstream and it might be replaced by another tool as soon as “a maintained software with a comparable feature set becomes available”. Usage:

apg

A post on Unix & Linux Stack Exchange recommends using /dev/urandom as such

cat /dev/urandom | tr -dc 'a-zA-Z0-9' | head -c 20

To create passwords that are hard for attackers to guess but easy to remember, have a look at this xkcd comic strip. It explains how password entropy rises faster with the number of characters than with the character set weirdness.

When possible, it’s better to authenticate with private and public key cryptography.

SSH Secure Shell

Generate SSH keys and add them to the agent

Mon Sep 28 09:36:26 CEST 2020 Generate a new key

cd ~/.ssh
ssh-keygen <id_rsa_keyname>

Temporarily add a key to ssh

ssh-add ~/.ssh/<id_rsa_keyname>

List keys

ssh-add -l

After a restart of the ssh agent

killall ssh-agent
eval `ssh-agent`

This key will not be available anymore to permanently add a key to the ssh config file (SO) Edit your ~/.ssh/config file:

IdentityFile ~/.ssh/<id_rsa_keyname>

If the key was generated with a password,and you don’t exactly remember the password, you can test a few password you might remember with:

ssh-keygen -y -f <private_key_file_name>

Note if you copy keys from one machine to another, you might encounter this error: ssh “permissions are too open” error

Change the permission so only the current user has access to the file:

chmod 600 id_rsa

Load SSH keys into the remote server

Based on Install the public key in remote server Once you have an ssh key on your local machine, you need to load it into the remote server’s authorized keys as such:

ssh-copy-id -i $HOME/.ssh/id_rsa.pub user@server

OR just copy the public key in remote server as authorized_keys in ~/.ssh/ directory:

scp $HOME/.ssh/id_rsa.pub user@server:~/.ssh/authorized_keys

The remainder of the cybercity.biz document explains how to disable the password based login on a server.

Check a key’s fingerprint

On github, ssh key fingerprints are displayed in md5format. To display the fingerprint in md5format, enter:

ssh-keygen -l -E md5 -f ~/.ssh/id_rsa.pub

How to generate an SSH key fingerprint

Forward agent and alternatives

Alternatives: - use ProxyCommand - Use a github or gitlab deploy token to clone a repository on the server

Log in to a secure shell and secure copy

log into a remote machine

ssh user@remote_machine

Copy a local file to a file on the remote machine

scp local_file_name user@remote_machine:path_to_file/file_name

Copy a file from the remote machine to a local file

scp user@remote_machine:path_to_file/file_name  local_file_name

Copy a full directory (dmouraty) from the remote machine:

 scp -rp user@dest:/path destdirectory

SSH config

To speed up login aliases can be set up in ~/.ssh/config

Host host_name
  user paul
  hostname ip_address

SSH tunnel

Tunnel to a jupyter notebook on a server

alias tunnelnotebook="ssh -L 8888:localhost:8888 servername -N"

Passwordless access for root user

How to set up passwordless SSH access for root user

Two factor authentication

Note this section is not related to bash. I put it here for now, may find another place later.

On your device (usually your phone): Install a compatible application, like:

  • Authenticator: open source app for iOS devices.

  • andOTP: feature rich open source app for Android which supports PGP encrypted backups.

  • FreeOTP: open source app for Android.

  • Google Authenticator: proprietary app for iOS and Android.

  • SailOTP: open source app for SailFish OS.

GPG signing keys

See also the communication page on security issues related to PGP such as the Efail vulnerability. According to several authors PGP’s cryptography is really old and it is probably not the best encryption tool nowadays. Part of the issue is also related to the fact that email is a fundamentally unsecure communication tool.

Links:

Generate a PGP signing key

Generate a PGP key

gpg --gen-key

List the keys

gpg --list-keys

Export the key to a file

gpg -a --export keyid > file.txt

Public key servers

To submit your key to a server, export the key

gpg -a --export name@emailserver.com > public_key_sender.asc

My keys were visible on pgp.mit.edu search=rougieux. 2019 Update following a look at a super user answer on public key servers, I uploaded my public key to

https://keys.openpgp.org/upload

My keys are searchable at

https://keys.openpgp.org/

Import a key from a public key server

curl -s 'https://keys.openpgp.org/vks/v1/by-fingerprint/6813A4C1B326F4A6A65336D9B6838AB252DD02F7' | gpg --import

GNUPG instructions from openpgp.org

Encrypt and decrypt

Encrypt a document for a particular recipient

gpg --encrypt --recipient user@email file.ext
gpg -er user@email file.ext

Encrypt and sign a document

gpg --encrypt --sign --recipient user@email file.ext
gpg -esr user@email file.ext

Decrypt a document

gpg --output file.ext --decrypt file.ext.gpg

Sign files

Sign a file

gpg --sign filename.ext

Export GPG private and public key to a file

Verify the signature of a file

gpg --verify filename.ext.gpg

Test a signature

Test if GPG signing is working with a particular key that you own

echo "test" | gpg --clearsign -bsau CC44A3EC868FDAE69F0E57148A77836103E2ADB0

# gpg: skipped "CC44A3EC868FDAE69F0E57148A77836103E2ADB0": No secret key
# gpg: [stdin]: clear-sign failed: No secret key

Trust other public keys

Sign public keys

“Once a key is imported it should be validated.”

alice% gpg --edit-key blake@cyb.org

pub  1024D/9E98BC16  created: 1999-06-04 expires: never      trust: -/q
sub  1024g/5C8CBD41  created: 1999-06-04 expires: never
(1)  Blake (Executioner) <blake@cyb.org>

Command> fpr
pub  1024D/9E98BC16 1999-06-04 Blake (Executioner) <blake@cyb.org>
             Fingerprint: 268F 448F CCD7 AF34 183E  52D8 9BDE 1A08 9E98 BC16

“A key’s fingerprint is verified with the key’s owner. This may be done in person or over the phone or through any other means as long as you can guarantee that you are communicating with the key’s true owner. If the fingerprint you get is the same as the fingerprint the key’s owner gets, then you can be sure that you have a correct copy of the key.

After checking the fingerprint, you may sign the key to validate it. Since key verification is a weak point in public-key cryptography, you should be extremely careful and always check a key’s fingerprint with the owner before signing the key.”

Command> sign

pub  1024D/9E98BC16  created: 1999-06-04 expires: never      trust: -/q
             Fingerprint: 268F 448F CCD7 AF34 183E  52D8 9BDE 1A08 9E98 BC16

     Blake (Executioner) <blake@cyb.org>

Are you really sure that you want to sign this key
with your key: "Alice (Judge) <alice@cyb.org>"

Really sign?

“Once signed you can check the key to list the signatures on it and see the signature that you have added. Every user ID on the key will have one or more self-signatures as well as a signature for each user that has validated the key.”

Command> check
uid  Blake (Executioner) <blake@cyb.org>
sig!       9E98BC16 1999-06-04   [self-signature]
sig!       BB7576AC 1999-06-04   Alice (Judge) <alice@cyb.org>

Edit the trust level

Edit the trust level of a key

Validating other keys on your public keyring

alice% gpg --edit-key blake

pub  1024D/8B927C8A  created: 1999-07-02 expires: never      trust: q/f
sub  1024g/C19EA233  created: 1999-07-02 expires: never     
(1)  Blake (Executioner) <blake@cyb.org>

Command> trust
pub  1024D/8B927C8A  created: 1999-07-02 expires: never      trust: q/f
sub  1024g/C19EA233  created: 1999-07-02 expires: never     
(1)  Blake (Executioner) <blake@cyb.org>

Please decide how far you trust this user to correctly
verify other users' keys (by looking at passports,
checking fingerprints from different sources...)?

 1 = Don't know
 2 = I do NOT trust
 3 = I trust marginally
 4 = I trust fully
 s = please show me more information
 m = back to the main menu

Your decision? 3
                
pub  1024D/8B927C8A  created: 1999-07-02 expires: never      trust: m/f
sub  1024g/C19EA233  created: 1999-07-02 expires: never     
(1)  Blake (Executioner) <blake@cyb.org>

Command> quit

Manage keys

List keys

gpg --list-keys
gpg --list-public-keys
gpg --list-secret-keys

List files in the keyring

tree ~/.gnupg/

Delete a key from the public keyring

gpg --delete-keys name

Delete a key from the private keyring

gpg --delete-secret-keys name

SKS pool

keys.openpgp.org FAQ

“Is this server part of the”SKS” pool? ”

No. The federation model of the SKS pool has various problems in terms of reliability, abuse-resistance, privacy, and usability. We might do something similar to it, but keys.openpgp.org will never be part of the SKS pool ITSELF. ”

Linux Kernel Maintainer PGP guide

  • Linux Kernel Maintainer PGP guide

    “Trusting the developers, not infrastructure

    Ever since the 2011 compromise of core kernel.org systems, the main operating principle of the Kernel Archives project has been to assume that any part of the infrastructure can be compromised at any time. For this reason, the administrators have taken deliberate steps to emphasize that trust must always be placed with developers and never with the code hosting infrastructure, regardless of how good the security practices for the latter may be.

    The above guiding principle is the reason why this guide is needed. We want to make sure that by placing trust into developers we do not simply shift the blame for potential future security incidents to someone else. The goal is to provide a set of guidelines developers can use to create a secure working environment and safeguard the PGP keys used to establish the integrity of the Linux kernel itself.”

    Start by getting a small USB “thumb” drive (preferably two!) that you will use for backup purposes. You will need to encrypt them using LUKS – refer to your distro’s documentation on how to accomplish this. For the encryption passphrase, you can use the same one as on your PGP key. Once the encryption process is over, re-insert the USB drive and make sure it gets properly mounted. Copy your entire .gnupg directory over to the encrypted storage:

    “Remove the Certify key from your homedir - The files in our home directory are not as well protected as we like to think. They can be leaked or stolen via many different means: , by accident when making quick homedir copies to set up a new workstation, by systems administrator negligence or malice, via poorly secured backups, via malware in desktop apps (browsers,pdf viewers,etc), via coercion when crossing international borders Protecting your key with a good passphrase greatly helps reduce the risk of any of the above, but passphrases can be discovered via keyloggers, shoulder-surfing, or any number of other means. For this reason, the recommended setup is to remove your Certify key from your home directory and store it on offline storage.”

    Configure git to use your PGP key If you only have one secret key in your keyring, then you don’t really need to do anything extra, as it becomes your default key. However, if you happen to have multiple secret keys, you can tell git which key should be used ([fpr] is the fingerprint of your key):

      git config --global user.signingKey [fpr]

System

Disk usage and memory

Disk usage

du -hd1  # sub-folder size at depth=1 
du -hs * # sub-folder size summary

Display available space on drives

df -h

Display available RAM memory

cat /proc/meminfo

Environment variables

Show environment variables

printenv 
printenv LC_TIME

Jobs and processes

Monitor processes

Display Linux processes

top

Interactive process viewer

htop

Run a command every 2 seconds, for example to list a folder as new files are being written to it:

watch ls

Follow the end of a log file, as it is written

tail filename.log

Find out which program is listening to which port

sudo netstat -pna | grep <port_number>

Start, stop, move to background

A command followed by an & will run a job in the background.

List

jobs

Bring a job to the foreground

fg job_number

Stop a job

CTRL ^ Z

Move a stopped job to the background

bg

How to manage foreground and background processes “By default, the bg command operates on the most recently stopped process. If you’ve stopped multiple “

Quit a job

 CTRL ^ C

Kill a malfunctioning program:

kill process_id
kill -9 process_it # -9 forces the kill

Find a program id with:

ps aux

Kill a graphical program, by clicking on it:

 xkill

Keyboard

bash french blogger recommended a simple shell command to change keyboard layout :

sudo loadkeys fr

fr-keyboard on Debian wiki for a more permanent system configuration and use in GUI apps. Switching between keyboads can then be done with:

setxkbmap de
setxkbmap fr

Information about the system Debian-Admin Finding out basic information about your Hardware:

cat /proc/meminfo
cat /proc/cpuinfo

Find your Debian version

cat /etc/debian_version
lsb_release -a

Shortcuts Keyboard shortcuts for bash for example :

  • Ctrl+A to go to the beginning of a line
  • Ctrl+E to go to the end of a line

Documentation

Creating and running bash scripts
Unofficial Ubuntu starter guide.

Locale

Debian wiki page on Locale settings

Display locale settings

locale

To install new locales, run :

sudo dpkg-reconfigure locales

SSH forwards locales to the server. The server will try to use the given locale, but it might not have them installed. Locale variables have no effect in remote shell

How do I add a locale to a Ubuntu Server? Check which locales are supported:

locale -a

add the locales you want (for example ru):

sudo locale-gen ru_RU
sudo locale-gen ru_RU.UTF-8

run this update command

sudo update-locale

OS version and system name

OS release

cat /etc/os-release

System name

uname -a

file /sbin/init

hostname -f

Partitioning and formatting drives

Manipulate the disk partition table See forum post July 21st, 2012, 09:08 AM

sudo fdisk /dev/sdb
d # delete
w # write table to disk
sudo fdisk /dev/sdb
n # create a new partition
# yes to all default choices
w # write table to disk

Create a MS-DOS file system for USB drives

mkfs.vfat -n 'usb_drive_name' -I /dev/sdb1

You can also use the GUI gparted.

Format a USB drive

Format a USB drive as a FAT 32 partition.

sudo fdisk -l 
sudo umount /dev/mmcblk0p1 
sudo mkdosfs -F 32 -I /dev/mmcblk0

format a USB drive as a FAT 32 partition smaller than the drive size

sudo mkdosfs -I -F 32 /dev/mmcblk0 $((2*1024*1024))

I used the GUI tool Gparted. To see if a partition is primary, use the command: sudo parted /dev/mmcblk0 # then: print

This will print the type (primary or logical) of the partition.

Processors

Print the number of processors available

nproc

Print CPU architecture

lscpu

User activity

Monitor user activity

Last time the system was started

last reboot 
last

Super user

Start and quit a super user session

su
exit

Terminal multiplexers

Superuser.com tmux or screen

Screen

I used screen to detach and reattach bash sessions on remote hosts. This enabled me to run long algorithms on the remote host. I could disconnect in the evening from the ssh session and reconnect the following morning to check if the algorithm had finished running. The advantage of screen is that it is old and available on many systems.

Tmux

Tmux is under more active development than Screen.

Sessions

To start a session

tmux

To start and name a session:

tmux new -s session_name

To detach from a session:

tmux detach (CTRL + b + d)

To reattach to a session:

tmux attach -t session_name

If there is only one session it’s enough to call

tmux attach

To list current sessions:

tmux list-sessions

To switch to another session:

tmux switch -t session_name

To delete a session:

tmux kill-session -t targetSession

Windows (tabs)

Create a new windows

tmux new-window (CTRL + b + c)

Move to a window

tmux select-window -t :0-9 (CTRL + b + 0-9)

Rename the current window

tmux rename-window (CTRL + b + ,)

Panes (splits)

Following commands should be prefixed by CTRL + b.

Vertical split

%

Horizontal split

"

Kill a pane

x

Display pane numbers

q

How to switch directly to a pane in tmux?:

“You can jump directly to a pane by typing pane’s index while it is showed by display panes command.”

Swap and change pane positions

Zoom a pane and de-zoom it:

z

Switch layout from horizontal to vertical

space

Swap pane up and down or left and right

{
}

Bind the s key to create a small split window at the bottom of the screen

bind-key s split-window -v \; resize-pane -y 2

Uses an escaped semi colon to separate the two commands as explained on super user.

Movements

To scroll inside a pane press

CTRL + b + Page up down

tmux configuration

This answer explains how to create a tmux source-file. The configuration file tells tmux to create windows and panes and to start programs in each one of them. I started to create a tmux source file to start my favorite programs: A particular setup I use forces me to start these GUI programs from a command line:

neww
split-window -v 'lyx'  
split-window -v 'jabref'  
split-window -h 'rstudio' 
new-window

Start tmux with this configuration file:

tmux new-session -s work "tmux source-file ~/.tmux/work"

Make it a bash alias, by adding the following line to ~/.bashrc:

alias tmuxw='tmux new-session -s work "tmux source-file ~/.tmux/work"'

Reddit discussion on vim splits and tmux tabs

Reload the tmux configuration file

source-file ~/.tmux.conf

VIM and TMUX Navigation

I simply copie the tmux side of the configuration here:

Smart pane switching with awareness of Vim splits. See: https://github.com/christoomey/vim-tmux-navigator

is_vim="ps -o state= -o comm= -t '#{pane_tty}' \
    | grep -iqE '^[^TXZ ]+ +(\\S+\\/)?g?(view|n?vim?x?)(diff)?$'"
bind-key -n 'C-h' if-shell "$is_vim" 'send-keys C-h'  'select-pane -L'
bind-key -n 'C-j' if-shell "$is_vim" 'send-keys C-j'  'select-pane -D'
bind-key -n 'C-k' if-shell "$is_vim" 'send-keys C-k'  'select-pane -U'
bind-key -n 'C-l' if-shell "$is_vim" 'send-keys C-l'  'select-pane -R'

Time

Current date

Display the current data

date

Countdown timer

A superuser question gives a basic loop

N=100; while [[ $((--N)) >  0 ]]; do echo $N && sleep 1; done

There are many more elaborate answers under that question. One of them suggests the python based termdown

pip3 install termdown
termdown 10
termdown 25m

Users

Manage users and groups

Check your user id

id 
echo $UID
echo $USER

What group do you belong to as a user

groups

Add a new user

useradd username

Set a password for the new user

passwd username

Delete a user

userdel username

Temporary log in as a given user

su username

Add a user to the super users

adduser username sudo

That user needs to re-log into the shell for the change to take effect.

Show all users

getent passwd

Show all groups

getent group

List users and groups

List all users

less /etc/passwd

List all groups

less /etc/group
getent group

List all groups a user is a member of

groups paul

List all members of a group

getent group www-data
getent group sudo

Change file permission

Change user and group of a file

chown groupname:username filename

Change user and group of all files in a folder, recursively

# In one command
sudo chown -R paul:paul scenarios
# In two commands
sudo chown -R paul scenarios
sudo chgrp -R paul scenarios

Change file permission

chmod a=rwx filename
chmod 777 filename # all permissions to every one

Change file permissions recursively:

chmod 755 directoryname # all permissions to the user, only read and execute for other users
sudo chmod -R 770 scenarios

nice illustration of read, write execute permission codes on Linux.

Chmod instructions can be given with characters or numbers, chmod 777 or chmod a=rwx is a question of preference.

Some prefer 755 over 777 because giving write access to group and other users could be a security risk. 755 leaves read and execute rights to groups and other users. 755 is visible as "rwxr-xr-x" in ls -l. 
The default for document files on Debian seems to be chmod 644, visible as "-rw-r--r--" in ls -l.

Make the bash shell by default for a user

No path, or command history through SSH when logged in to server as someone other than root

Edit the users file

sudo vim /etc/passwd

Change

paul:x:1000:1000::/home/paul:/bin/sh

to

paul:x:1000:1000::/home/paul:/bin/bash

VPN

A VPN was useful when:

  • SMS interface didn’t work on orange.fr. I could access it through a VPN server based in Paris.
  • FTP download site based in Canada didn’t work from Italy. A colleague could access it through a VPN server based in Canada.
  • SSH access to remote servers didn’t work on company wifi. I could access the machines through the VPN.

The VPN didn’t manage to connect:

  • At Malpensa Airport

VPN access also removes localisation of site searches on google and google maps.

Example session,from a company wifi system which is probably blocking SSH traffic. The SSH connection breaks:

paul@debian:~/repos$ ssh host 
ssh_exchange_identification: Connection closed by remote host

I start the VPN:

paul@debian:~/repos$ vpn connect
Connecting to France - Paris - 1...     100.0%                       
Connected to France - Paris - 1

I try to reconnect to that server through SSH:

paul@debian:~/repos$ ssh host 
The authenticity of host '[.......eu-west-1.compute.amazonaws.com] ([__IP__])' can't be established.
ECDSA key fingerprint is SHA256:PTwO1p7Q3XqPE4Gs+KH37XqH88ePDYh5bNqKXyVpGAE.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[......eu-west-1.compute.amazonaws.com],[....]' (ECDSA) to the list of known hosts.

Now the connection works.

Some VPN access points caused issues when connecting to the corporate email server and intranet. The connection was blocked for the access point in Milan, Italy. It did work when using the access point in Strasbourg, France.

The access point from Strasbourg prevented access to Elsevier publications. With the message: > “There was a problem providing the content you requested”

I complained to and received the following message::

Dear Paul Thank you for your patience. My colleagues have informed me that the IP <………….> where you are accessing from has been tagged with high amount of retrieving content from ScienceDirect.

We do understand that your purpose of downloading and searching of content is for personal, non-commercial or scholarly use and it is difficult to be getting warnings every time you download content from the site. Please be advised that Elsevier cares about the integrity of our product and ensure data protection. With this, ScienceDirect continuously monitor signals of excessive downloading of content thus explains the warning you might have encountered. Please note as well that only the IP has been tagged and this concern may be resolved by using a different IP network.

Please reach out to your institution’s or organization’s administrator or librarian for an alternate access in ScienceDirect.

Thank you for your understanding. Please feel free to message us if you have any questions or require further assistance.

Changing the VPN location to Frankfurt allowed me to see these Elsevier publications again.

VPN software

wireguard wikipedia article wireguard

Service providers