Taming the TerminalIn the previous two instalments (17 & 18) of this series we learned how to represent patters with regular expressions, or, to be more specific, with POSIX Extended Regular Expression (or EREs). We used the egrep command to test our regular expressions, but we didn’t discus the command itself in detail. Now that we understand regular expressions, it’s time to take a closer look at both egrep, and it’s older brother grep, both commands for filtering and searching text.

Read more

I’m betting most people won’t be interested in this, but if anyone’s curious how this programmer goes about building up a perl module from scratch, you can watch along on as I build XKPasswd 2.0 over on GITHub: github.com/bbusschots/xkpasswd.pm.

I’m re-implementing XKPasswd from scratch. The resulting functionality will be mostly the same, but with some additions. The reason for starting over is two-fold. Firstly, the first implementation of XKPasswd was very much a prototype, and as with any prototype, I learned some valuable lessons, and there are lots of things I’d do differently if I was doing it again. Secondly, the first version of XKPasswd is almost three years old now, and since then, my Perl skills have increased a lot. Probably the single biggest difference between the me of 2014 and the me of 2011 is that I read Perl Best Practices, and started to run all my Perl code through Perl Critic. Another big difference is that, thanks to JQuery, I’ve fallen in love with Code References in all the languages I use that support them, including Perl.

Since this is a re-write, I’m really focusing on building a solid foundation, so I’m starting with the utility functions that will let me build up functionality gradually, and I’m writing the user documentation in parallel with the code. Before every commit to GITHub, everything that’s done so far is getting documented with POD, and, the code has to pass Perl Critic with no warnings.

Taming the TerminalIn the previous instalment we introduced the concept of Regular Expressions, and started to learn the POSIX ERE regular expression language, noting that POSIX ERE is a sub-set of the very commonly used Per Compatible Regular Expression (PCRE) language.

In this instalment we’ll learn more POSIX ERE syntax, and have a look at some examples of REs in GUI apps.

Read more

Taming the TerminalThis instalment is the start of a series of instalments relating to searching from the command line. Searching is all about patterns, and that means getting to grips with Regular Expressions (also called RegExps, RegExes or REs for short). Regular Expressions are languages for representing patterns, and are used throughout IT, not just on the command line. While this series focuses on the Terminal, an understanding of regular expressions will be helpful in many other places, from programming languages to GUI apps like programming editors, search utilities or file re-namers. It’s going to take us two instalments to properly describe regular expressions, but when we’re done we’ll have gained a very useful skill.

Read more

I sometimes take some stick for having a very defensive coding style. When ever I find myself making an assumption I throw in code to test it. “This function will only ever be called with a positive number”, OK, then add a test to throw an exception should a negative number be passed. You don’t want bad data ricocheting through your code because goodness knows what damage it will do! Similarly, my style is to always use explicit syntax, and, to avoid syntax shortcuts – sure, the ternary operator takes up less space on the screen, but there’s a price to pay for that terseness – it makes your code harder to read and hence to debug and maintain.

However, one of my very biggest bug-bears is the failure to brace control statements like conditionals and loops when they’re operating on a single line of code. This is the trap Apple fell into so spectacularly this week.

Read more

Taming the TerminalIn the previous instalment we introduced the concepts of streams, and looked at how every process has references to three streams as part of their environment – STDIN, STDOUT & STDERR. We went on to introduce the concept of operators that manipulate these streams, and we focused on the so-called ‘pipe’ operator which connects STDOUT in one process to STDIN in another, allowing commands to be chained together to perform more complex tasks. We mentioned the existence of operators for connecting streams to files, and the possibility of streams being merged together, but didn’t go into any detail. Well, that’s what we’ll be doing in this instalment.

Read more

Taming the TerminalRight back in the very first instalment we described the Unix philosophy as being Lego-like, that is, having lots of simply commands that do one thing well, and then assembling them together to do something really powerful. So far, we’ve only been working with a single command at a time, but that changes with this instalment. We’ll be introducing the concept of streams, which can be used to connect commands and files together.

Read more

Taming the TerminalIn the previous instalment we looked at how to make permanent changes to our environment. We made a permanent change to the PATH environment variable to demonstrate how it’s done (by editing ~/.bash_profile on a Mac, or ~/.bashrc on Linux). In this instalment we’ll look at two other kinds of environment changes you may wish to make by editing these files – specifically, aliases, and custom prompts.

Read more

Taming the TerminalIn the previous instalment we introduced the concept of the command shell environment, and we looked in detail at how shell and environment variables work. In this instalment we’ll focus on probably the single most important environment variable, PATH. We’ll look at what it does, how it’s initialised, and, in the process, we’ll learn how to make persistent customisations to our shell environment.

So far in this series I have been a little loose with the term command, I’ve avoided putting too fine a point on exactly what a terminal command is, but we’ll remedy that today. If you remember right back to the second instalment, we said that when entering commands on the command line, the first word is the command, and the other words (separated by spaces) formed the arguments to that command. We spent a lot of time discussing the vagaries of quoting the arguments, but we didn’t discus the command itself in any detail.

In BASH, when you enter a command, that command can actually be one of two things, a builtin BASH command, or, an executable file which BASH will execute for you. You can see the list of builtin commands on BSD-style Unixes (including OS X) with man builtin. On Linux you need to navigate to the SHELL BUILTIN COMMANDS section of the VERY long BASH man page for the same information.

When you enter a command in BASH the first thing it does is figure out whether or not the command is a builtin. If it is a builtin then BASH just does what ever it is you asked. Where things get interesting is when you enter a command that is not a builtin. What BASH does then is interpret the command as a request to run an executable file with that name. If BASH finds such a file it runs it, and if not, it gives an error like:

bart-imac2013:~ bart$ donky
-bash: donky: command not found
bart-imac2013:~ bart$

The obvious question is, how does BASH find the executable files to run? This is where PATH comes in.

Before we continue, lets print out the current value of PATH with the echo command and $ operator we learned about in the previous instalment:

echo $PATH

You should see a value that looks something like the following (though yours may well be shorter, mine is extra long because I use MacPorts to install Linux command line tools onto my Mac):

/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin

The value of PATH is a : delimited ordered list of folders.

Each time you enter a command that is not a builtin, what BASH does is search each of the folders listed in PATH in order until it finds an executable file with the name you entered. The order is important, if two folders in your path contain files with the same names, it’s the files in the folders nearest the front of the list that will get executed.

Aside: notice that the folder . is not in the standard PATH on Unix or Linux. This means that executable files in the present working directory are not searched. This is different to Windows/DOS where . is included in the standard path. Not including . in the path is a security feature, it stops malicious versions of common commands dropped into folders being inadvertently executed.

You can of course still run executable files in the present working directory on Unix/Linux, but you need to be explicit about it by pre-fixing the command with ./, e.g.:

cd /bin
./pwd

The which command can be used to show you which file will be executed when you use a given command, e.g.

which bash

Asside: the location of common commands on the file system may seem random at first, but there is a logic to it.

Firstly, command regular users can run are usually in folders ending in bin (short for binary), while commands which require root/admin privileges are usually in folders ending in sbin.

Secondly, there is a hierarchy of importance:

  1. Core OS commands will be in /bin and /sbin. E.g. /bin/ls & /bin/bash, and /sbin/mount
  2. Commands that are supported as part of the OS, but not considered core are one step down the hierarchy in /usr/bin and /usr/bin. E.g. /usr/bin/man& /usr/bin/perl, and /usr/sbin/automount
  3. Finally, third-party commands tend to show up in two distinct sets of locations, /usr/local/bin and /usr/local/sbin and/or /opt/local/bin and /opt/local/sbin. E.g. MacPorts installs all it’s binaries in /opt/local/..., so when I install Image Magick via MacPorts the convert binary is installed to /opt/local/bin/convert.

Something people often find confusing is that many of the builtin commands are actually executable files, as can be demonstrated with which (which is itself a builtin):

which cd
which pwd

What makes these commands special is that BASH does not use PATH to figure out where they are, it maps to them directly, so, even if you delete your PATH, the builtins will continue to work.

In fact, lets do just that (in a safe way that won’t do any harm to your computer)!

export PATH=''

We have now blanked the PATH environment variable in our command shell – note that we have ONLY altered the copy of PATH stored in this one command shell – all other command shells, including any new ones opened in the future, are totally unaffected by this change.

cd ~/Desktop
pwd

But we can’t do things like:

ls -alh
which nano
nano testFile.txt

It’s not that the executable files have gone, or no longer work, it’s that our instance of BASH has lost the ability to find them because it’s PATH is blank. We can still run the executables by using their full paths, e.g.:

/bin/ls -alh
/usr/bin/which nano
/usr/bin/nano testFile.txt

Before we continue, lets restore our PATH to its normal value by closing this command shell and opening a new one.

When you get to the stage of writing your own scripts (or downloading other people’s scripts), you’ll probably want your scripts to run without needing to give the full paths to the scripts each time. As an example lets create a new folder in our home directory and create a simple script within it:

mkdir ~/myScripts
nano ~/myScripts/whereAmI

Enter the following content into the file whereAmI and save:

  1. #!/usr/bin/perl
  2.  
  3. print "Hi $ENV{USER}, you are currently in $ENV{PWD}\n";

Asside: in the last instalment we used a BASH shell script for our example, this time, for some variety, I’m using a Perl script, the language used has no baring on how all this works.

Then make the script executable, and test it:

chmod 755 ~/myScripts/whereAmI
~/myScripts/whereAmI

At the moment we have to enter the full path to whereAmI each time we want to use it, lets remedy that by adding our new myScripts folder to the end of our PATH:

export PATH="$PATH:$HOME/myScripts"

Note that we have to include the current value of PATH in the new value we set for PATH or we would be replacing the PATH rather than adding to it. This is a very common pit-fall, and the effect would be that all non-builtin commands apart from those in the one new folder would break. Note also that we used $HOME instead of ~ because you can’t use ~ in PATH.

Verify that PATH has been updated:

echo $PATH
which whereAmI

We can now use our script as a command without having to specify the full path:

whereAmI

Now, close your command shell, and open a new one, and try to use your script as a command again:

bart-imac2013:~ bart$ whereAmI
-bash: whereAmI: command not found
bart-imac2013:~ bart$ 

Why was the command not found? The answer is simply that the change we made to PATH in our previous shell’s environment vanished the moment we closed that shell. What we need to do is make a permanent change, and to do that we need to understand how BASH initialises it’s environment.

When SH or BASH are initialising they they start the environment building process by sourcing the file /etc/profile. The out-of-the-box content of this file will be determined by your choice of OS. On my Ubuntu server /etc/profile contains the following:

  1. # /etc/profile: system-wide .profile file for the Bourne shell (sh(1))
  2. # and Bourne compatible shells (bash(1), ksh(1), ash(1), ...).
  3.  
  4. if [ "$PS1" ]; then
  5.   if [ "$BASH" ] && [ "$BASH" != "/bin/sh" ]; then
  6.     # The file bash.bashrc already sets the default PS1.
  7.     # PS1='\h:\w\$ '
  8.     if [ -f /etc/bash.bashrc ]; then
  9.       . /etc/bash.bashrc
  10.     fi
  11.   else
  12.     if [ "`id -u`" -eq 0 ]; then
  13.       PS1='# '
  14.     else
  15.       PS1='$ '
  16.     fi
  17.   fi
  18. fi
  19.  
  20. # The default umask is now handled by pam_umask.
  21. # See pam_umask(8) and /etc/login.defs.
  22.  
  23. if [ -d /etc/profile.d ]; then
  24.   for i in /etc/profile.d/*.sh; do
  25.     if [ -r $i ]; then
  26.       . $i
  27.     fi
  28.   done
  29.   unset i
  30. fi

While OS X comes with a much shorter and easier to understand /etc/profile:

  1. # System-wide .profile for sh(1)
  2.  
  3. if [ -x /usr/libexec/path_helper ]; then
  4.     eval `/usr/libexec/path_helper -s`
  5. fi
  6.  
  7. if [ "${BASH-no}" != "no" ]; then
  8.     [ -r /etc/bashrc ] && . /etc/bashrc
  9. fi

In this series we are focusing on OS X, so we'll only look at how OS X initialises it's Environment in detail.

What the above OS X /etc/profile does is two things:

  1. assuming it exists and is executable, it loads the output of /usr/libexec/path_helper into it's environment
  2. if the process starting up is a BASH process (rather than an SH process), it executes /etc/bashrc

As you might guess from the name, path_helper is a utility for constructing the default path. You can run it yourself to see what it produces:

/usr/libexec/path_helper

If you're curious, you can learn how it builds the path by reading the relevant man page with: man path_helper. The skinny version is that it reads the system-wide default path from /etc/paths, and then adds any extra paths defined in files contained in the folder /etc/paths.d. To have a look at the default paths you can use:

cat /etc/paths
cat /etc/paths.d/*

(On a default OS X install the last command will fail because there are no files present in /etc/paths.d by default)

If we wanted to add our new scripts folder to the default path for all users on the system we could edit /etc/paths, or add a new file with the path or our scripts folder in /etc/paths.d, but don't do that! These system-level paths should only be used for system-level things, as we'll see shortly, there is a better way to make user-specific customisations.

For completeness, lets have a look at /etc/bashrc.

  1. # System-wide .bashrc file for interactive bash(1) shells.
  2. if [ -z "$PS1" ]; then
  3.    return
  4. fi
  5.  
  6. PS1='\h:\W \u\$ '
  7. # Make bash check its window size after a process completes
  8. shopt -s checkwinsize
  9. # Tell the terminal about the working directory at each prompt.
  10. if [ "$TERM_PROGRAM" == "Apple_Terminal" ] && [ -z "$INSIDE_EMACS" ]; then
  11.     update_terminal_cwd() {
  12.         # Identify the directory using a "file:" scheme URL,
  13.         # including the host name to disambiguate local vs.
  14.         # remote connections. Percent-escape spaces.
  15.     local SEARCH=' '
  16.     local REPLACE='%20'
  17.     local PWD_URL="file://$HOSTNAME${PWD//$SEARCH/$REPLACE}"
  18.     printf '\e]7;%s\a' "$PWD_URL"
  19.     }
  20.     PROMPT_COMMAND="update_terminal_cwd; $PROMPT_COMMAND"
  21. fi

What's going on here is mostly OS X-specific customisations to BASH. The Ubuntu equivalent to this file is /etc/bash.bashrc, and just like with /etc/profile, the contents of the file is completely different to what you get on OS X.

There is really only one line in this file that I want to draw your attention to, and then, only as a preview of the next instalment. The line in questions is:

  1. PS1='\h:\W \u\$ '

It looks like gobbledegook, but, it's actually the line that sets the format of the command prompt. \h is the host name, \W is the current folder, and \u the current user. You should recognise that as the format of the command prompt in you OS X Terminal windows. We'll look at this in more detail next time.

So far there are two files doing the customisation of BASH for us, /etc/profile and /etc/bashrc. These are both system files, and if you try to edit them as a regular user you'll find your don't have permission:

bart-imac2013:~ bart$ ls -l /etc/profile /etc/bashrc
-r--r--r--  1 root  wheel  745 10 Nov 18:55 /etc/bashrc
-r--r--r--  1 root  wheel  189 10 Nov 18:55 /etc/profile
bart-imac2013:~ bart$

It's with good reason that you don't have editing rights to these files - you could do serious damage to your system if you make a mistake in these files. Unless you really know what you are doing, never edit either of them!

The system level configuration files are only the first half of BASH's startup procedure, when a new BASH process has finished running those files, it moves on to a new phase where it checks the user's home directory for certain specially named files.

For reasons we won't go into now, if you're a Linux user the user-level file to create/edit is ~/.bashrc, while Mac users should create/edit ~/.bash_profile (if you really care about why there is a difference you can have a read of this short article).

So, any customisations we wish to make to BASH on our Macs should be made in ~/.bash_profile. Lets go ahead and set a custom PATH that includes the folder we created earlier:

nano ~/.bash_profile

Enter the following and save the file (BE CAREFUL TO GET IT RIGHT):

  1. # print warning message (leave out the echo lines if you prefer)
  2. echo "NOTE - applying customisations in ~/.bash_profile"
  3. echo "       If you make a mistake and need to remove the customisations"
  4. echo "       execute the following then restart your Terminal:"
  5. echo "       /bin/mv ~/.bash_profile ~/bash_profile.disabled"
  6.  
  7. # update the path
  8. export PATH="$PATH:$HOME/myScripts"

Asside: Note that any line in a shell script starting with a # is a comment, so it is ignored by the computer and there purely for your information. Also, note that the echo lines are there only as a helpful hint in case you make a mistake and break your PATH. The command simply renames ~/.bash_profile to ~/bash_profile.disabled, hence disabling it, and, because the new name does not start with a ., making it visible in the Finder should you want to delete or edit it easily. You can test any changes you make to try fix what ever problem you were having by editing the file and then running:

source ~/bash_profile.disabled

When you're happy you've fixed the problem you can move it back into place with:

mv ~/bash_profile.disabled ~/.bash_profile

To test your newly customised environment simply open a new Terminal. If you've done everything right you should see the warning message telling you ~/.bash_profile has been executed, and, your path should have been updated to include ~/myScripts. You can verify this by running:

echo $PATH
whereAmI

You should use ~/.bash_profile to make all your BASH customisations, not just customisations to your PATH. In the next instalment we'll have a look at some of the other customisations you might like to configure in your ~/.bash_profile file.

Taming the TerminalNote: this blog post was written to accompany an audio segment on episode 449 of the NosillaCast Mac Podcast.

Given the times we live in, the word ‘environment’ probably invokes images of polar bears and melting ice, but the Al Gore definition of the word ‘environment’ is a relatively recent narrow definition of a much broader word. The first definition of the work in the OS X dictionary is:

The surroundings or conditions in which a person, animal, or plant lives or operates

In this instalment we’ll introduce a digital extension of this concept – the digital conditions within which a process exists, and specifically, in which a BASH command shell exists. Although this might sound like a simple topic, there’s actually a lot to cover, so we’ll be spreading it out over a few instalments.

Read more

keep looking »