The Apple/Mac community lost one of it’s finest podcasters today. Tim Verpoorten wasn’t the first Apple/Mac podcaster, but he was one of the very earliest generation. I think it would be fair to call him a father figure to many of us who followed. I know he was one of the podcasters who inspired me to pick up the microphone myself, and I doubt I’m alone in that.

Tim had been unwell for some time, and hung up his microphone to concentrate on his health a while ago, but we all hoped it would just be a temporary hiatus. I don’t think any of us in the community wanted to believe we’d heard the last of Tim’s distinctive and friendly voice.

Every good Apple/Mac podcast brings something unique to the table, and Tim’s Mac Review Cast brought fantastic reviews week after week after week for years and years. Tim had a knack for finding great apps, particularly free ones, and he was able to find and review them at a truly impressive rate. Most people can mange either quantity or quality, but Tim could do both at the same time. Although he reviewed many many apps, you could always tell when an app really appealed to him. Those apps were almost never large apps with lots of features, but small apps that did just one thing, but did it really well. It’s fair to say Tim had a bit of a thing for menubar apps.

Because I learned about so many great apps on the Mac Review Cast, I regularly look up at my menu bar, or into my dock, and think of Tim. One app in particular that I’ll always associate with him is the light-weight Mac-like text editor Smultron. I’d almost given up on finding an editor like this for the Mac, when I heard Tim review Smultron, and gave it a go. It was love at first sight, and that cute red strawberry icon will always bring back fond memories of Tim.

Tim was one of the founders of the Mac Round Table Podcast (MRT), and it was through that podcast that I was fortunate enough to get to ‘work’ (play more like) with Tim. One of the great things about the MRT is how different all the contributors are, and how that opens up some great conversations. We often agreed on things, but when it comes to temperament, I think myself and Tim were polar opposites – I’m know for being the cranky Irishman (sorta) who’s prone to impassioned (and hopefully entertaining) rants, while Tim was always as cool as a cucumber – I can’t remember him ever getting flapped, and I can’t remember him ever having a bad word to say about anyone. I think it’s much easier to go on a rant than it is to remain calm and collected, and I greatly admired Tim’s coolness.

I never met Tim in the real world, yet I feel I’ve lost a friend. The Mac community has certainly lost one of it’s finest ambassadors, but my thoughts are with the Verpoorten family tonight – their loss is so much greater than ours.

The photo that accompanies this post is a crop from this image by Allison Sheridan.

In the first installment we started with the 40,000ft view, looking at what command shells are, and why they’re still relevant in today’s GUI-dominated world. In the second instalment we looked at OS X’s Terminal.app, the anatomy of the Bash command prompt, and the anatomy of a Unix/Linux command. This time we’ll be looking at the anatomy of file systems in general, and the Unix/Linux file system in particular, and how it differs from the Windows/DOS file system many of us grew up using.

Physical storage media are nothing more than a massive array of virtual pigeon holes, each of which can hold a single 1 or 0. All your information is stored by grouping together a whole bunch of these pigeon holes and giving that grouping of 1s and 0s some kind of name. Humans simply could not deal with remembering that the essay they were working on is stored in sectors 4 to 1024 on cylinder 213 on the disk connected to the first SATA channel on the motherboard. We need some kind of abstraction to bring order to the chaos and to allow us to organise our data in a human-friendly way.

A good analogy would be a pre-computer office where the unit of storage was a single sheet of paper. Without some sort of logical system for organising all this paper no one would ever be able to find anything, hence, in the real world we developed ‘systems’ for ‘filing’ paper. Or, to put it another way, we invented physical filesystems, based around different ways of grouping and naming the pieces of paper. If a single document contained so much information that it ran over multiple pages, those piece of paper were physically attached to each other using a tie, a paperclip, or a staple. To be able to recognise a given document at a glance, documents were given titles. Related documents were then put together into holders that, for some reason were generally green, and those holders were then placed into cabinets with rails designed to hold the green holders in an organised way. I.e. we had filing cabinets containing folders which contained files. The exact organisation of the files and folders were up to the individual clerks who managed the data, and were dependant on the kind of data being stored. Doctors tend to store files alphabetically by surname, while libraries love the Dewey Decimal system.

When it comes to computers, the job of bringing order to the chaos falls to our operating systems. We call the many different schemes that have been devised to provide that order filesystems. Some filesystems are media dependent, while others are operating system dependent. E.g. the Joliet file system is used on CDs and DVDs regardless of OS, while FAT and NTFS are Windows filesystems, EXT is a family of Linux file systems, and HFS+ is a Mac file system.

There are an infinite number of possible ways computer scientists could have chosen to bring order to the chaos of bits on our various media, but, as is often the case, a single real-world analogy was settled on by just about all operating system authors. Whether you use Linux, Windows, or OS X, you live in a world of filesystems that contain folders (AKA directories) that contain files and folders. Each folder and file in this recursive hierarchical structure has a name, so it allows us humans to keep our digital documents organised in a way that we can get our heads around. Although all our modern filesystems have their own little quirks under the hood, they all share the same simple architecture, your data goes in files which go in folders which can go in other folders which eventually go into file systems.

You can have lots of files with the same name in this kind of file system, but, you can never have two items with the same name in the same folder. This means that each file and folder can be uniquely identified by listing all the folders you pass to get from the so-called ‘root’ of the filesystem as far as the file or folder you are describing. This is what we call the full path to a file or folder. On all modern consumer operating systems we write file paths as a list of folder and file names separated by some character, called the ‘path separator’. Where operating systems diverge is in their choice of separator, and in the rules they impose on file and folder names. DOS and Windows use \ (the backslash) as the path separator, on classic MacOS it was : (old OS X apps that use Carbon instead of Cocoa still use : when showing file paths, iTunes did this up until the recent version 11!), and on Linux/Unix(including OS X), / (the forward-slash) is used.

A single floppy disk and a single CD or DVD contain a single file system to hold all the data on a given disk, but that’s not true for hard drives, thumb drives, or networks. When formatting our hard drives or thumb drives we can choose to sub-divide a single physical device into multiple so-called partitions, each of which will then contain a single filesystem.

You’ve probably guessed by now that on our modern computers we tend to have more than one filesystem. Even if we only have one internal hard disk in our computer that has been formatted to have only a single partition, every CD, DVD, or thumb drive we own contains a filesystem, and, each network share we connect to is seen by our OS as yet another file system. In fact, we can even choose to store an entire filesystem (even an encrypted one) in a single file, e.g. DMG files, or TrueCrypt vaults.

So, all operating systems have to merge lots of file systems into a single over-arching namespace for their users. Or, put another way, even if two files have identical paths on two filesystems mounted by the OS at the same time, there has to be a way to distinguish them from each other. There are lots of different ways you could combine multiple filesystems into a single unified namespace, and this is where the DOS/Windows designers parted ways with the Unix/Linux folks. Microsoft combines multiple file systems together in a very different way to Unix/Linux/OS X.

Lets start by looking at the approach Microsoft chose to. In DOS, and later Windows, each filesystem is presented to the user as a separate island of data named with a single letter, referred to as a drive letter. THis approaches has an obvious limitation, you can only have 26 file systems in use at any one time! For historical reasons, A:\ and B:\ were reserved for floppy drives, so, the first partition on the hard drive connected to the first IDE/SATA bus on the motherboard is given the drive letter C:\, the second one D:\ and so on. When ever you plug in a USB thumb drive or a memory card from a camera it gets ‘mounted’ on the next free drive letter. Network shares also get mounted to drive letters.

Just like files and folders, filesystems themselves have names too, often referred to as Volume Names. Windows makes very little use of these volume names though, they don’t show up in file paths, but, Windows Explorer will show them in some situations to help you figure out which of your USB hard drives ended up as K:\ today.

An analogy you can use for file systems is that of a tree. The trunk of the tree is the base of the file system, each branch is a folder, and each leaf a file. Branches ‘contain’ branches and leaves, just like folders contain folders and files. If you bring that analogy to Microsoft’s way of handling filesystems, then the global namespace is not a single tree, but a small copse of between 1 and 26 trees, each a separate entity, and each named with a single letter.

If we continue this analogy, Linux/Unix doesn’t plant a little copse of separate trees DOS/Windows does, instead they construct one massive franken-tree by grafting smaller trees onto the branches of a single all-containing master tree. When Linux/Unix boots, one filesystem is considered to be the main filesystem, and used as the master file system into which other file systems get inserted as folders. In OS X parlance, we call the partition containing this master file system the System Disk. Because the system disk becomes the root of the entire filesystem it is gets assigned the shortest possible file path, /.

If your system disk’s file system contained just two folders, folder_1 and folder_2, they would get the file paths /folder_1/ and /folder_2/ in Linux/Unix/OS X. The Unix/Linux command mount can then be used to ‘graft’ filesystems into the master filesystem using any empty folder as the so-called mount point.

On Linux systems it’s common practice to keep home folders on a separate partition, and to then mount that separate partition’s file system as /home/. This means that the main filesystem has an empty folder in it called home, and that as the computer boots, the OS mounts a specified partition’s file system into that folder. A folder at the root of the that partition’s file system called just allison would then become /home/allison/.

On regular Linux/Unix distributions the file /etc/fstab (file system table) tells the OS what filesystems to mount to what mount points. A basic version of this file will be created by the installer, but in the past, when ever you added a new disk to a Linux/Unix system you had to manually edit this file. Thankfully, we now have something called automount to automatically mount any readable filesystems to a pre-defined location on the filesystem when they are connected.

The exact details will change from OS to OS, but on Ubuntu, the folder /media/ is used to hold mount points for any file system you connect to the computer. Unlike Windows, most Linux/Unix systems make use of filesystems’ volume names, and use them to give the mount points sensible names, rather than random letters. If I connect a USB drive containing a single partition with a filesystem with the volume name Allison_Pen_Drive, Ubuntu will automatically mount the filesystem on that thumb drive when you plug it in, using the mount point /media/Allison_Pen_Drive/. If that pen drive contained a single folder called myFolder containing a single file called myFile.txt, then myFile.txt would be added to the filesystem as /media/Allison_Pen_Driver/myFolder/MyFile.txt.

Having the ability to mount any filesystem as any folder within a single master filesystem allows you to easily separate different parts of your OS across different drives. This is very useful if you are a Linux/Unix sysadmin or power user, but it can really confuse regular users. Because of this, OS X took a simpler route. There is no /etc/fstab (actually, there sort of is, but it just contains some text telling you that the file does nothing and not to use it – for a LOL try typing cat /etc/fstab.hd into a terminal). The OS X installer does not allow you to split OS X over multiple partitions, everything belonging to the OS X system, including all the users home folders, are installed on a single partition, the system disk, and all other file systems, be they internal, external, network, or disk images, get automatically mounted in /Volumes/ as folders named for the file systems’ volume labels.

Going back to our imaginary thumb drive called Allison_Pen_Drive (which Ubuntu would mount as /media/Allison_Pen_Drive/), OS X will mount that as /Volumes/Allison_Pen_Drive/ when you plug it in. If you had a second partition, or a second internal drive, called, say, Fatso (a little in-joke for Allison), OS X would mount that as /Volumes/Fatso/. Likewise, if you double-clicked on a DMG file you downloaded from the net, say with the Adium installer, OS X would mount that as something like /Volumes/Adium/ until you eject the DMG. The ‘disks’ listed in the Finder side bar in the section headed Devices are just links to the contents of /Volumes/. You can see this for yourself by opening a Finder Window and either hitting the key-combo cmd+shift+g, or navigating to Go→Go To Folder ... in the menubar to bring up the Go To Folder text box, and then typing the path /Volumes and hitting return.

OS X’s greatly simplified handling of mount points definitely makes OS X less confusing, but, the simplicity comes at a price. If you DO want to do more complicated things like have your home folders on a separate partition, you are stepping outside of what Apple consider the norm, and into a world of pain. On Linux/Unix separating out home folders is trivial, on OS X it’s a mine-field!

We’ll leave it here for now, next time we’ll learn how to navigate around a Unix/Linux/OS X filesystem.

Last time I focused on the shell, and avoided getting in any way specific about the actual commands that we will be executing within the Bash shell. I thought it was very important to make as clear a distinction between command shells and commands as possible, so I split the two concepts into two separate segments. Having focused on command shells last time, this instalment will focus on the anatomy of a command, but will start with a quick intro to the Terminal app in OS X first.

Introducing the Terminal Window

You’ll find Terminal.app in /Applications/Utilities. Unless you’ve changed some of the default settings (or are using a very old version of OS X), you will now see a white window that is running the bash command shell that looks something like this:

Lets start at the very top of the window with its title bar. At the left of the title is a proxy icon representing the current directory for the current Bash command shell, and beside it the name of that folder. (Note that directory is the Unix/Linux/DOS word for what OS X and Windows refers to as a folder.) Like Finder windows, Terminal sessions are always “in” a particular directory/folder. After the current directory will be a dash, followed by the name of the process currently running in the Terminal session (in our case, a bash shell). The current process is followed by another dash, and then the dimensions of the window in fixed-width characters.

Within the window itself you will likely see a line of output telling you when you last logged in, and from where (if it was on this computer it will say ttys followed by some number, if it was from another computer, it will give the computer’s hostname). This will be followed on the next line by the so-called command prompt, and then the input cursor.

Lets have a closer look at the command prompt. As with almost everything in Linux/Unix, the prompt is entirely customisable, so although Bash is the default shell on lots of different operating systems, the default prompt on each of those systems can be different. Lets first look at the default Bash command prompt on OS X:

On OS X, the prompt takes the following form:

hostname:current_directory username$

First you have the hostname of the computer on which the command shell is running (defined in System Preferences → Sharing → Computer Name). This might seem superfluous, but it becomes exceptionally useful once you start using ssh to log in to other computers via the Terminal.

The hostname is followed by a : and then the command shell’s current directory (note that ~ is short-hand for “the current user’s home folder”, more on this next time).

The current directory is followed by a space, and then the Unix username of the user running the command shell (defined when you created your OS X account, defaults to your first name if available). Finally, there is a $ character (which changes to a # when you run bash as the root user). Again, this might not seem very useful at first, but there are many reason you may want to switch your command shell to run as a different user from time to time, so it is also very useful information.

As an example of how the default shells differ on different operating systems, below is an example from a RedHat-style Linux distribution (CentOS in this case):

As you can see, it contains the same information, but arranged a little differently:

[username@hostname current_directory]$

Finally, Debian-style Linux distributions like Ubuntu use a different default promote again, but also showing the same information:

username@hostname:current_directory$

Handy Tip: if you find the text in the Terminal window to small to read, you can make it bigger or smaller with cmd+ or cmd-. This will affect just your current Terminal window. You can permanently change the default by editing the default profile in Terminal → Preferences ... → Settings

The Anatomy of a Command

Now that we understand the different parts of our Terminal window, lets have a look at the structure of the actual commands we will be typing at that cursor!

I want to start by stressing that the commands executed by a command shell are not determined by the command shell, but by the operating system. Regardless of whether you use Bash on OS X, or zsh on OS X, you will have to enter OS X commands. Similarly, if you use Bash on Linux, you will have to enter Linux commands. Thankfully Linux and Unix agree almost entirely on the structure of their basic commands, so with a very few (and very annoying) exceptions, you can use the same basic commands on any Linux or Unix distribution (remember that at its heart OS X is Free BSD Unix).

Commands take the form of the command itself optionally followed by a list of arguments separated by spaces i.e.:

command argument_1 argument_2 ... argument_n

Arguments are a mechanism for passing information to a command. Most commands need at least one argument to be able to perform their task, but some don’t. Both commands and arguments are case-sensitive, so beware your capitalisation!

For example, the cd (change directory) command takes one argument (a directory path):

bart-imac:~ bart$ cd /Users/Shared/
bart-imac:Shared bart$

In this example, the command is cd, and the one argument passed is /Users/Shared/.

Some commands don’t require any arguments at all, e.g. the pwd (present working directory) command:

bart-imac:~ bart$ pwd
/Users/bart
bart-imac:~ bart$

It is up to each command to determine how it will process the arguments it is given. When the developer was creating the command he or she will have had to make decisions about what arguments are compulsory, what arguments are optional, and how to parse the list of arguments the command is given by the shell when being executed.

In theory every developer could come up with their own mad complex scheme for parsing argument lists, but in reality most developers loath re-inventing the wheel (thank goodness), so a small number of standard libraries have come into use for parsing arguments. This means that many apps use very similar argument styles.

As well as accepting simple arguments like the cd command above, many apps accept specially formatted arguments referred to as flags. Flags are usually used to specify optional extra information, with information that is required taken as simple arguments. Flags are arguments (or pairs of arguments) that start with the - symbol.

The simplest kinds of flags are those that don’t take a value, they are specified using a single argument consisting of a - sign followed by a single letter. For example, the ls (list directory) command can accept the flag -l (long form listing) as an argument. e.g.

bart-imac:Shared bart$ ls -l
total 632
drwxrwxrwx  3 root   wheel     102  5 Dec  2010 Adobe
drwxrwxrwx  3 bart   wheel     102 27 Mar  2012 Library
drwxrwxrwx@ 5 bart   wheel     170 28 Dec 21:24 SC Info
drwxr-xr-x  4 bart   wheel     136 22 Feb 21:42 cfx collagepro
bart-imac:Shared bart$

The way the standard argument processing libraries work, flags can generally be specified in an arbitrary order. The ls command also accepts the flag -a (list all), so the following are both valid and equivalent:

bart-imac:Shared bart$ ls -l -a

and

bart-imac:Shared bart$ ls -a -l

The standard libraries also allow flags that don’t specify values to be compressed into a single argument like so:

bart-imac:Shared bart$ ls -al

Sometimes flags need to accept a value, in which case the flag stretches over two arguments which have to be contiguous. For example, the ssh (secure shell) command allows the port to be used for the connection to be specified with the -p flag, and the username to connect as with the -l flag, e.g.:

bart-imac:Shared bart$ ssh bw-server.localdomain -l bart -p 443

These single-letter flags works great for simple commands that don’t have too many options, but more complex commands often support many tens of optional flags. For that reason another commonly used argument processing library came into use that accepts long-form flags that start with a -- instead of a single -. As well as allowing a command to support more flags, these longer form flags also allow values to be set within a single argument by using the = sign.

As an example, the mysql command (needs to be installed separately on OS X) allows the username and password to be used when making a database connection to be specified using long-form flags:

...$ mysql --username=bart --password=open123 example_database

Many commands support both long and short form arguments, and they can be used together, e.g.:

...$ mysql --username=bart --password=open123 example_database -v

So far we know that commands consist of a command optionally followed by a list of arguments separated by spaces, and that many Unix/Linux commands use similar schemes for processing arguments where arguments starting with - or -- are treated in a special way, and referred to as flags. That all seems very simple, but, there is one important complication that we have to address before finishing up for this segment, and that's special characters.

Within Bash (and indeed every other command shell), there are some characters that have a special meaning, so they cannot be used in commands or arguments without signifying to the command shell in some way that is should interpret these symbols as literal symbols, and not as representations of some sort of special value or function.

The most obvious example from what we have learned today is the space character, it is used as the separator between commands and the argument list that follows, and within that argument list as the separator between individual arguments. What if we want to pass some text that contains a space to a command as an argument? This happens a lot because spaces are valid characters within file and folder names on Unix and Linux, and file and folder names are often passed as arguments.

As well as the space there are other symbols that have special meanings. I won’t explain what they mean today, but I will list them:

• space
• #
• ;
• "
• '
• `
• \
• !
• $
• (
• )
• &
• <
• >
• |

You have two choices for how you deal with these special characters when you need to include them within an argument, you can escape each individual special character within the argument, or you can quote the entire argument.

Escaping is easy, you simply pre-fix the special character in question with a \. If there are only one or two special characters in an argument this is the simplest and easiest solution. But, it can become tedious if there are many such special characters.

Lets use the echo command to illustrate escaping. The echo command simply prints out the input it receives. The follow example passes the phrase Hello World! to the echo command as a single argument. Note that this phrase contains two special characters that will need to be escaped, the space and the !:

bart-imac:~ bart$ echo Hello\ World\!
Hello World!
bart-imac:~ bart$

If you don’t want to escape each special character in an argument, you can quote the argument by prepending and appending either a " or a ' symbol to it. There is a subtle difference between using ' or ".

When you quote with ' you are doing so-called full quoting, every special character can be used inside a full quote, but, it is impossible to use a ' character inside a fully quoted argument. For example:

bart-imac:~ bart$ echo '# ;"`\!$()&<>|'
# ;"`\!$()&<>|
bart-imac:~ bart$

When you quote with " on the other hand you are doing so-called partial quoting, which means you can use most special characters without escaping them, but not all. Partial quoting will become very important later when we start to use variables and things, because the biggest difference between full and partial quoting is that you can’t use variable substitution with full quoting, but you can with partial quoting (don’t worry if that makes no sense at the moment, it will later in the series).

When using partial quoting you still have to escape the following special characters:

• "
• `
• \
• $

For example:

bart-imac:~ bart$ echo "# ;\!()&<>|"
# ;\!()&<>|
bart-imac:~ bart$

and:

bart-imac:~ bart$ echo "\\ \$ \" \`"
\ $ " `
bart-imac:~ bart$

There are a few other peculiar edge cases with partial quoting – for example, you can’t end a partial quote with a !, and you can’t quote just a * on its own (there may well be more edge cases I haven’t bumped into yet).

That’s where we’ll leave it for this segment. We’ve now familiarised ourselves with the OS X Terminal window, and we’ve described the anatomy of a Unix/Linux command. In the next segment we’ll look at the Unix/Linux file system, and at some of the commands used to navigate around it.

In the last CCATP we did a very detailed segment on email security, and despite the fact that with the benefit of hind-sight I realise it was too much to do at once and should have been split into two segments, it received the strongest listener response of anything of any of my many contributions to the NosillaCast in the last 5 or more years. I hope I’m right in interpreting that as evidence that there are a lot of NosillaCast listeners who want to get a little more technical, and get their hands dirty with some good old-fashioned nerdery!

In this first segment I just want to lay a very basic foundation. I plan to take things very slowly with this series, so I’m going to start the way I mean to continue. Lets start with some history and some wider context.

Before the days of GUIs (Graphical User Interfaces), and even before the days of simple menu-driven non-graphical interfaces like the original Word Perfect on DOS, the way humans interacted with computers was through a “command shell”. Computers couldn’t (and still can’t) speak or interpret human languages properly, and humans find it very hard to speak in native computer languages like binary machine codes (though early programers did actually have to do that), so new languages were invented to help bridge the gap, and allow humans and computers to meet somewhere in the middle. The really big problem is that computers have absolutely no intelligence, so they can’t deal with ambiguity at all. Command shells use commands that look Englishy, but they have very rigid structures (or grammars if you will) that remove all ambiguity. It’s this rigid structure that allows the automated translation from shell commands to binary machine code the computer can execute.

When using any command shell, the single most important thing to remember is that computers are absolutely stupid, so they will do EXACTLY what you tell them to, no matter how silly the command you give them. If you tell a computer to delete all the files on a hard drive, it will, because, well, that’s what you asked it to do! Another important effect of the computers’ total lack of intelligence is that there is no such thing as “close enough” – if you give it a command that’s nearly valid, a computer can no more execute it than if you’d just mashed the keyboard with your face. Nearly right and absolute gibberish are just as unintelligible to a computer. You must be exact, and you must be explicit at all times.

No wonder we went on to invent the GUI, this command shell malarky sounds really complicated! There is no doubt that if the GUI hadn’t been invented the personal computer wouldn’t have taken off like it has. If it wasn’t for the GUI there’s no way there would be more computers than people in my parents home (two of them, and including iPhones and tablets, 6 computers)! Even the nerdiest of nerds use GUI operating systems most of the time, because they make a lot of things both easier and more pleasant. BUT – not everything.

We all know that a picture says a thousand words, but when you are using a GUI it’s the computer that is showing you a picture, all you get to do is make crude hand gestures at the computer, which I’d say is worth about a thousandth of a word – so, a single shell command can easily be worth a thousand clicks. This is why all desktop OSes still have command shells built in – not as their ONLY user interface like in times past, but as a window within the GUI environment that lets you communicate with the computer using the power of a command shell. Even Microsoft understands the power of the command shell, DOS may be dead, but the new Windows Power Shell is giving Windows power users a new, more modern, and more powerful command shell than ever before. Windows 8 may have removed the Start menu, but Powershell is still there! All Linux and Unix distros have command shells, and OS X gives you access to an array of different Command Shells through Terminal.app.

Just like there is no one GUI interface, there is no one command shell. Also, just like most GUIs are at least somewhat similar to each other, they all use icons for example, most command shells are also quite similar, having a command prompt that accepts commands with arguments to supply input to the commands or alter their behaviour. OS X does not ship with one command shell, it ships with SIX (sh, bash, zsh, csh, tcsh, and ksh)!

You can see the list of available shells (and set your default shell) by opening System Preferences, going to the Users & Groups pref pane, unlocking it, then right-clicking on your username in the side bar, and selecting Advanced Options ...:

The default shell on OS X is the Bourne Again Shell (bash), so that’s the shell we’ll be using for the remainder of this series. If you’ve not altered the defaults, then when ever you open a Terminal window on OS X, what you’re presented with is a bash command shell. Bash is an updated and improved version of the older Bourne Shell (sh), which was the default shell in Unix for many years. The original Bourne Shell dates all the way back to 1977, and is called after it’s creator, Stephen Bourne. The Bourne Again Shell is a ‘newer’ update to the Bourne Shell dating back to 1989, the name being a nerdy religious joke by it’s author, Brian Fox. The Bourne Again Shell was not the last in the line of shells tracing their origins to the Bourne Shell, there is also zsh which dates back to 1990, but despite being a more powerful shell, it hasn’t taken off like bash has.

So, what does a shell do? Does it just let you enter a single command and then run it? Or is there more to it? Unsurprisingly, there’s a lot more to it! The shell does it’s own pre-processing before issuing a command for you, so a lot of things that we think of as being part of how a command works, are actually features provided by the command shell. The best example is the almost ubiquitous * symbol. When you issue a command like chmod 755 *.php, the actual command is not given *.php as a single argument that it must then interpret, no, the * is interpreted and processed by the shell before being passed on to the chmod command. It’s the shell that goes and looks for all the files in the current folder that end in .php, and replaces the *.php bit of the command with a list of all the actual files that end in .php, and passes that list on to the chmod command.

As well as providing wildcard substitution (the * thing), almost all shells also provide ‘plumbing’ for routing command inputs and outputs between commands and files, the definition of variables to allow sequences of commands to be generalised in a more re-usable way, simple programming constructs to enable conditional actions, looping, and the grouping of sequences of commands into named functions, and the execution of a sequence of commands inside a file (scripting), and much more. Different shells also provide their own custom features to help make life at the command prompt easier for users – my favourite is tab-completion which is the single best thing bash has to offer over sh IMO. OS X also brings some unique features to the table, with superb integration between the GUI and the command shell through features like drag-and-drop support in Terminal.app and shell scripting support Automator. Of all the OSes I’ve used, OS X is the one that makes it the easiest to integrate command line programs into the GUI.

I’ll end today by explaining an important part of the Unix philosophy, a part that’s still very much alive and well within OS X today – Unix aims to provide a suite of many simple command line tools that each do just one thing, but do it very well – complex tasks can then be achieved by chaining these simple commands together using a powerful command shell. Each Unix/Linux command line program can be seen as a lego brick – not all that exciting on their own, but using a bunch of them, you can build fantastic things! My hope for this series is to help readers and listeners like you develop the skills to build your own fantastic things to make your computing lives easier. Ultimately the goal is to help you create more and better things by using automation to free you from as many of your repetitive tasks as possible!

I’ve blogged about the tyranny of free before, so I don’t want to focus on that today, instead I want to take a step back and talk about the importance of following the money.

When it comes to software we generally don’t tend to get locked in anymore these days. Not even Office and Photoshop lock you in now. If we are using an app that suddenly goes out of existence, or changes in some way that turns us off, we are almost always free to switch to an alternative app now. So, understanding the finances behind most of the apps we use is not that important. There is one very important caveat though – apps that act as interfaces to cloud services are very different beasts, because we are using those apps to share our data with a company or organisation.

The really important thing is understanding the financial underpinnings of the cloud services we use. Cloud services are entrusted with a lot of our information, and we can very easily find ourselves locked into them. This is why I think it’s very important to understand how a service is supposed to make money before starting to use it. In some rare cases like wikipedia, online services are run by non-profit charitable organisations, but in general, services are run by companies, and companies exist to make money. Some people like to fantasise that companies are altruistic beings that are there to serve their admirers, but those are the ravings of fanboys. All companies exist to make money. Apple, Microsoft, FaceBook, Google, Amazon, Evernote, DropBox, Yahoo, Instapaper …

While all companies exist to make money, all companies are not the same, because there are many different strategies for making money, and those different approaches affect users in very different ways. You can take it for granted that companies exist to make money, what you need to concern yourself with is how those companies make their money, because that’s what will determine the incentives acting on those companies, and affect how they see you and your data.

As strange as it sounds, the first question you have to ask about a company is whether or not they have a business model at all! You’d imagine that no company could get funding without knowing how it will turn a profit some day, but you’d be wrong. The digital equivalents of South Park’s underpants gnomes are hard at work in silicone valley! Step 1 – get lots of users, step 3 profit!

When you become dependant on a service that’s run by digital underpants gnomes you’re likely to end up suffering in one of two ways. Either, the service collapses because running losses year after year after year is actually a really bad way to run a business. Or, the service changes dramatically once it has attracted enough users to be able to sell them. Sometimes that takes the form of the entire company being sold, e.g. Instagram, and sometimes it takes the form of the company pivoting in some way to convert their users into profit themselves.

Instagram is a good recent example of the first way digital underpants gnomes can convert users into profit, and for me Twitter is the ultimate example of the second. Users, and particularly third party app developers, played a major role in making Twitter the success it is today. Many of the things we consider core features of Twitter were actually developed by third party developers, not Twitter themselves. The two most striking examples are probably the word ‘tweet’ (Twitter called them ‘twits’ originally), and the concept of @replies. Now, Twitter are pivoting to try sell their user base to advertisers. Originally, they had no customers, now advertisers are their customers. Twitter users are not Twitter’s customers, they are Twitter’s product, and the third party developers that played a large part in making Twitter are now a problem that needs to be death with, hence the crushing blows against third part app developers over the last year. As a Twitter user who has relied on a thriving third-party client ecosystem to get a good Twitter experience, I am suffering from Twitter’s pivot. Had I understood what I do now about the dangers of digital underpants gnomes I’d never have joined Twitter in the first place.

Assuming the company you’re evaluating does have a business model, the next key question is who their customers are. The primary concern of any company is to keep their customers happy and coming back for more. It’s in a company’s interest to do the things that their customers like, and it’s against a company’s interests to upset their customers.

When the users who use a service are also the customers, things are pretty simple – companies are incentivised to do right by their users. When users are not the customers though, then their interests are inevitably secondary to the interests of the customers, and when the interests of the customers are at odds with the interests of users, then the customers win and users lose out.

Because, for what ever reason, many people in our society want something for nothing, many services rely on advertising as their income stream. This sets up precisely the kind of perverse incentive I worry about. If users were customers then it would be in the company’s interest to protect their data, but since they are not, that’s not the case. But, it gets worse, because it is in the advertisers interests to get as much information as possible about the users they are paying to advertise at, so, companies are incentivised to give as much data about their users to their customers, the advertisement agencies, as they can get away with.

Ultimately, we are dealing with market forces here, and as anyone living in a capitalist society knows, market forces are very strong, so when they are working against you, that’s dangerous.

Perhaps it might be helpful to look at some real-world examples. Please bear in mind that these are my personal opinions, obviously I think I’m right, but I could of course be wrong.

1. FaceBook – it seldom gets more black and white than this IMO. FaceBook’s customers are advertisers! Recently the lines have gotten a little blurred when FaceBook started to treat some of their users like advertisers by charging to have posts ‘promoted’ (i.e shown to all your ‘friends’ and not just to a random sub-set of them), but, even still, it seems clear to me that the vast majority of FaceBook users not FaceBook’s customers, while advertisers clearly are. I think you can understand the way their terms of services have evolved over the years as the inevitable result of the perverse incentives created by having advertisers as their customers rather than their users.
2. Google are a little more complicated than FaceBook because they do a lot. They are best know as a search engine, but they are a big service provider too, with extensive cloud offerings, and they also make two operating systems, and a number of apps including a popular browser. They are also stating to directly monetize some of their web services by charging customers for their use. E.g., if you want Google Apps on your own domain, you have to pay now, the free tier has been removed. However, when it comes to making the big money, I think it’s still all about the ads.
3. Amazon – I find Amazon an interesting case – they clearly gather a lot of data about their users – they know everything you search for on their store, and everything you buy there, but they use that data differently to Google, they use it to figure out what else you might like to buy from them via their uncannily good suggestion engine. At first glance, you might think Amazon’s data gathering is no different to FaceBook’s, but I think it is, Amazon are not using our data as their product, they are using our data to sell, us, their customers, more of their products, the things they sell in their stores. I sometimes think of Amazon as the digital Wall Mart.
5. Apple are not clear-cut either. They sell hardware, they sell apps, they have a cloud service, and they sell ads. When you look at Apple’s earnings reports though, it’s clear that their primary customers are people who buy iThings, followed by people who buy media and software through Apple’s stores. iAds is really quite insignificant in the grand scheme of things. iCloud is also a very different cloud service to Google’s. There are no ads, it’s a freemium service to make iThings more attractive. I think Apple had no choice but to make a free iCloud tier because Google have a free cloud as do Microsoft. Google use their cloud as an ad platform though, while Apple don’t. On the whole, Apple primarily make their money by selling things to their users, so, Apple’s users are their main customers.
6. I find Microsoft very difficult to analyse at the moment. Historically they were very easy to understand, they made software, and they primarily sold it to corporate IT departments. You could see by their feature sets that corporate IT was who Microsoft focused on. Now It’s very hard to tell what Microsoft’s focus is. They still do the software thing, and they still sell to corporate IT, but, they are also trying to copy Google and become a search and advertisement company, and they are also starting to focus on end-users as well as corporate IT with the XBox and their new phones and tablets. The fact that ads are starting to infest XBox Live shows how complicated the incentives are. I’m not sure if even Microsoft know what their long-term strategy is, so it’s very hard to divine what incentives are driving Microsoft’s decisions, and how they will affect users going forward.
7. Evernote and DropBox both have, and have had for a long time, business models that work by selling a premium services to a small number of their users, while offering a free service to many more users to help support their ecosystem. Free users on Evernote and DropBox are not their products though, they are their market. The hope is not to sell free users’ data for profit, but to turn a few percent of those free users into paying customers. Those that never sign up are still adding value to the products because both consider easy sharing an important feature, and you can’t expect people to want to share with you if it will cost them! One of the reasons I chose to trust dropbox with my data is that I heard an interview with their CEO on a podcast where he explained their business model candidly I was impressed that even from the early days they knew where their incentives lay, and I thought they were healthy incentives.
8. Flickr, SmugMug and 500px also have a similar model to DropBox and EverNote, while Picassa by contrast has a very different model, and can be thought of as the GMail of photosharing.

The bottom line is that I’m not interested in telling people what companies or services to choose. Different people will make different decisions based on the same data because we all have different priorities. My hope is that people will stop, investigate, and think, before signing up to a service. It doesn’t matter what decision you make, as long as it’s an informed one – so follow the money before you invest your time and data into a cloud service! Oh, and watch out for those digital underpants gnomes too

]]> http://www.bartbusschots.ie/blog/?feed=rss2&p=2409 0 http://www.bartbusschots.ie/blog/?p=2393 http://www.bartbusschots.ie/blog/?p=2393#comments Sat, 29 Dec 2012 17:57:41 +0000 Bart Busschots http://www.bartbusschots.ie/blog/?p=2393 Back in 2011 I wrote a blog post explaining how to create an OS X Service for stripping keywords from image files. In this post we’ll use the same technique to create a Service for stripping geotags from JPEG images.

As with the keyword stripping service, there are two prerequisites for this action, one is required, one is optional. You absolutely MUST have install EXIFTool installed, and it would be good if you also had Growl installed, but it’s not essential.

Start by opening Automator, and creating a new Service workflow.



At the top of the new service workflow, alter the settings so the services takes as it’s input Image Files and that the service is available in just the Finder.



Find the Run Shell Script action and drag and drop it into the workflow (you’ll find the action in the Utilities category, or by searching in the search bar). Change the Shell option at the top of the Run Shell Script action to /usr/bin/perl, and the Pass input option to as arguments.

To do the actual work we will add the following perl code into the action:

perl
< view plain text >
1. foreach my $image (@ARGV) {
2.     $image = quote_file_name($image);
3.     `/usr/bin/exiftool -geotag= -overwrite_original $image`;
4. }
5.  
6. # utility function to sanitise file names for use when shelling out
7. sub quote_file_name{
8.     my $file = shift;
9.     $file =~ s/([`"\\\$])/\\$1/g;
10.     return '"'.$file.'"';
11. }

Finally, it would be very helpful to have the service tell us when it is finished, so this is where Growl comes in. If you have it installed, drag and drop a Show Growl Notification into the workflow and enter a message of your choice.

Your Service should now look something like the screenshot below. You can now go ahead and save the Service and give it a sensible name like Strip Geotag.



You should now be able to use the service by right-clicking on a file in the Finder and choosing your new service from the Services section. If you have only a few services installed they will be shown at the bottom of the contextual menu, if you have more, they will show in a sub-menu titled Services.



I’ve included a zipped version of my service below. This service uses the Growl action, so if you don’t have Growl installed you’ll need to open the Service in Automator and remove the Growl action before you can use the Service.

Download

• Strip Geotag Service Version 1 (Requires Growl and EXIFTool)
]]> http://www.bartbusschots.ie/blog/?feed=rss2&p=2393 1 http://www.bartbusschots.ie/blog/?p=2383 http://www.bartbusschots.ie/blog/?p=2383#comments Sat, 22 Dec 2012 16:36:37 +0000 Bart Busschots http://www.bartbusschots.ie/blog/?p=2383 This week’s Insgragram TOS kerfuffle is nothing new. Instagram is not the problem, it’s just the latest symptom of a sick business model that has been allowed to become so dominant as to be almost un-challengeable – services on the web MUST be free, so you MUST give up your privacy and/or your intellectual property rights to enable the service providers profits. If you dare stand up for privacy then you are a greedy idiot who wants something for nothing, and you need to grow up and let the companies make money.

My problem is not that companies want to make profits, it’s their instance on selling our data to do it that I have a problem with. How about this for an idea – why not let people pay for services rather than insisting we all whore out our privacy and intellectual property?

In the past, when you wanted to set up a company you started with a business plan, and then you went out looking for the money to get your business off the ground. Before you started you knew how you were going to make money. In general, the users of your service were your customers, and you made your money by selling your service to them.

The modern ‘web2.0′ approach is different – you have an idea, you set up a free service (which you bank roll yourself or with ‘Angel’ or ‘Venture’ capital), you suck in as many users as you can, and then you sell the company to someone else and make a big profit. The whole time you are losing money hand-over-fist. The more popular your free service, the more money you lose, but the more valuable your company becomes. Although it’s ostensibly the company that is being sold, what’s really up for sale is the users – we are the cash-crop of this web2.0 business model.

Even if a free startup decides not to sell, like for example Twitter, the company still has to make the switch from losing to making money, and the way that is done is almost always by exploiting the users.

In order to make your human crop as valuable as possible you need to collect as much personal information about them and their activities as you can, and you need to get them to sign over as many of their rights to you as you can. It’s this perverse incentive that drives the massive erosion of our privacy that we are seeing at the moment.

It’s these perverse incentives that the free business model creates that I have such a problem with. As many people have said, if you are not paying for a service, then you are the product. I don’t want to be anyone’s product, I want to be their customer!

It’s true that there are non-free choices for a lot of things online. This blog is not hosted on a free service like Blogger or Tumblr because I want full control over my work, and the information I share. Instead, this blog is hosted on a server I pay for, so I am the customer. My chosen photo network is Flickr, because I can pay them for a pro account and become their customer. Similarly, I have a paid Instapaper account for managing my reading lists, and I pay to use the App.net social network.

However, the tyranny of free still hangs over me. I use many free services because I feel I have to. I have a Twitter account and a G+ account, because as a podcaster you need to be on at least SOME popular social networking sites. I’ve chosen those two over the likes of FaceBook and LinkedIn because I find their terms and practices the least worrying, but I’d still love to be able to pay for those services so they could stop selling me on to third parties to make their profit. My videos are hosted on YouTube because that’s where the community is. I’m perpetually tempted to switch to Vimeo’s paid service, but that’s not where the Irish railway nerds hang out, and YouTube’s terms are not too odious.

The tyranny of free is also spreading into applications. More and more companies are offering free apps that they then fill with ads. Thankfully there are enough companies that let you pay to remove the ads that I have only one app that I use regularly that is constantly advertising at me. When choosing apps I will always choose the paid alternative over the free add-supported one if I can.

Everyone has their limits on what they will tolerate in exchange for a free service, which is why you see so many angry responses to terms of service that push the boundaries. Unfortunately, over time all our boundaries are slowly being pushed further and further out. Had FaceBook’s TOS been what it is today on their first day, people would just not have joined, but like a frog in boiling water we have all been slowly ground down and we accept more and more without a fight.

I absolutely do not like the future we are heading towards. By exploiting our collective greed and dangling free stuff in front of our faces corporations have managed to entice our society into giving up our privacy in ways even totalitarian governments have failed to do. The Stasi would have killed for Google’s database!

It just does not have to be this way!

Some people will insist on not paying for things, and many services, particularly social networks, require a critical mass of people to become useful, so free can’t go away. However, it can co-exist with paid services. The simplest way to do this is with a so-called ‘freemium’ model. Your free users get sold to advertisers, while your paying users do not, and perhaps also get a more feature-rich service. This is how both Flickr and Instapaper work, and I’m a happy user of both services.

I’ve always hoped Twitter would go the freemium route. Imagine a world where you could pay a few Euro a month for a pro Twitter account which would enable the use of 3rd party apps free of silly limitations. The reason Twitter are hell-bent on destroying the 3rd party apps that made their service a success is that they need control of the user experience so they can sell their users to advertisers. Why not let the users who want apps pay for the use of the service instead!?

Another very interesting development is App.net, this is a completely paid social network. There are no free App.net accounts, which makes it an exceptionally brave move on their part. I’m really hoping it takes off and proves that there are alternatives to free, even in the social networking space. Because App.net was a paid service from day 1 they did not anger anyone by having to transition from a free loss-maker to a paid profit maker.

I say it’s time to stand up and demand to be allowed to pay for services. Each time someone puts up the straw-man that we HAVE to give up our privacy and intellectual property because a service is free, knock it down with the simple alternative that we be allowed pay for the service.

Let us become customers, I’m fed up of being the product!

]]> http://www.bartbusschots.ie/blog/?feed=rss2&p=2383 2 http://www.bartbusschots.ie/blog/?p=2375 http://www.bartbusschots.ie/blog/?p=2375#comments Mon, 06 Aug 2012 15:46:02 +0000 Bart Busschots http://www.bartbusschots.ie/blog/?p=2375 This afternoon I updated www.xkpasswd.net to version 0.2.1 of the XKpasswd.pm Perl module, and added the needed UI to expose some of the new features introduced in version 0.2.* of the module:
1. It is now possible not to use any separator between the words that form the basis of your randomly generated password
2. The padding character can now be set to be randomly chosen, independently of the separator character. This is now the default setting, and provides more entropy by default.
3. An additional care transform has been added, you can now choose to have the capitalisation alternate on each subsequent word.
]]> http://www.bartbusschots.ie/blog/?feed=rss2&p=2375 2 http://www.bartbusschots.ie/blog/?p=2367 http://www.bartbusschots.ie/blog/?p=2367#comments Mon, 06 Aug 2012 15:32:45 +0000 Bart Busschots http://www.bartbusschots.ie/blog/?p=2367 This is a minor bug-fix update for XKpasswd (my Perl random password generation module). It squashes two minor bugs which came to light while updating www.xkpasswd.net to use version 2 of the module.
1. When the custom_separator option was left blank, no separator was used, rather than the expected random separator.
2. When the custom_separator option was left blank or set to RANDOM, and the pad_char option to SEPARATOR, the results were un-expected, different random character was used for each, rather than the same random character.

For documentation and detailed release notes on version 2 of the module, see the release notes for version 2.0.



]]> http://www.bartbusschots.ie/blog/?feed=rss2&p=2367 1 http://www.bartbusschots.ie/blog/?p=2357 http://www.bartbusschots.ie/blog/?p=2357#comments Sat, 04 Aug 2012 16:33:53 +0000 Bart Busschots http://www.bartbusschots.ie/blog/?p=2357 A few weeks ago on the Chit Chat Across the Pond segment of the Nosillacast, I mentioned that I had an OS X service set up to generate a random password using my XKpasswd Perl module and copy it to the clipboard. Listeners enquired as to how they would go about doing that, so as promised, here’s a quick tutorial.

Obviously this tutorial is for Mac OS X users only, because OS-wide Services and Automator are OS X features. The screenshots are taken on 10.8 Mountain Lion, but this same technique definitely also works on OSX 10.7 Lion, and probably even on 10.6 Snow Leopard. This tutorial also assumes that you have downloaded the XKpasswd module, and saved it somewhere on your computer, along with either the sample dictionary file included with the module or one of your own making, and that you know where on your computer those files have been saved. In other words, you need to have XKpasswd.pm and a text file with one word per line somewhere on your hard drive. In my sample code I’m going to assume you’ve installed the Perl module to the suggested location, /usr/local/xkpasswd/XKpasswd.pm, and that you have customised the sample dictionary a little (more secure that way), and saved it to /usr/local/xkpasswd/dict.txt.




Lets get started by opening up the Automator app (Applications→Automator), and creating a new file. Automator will ask you what type of document you want to create, choose Service.



At the top of this new blank document there are some options which control the input the service will expect, and which apps the service will be available in. Since we are generating the password from scratch, there is no input required at all, and we want this service to be available everywhere, so set the Service receives drop-down to no input, and leave the in drop down at the default any application as shown below:



Next we need to add a Run Shell Script action to the service. The easiest way to find this action in the list of actions is to type ‘run’ in the search box. When you find it, drag and drop it into the service.



In this new action, we need to change the Shell from the default value /bin/bash, to /usr/bin/perl, and then delete the default code that is added to the text box. We will be starting with a blank slate. (Since we are taking no input, we can ignore the Pass Input drop down completely this time.) By Default the text box in the action is very small, so you’ll probably want to enlarge it by grabbing the ‘handle’ on the bottom right corner and dragging it down a bit. Better still, you could open a programming editor like the fantastic and cheap Smultron, write your code there, and then copy and paste it back into automator when you’re done.



Now we have to write the Perl code. I’m going to follow my own advice and use Smultron. Why? Because that way I get syntax highlighting which makes typos much easier to spot. It will also make the script easier to debug if you’re having problems, by making it easy to run the Perl code in the Terminal. If you don’t have Smultron or a similar coding editor, you can use TextEdit provided you use it in Plain Text mode (Format->Make Plain Text).

In Smultron (or TextEdit), start by creating a new document, and saving it somewhere on your hard drive with the extension .pl, I’m going to call my script xkpasswdService.pl and save it in ~/Documents/temp.

We’re now ready to start writing some Perl code. If you’ve never programmed before, don’t worry, but pay very close attention, programming is a precise art, ', ", and ` are different symbols with different meanings, and capitalisation matters, And is not the same as and, and so on. You’re probably best off copying and and pasting the final code further down in this article and then tweaking it for your needs, rather than trying to write it all yourself from scratch. Alternatively, I’ve included the code a file called xkpasswdService.pl in the downloadable ZIP archive linked near the bottom of the article. Although I want you to copy and paste the code (from either this article or the file in the archive), I’m going to build up the code piece by pice so I can explain each section clearly.

Lets start the script with some good practice. If you learn nothing more about Perl today than these first two lines, you’ll have learned a lot. You CAN write Perl without them, but you’re just making life harder for yourself, especially as your scripts get larger and more complex. Think about writing Perl without strict and warnings as being like tightrope walking without a safety net – best avoided

perl
< view plain text >
1. use warnings;
2. use strict;

Next, we need to include the XKpasswd module. This is not a part of standard Perl, so we first need to tell Perl which folder the .pm file is in, only then we can include it. To tell Perl where to look, we use the use lib directive, and to actually use the module, we use the use directive. If you’ve installed XKpasswd.pm in a location other than /usr/local/xkpasswd/ you’ll need to adjust this line accordingly.

perl
< view plain text >
1. use lib '/usr/local/xkpasswd';
2. use XKpasswd;

The next step is to build up a variable with the configuration details to be used by the XKpasswd module while generating passwords. We’re going to use a type of variable called a hashref, and we’re going to call it $xkpasswd_config (in Perl all ‘scalar’ variable names begin with a $ symbol). A hashref is simply a good data structure for storing and passing around key-value pairs in Perl. In the case, the keys will the configuration options we would like the module to use when generating the passwords. I’m going to build up this variable in steps, but you don’t have to do that, you could do it all in one go. The reason I’m doing it in steps is to split the options that are compulsory from the ones that are optional. It’s important to note at this stage that in Perl, any line beginning with a # symbol is ignored by the Perl interpreter, and so does not make up part of the program. These lines are called comments, and are there purely for the benefit of humans, they are there to help us understand what the code is doing.

Aside – there’s an old programming joke that code comments are like sex, even bad comments are better than none at all

perl
< view plain text >
1. # build up the config for the XKpasswd object,
2. # starting with the required fields
3. my $xkpasswd_config = {
4.     dictionary_file => '/usr/local/xkpasswd/dict.txt',
5.     min_word_length => 4,
6.     max_word_length => 8,
7. };

The dictionary_file option specifies the location of the dictionary file the XKpasswd module should use when generating the passwords. If you’ve saved your dictionary file somewhere else, you’ll have to adjust this line accordingly. The min_word_length and max_word_length options specify the minimum and maximum length of the words from the specified dictionary that should be used when building the password. The way it works is that the dictionary is read in once, and all words not within the range specified by these two values (inclusively) will be ignored and not loaded by the module. Also note that the module will throw an error if it does not load at least 1000 matching words from your dictionary file (this is to ensure your random passwords really are random, or, in nerd speak, that they have sufficient entropy).

Next we start adding any further optional options we desire. All these options can be left out, and the module will just use default values for you. I’m only going to show some of the commonly used ones here, for a full list of supported options see the extensive comment starting at line 36 of XKpasswd.pm, or the sample file (example.pl) distributed with the module.

First, lets decide how many words we’d like our password to be built from, and what, if any, case transformations we’d like to see applied to those words. The valid values for the case transformation are NONE (leave the words as they appear in the dictionary file), CAPITAL (capitalises the first letter of every word), UPPER (convert all words to entirely upper case), LOWER (convert all words to entirely lower case), ALTERNATE (convert each alternate word to all upper case, then all lower case, then all upper case and so on), and finally, RANDOM (randomly convert each word to all upper or all lower case). If the option is not specified at all, the module defaults to applying no case transformations. Note that we must surround the value in the capitalisation variable with so-called single quote symbols.

perl
< view plain text >
1. # choose how many words to use, and
2. # how they should be capitalised
3. $xkpasswd_config->{num_words} = 3;
4. $xkpasswd_config->{case_transform} = 'CAPITAL';

Second, lets decide what character we will use to separate the words that will make up the bulk of the generated passwords. We can specify a single character, or, the special values RANDOM, for a randomly chosen character each time a password is generated, or NONE if we don’t want any separator character to be used. BTW – if this option is not specified, the module defaults to using a random character. In this case I’m going to use the character +. Note that, again, we have to surround the value for the separator option with single quote characters.

perl
< view plain text >
1. # specify the character to use to separate
2. # the parts of the password
3. $xkpasswd_config->{custom_separator} = '+';

Thirdly, lets specify how many random digits to include before and after the words that make up our password. The numbers before and after don’t have to be the same, and if you don’t want any digits, you can set the values to 0. I’m going to add no random digits before the words, and three after. The default is three before and three after.

perl
< view plain text >
1. # specify the number of random digits to add
2. # before and after the words in the password
3. $xkpasswd_config->{prepend_numbers} = 0;
4. $xkpasswd_config->{append_numbers} = 3;

Finally, lets configure the padding of the password. The XKpasswd module can pad out your password to a bigger size by adding one or more symbols before and/or after the words and digits that make up your password. By default a random symbol is chosen, and three copies of that symbol are added to the front and back of the password. In this example lets change that behaviour to using 5 exclamation marks at the beginning and the end of our passwords.

perl
< view plain text >
1. # specfify how the password should be padded
2. $xkpasswd_config->{pad_char} = '!';
3. $xkpasswd_config->{pre_pad} = 5;
4. $xkpasswd_config->{post_pad} = 5;

Putting it all together, at this stage our code should look like this:

perl
< view plain text >
1. use warnings;
2. use strict;
3. use lib '/usr/local/xkpasswd';
4. use XKpasswd;
5.  
6. # build up the config for the XKpasswd object,
7. # starting with the required fields
8. my $xkpasswd_config = {
9.     dictionary_file => '/usr/local/xkpasswd/dict.txt',
10.     min_word_length => 4,
11.     max_word_length => 8,
12. };
13.  
14. # choose how many words to use, and
15. # how they should be capitalised
16. $xkpasswd_config->{num_words} = 3;
17. $xkpasswd_config->{case_transform} = 'CAPITAL';
18.  
19. # specify the character to use to separate
20. # the parts of the password
21. $xkpasswd_config->{custom_separator} = '+';
22.  
23. # specify the number of random digits to add
24. # before and after the words in the password
25. $xkpasswd_config->{prepend_numbers} = 0;
26. $xkpasswd_config->{append_numbers} = 3;
27.  
28. # specfify how the password should be padded
29. $xkpasswd_config->{pad_char} = '!';
30. $xkpasswd_config->{pre_pad} = 5;
31. $xkpasswd_config->{post_pad} = 5;

However, we could compress it a little by making all the declarations in one go. This will make the code shorter, but the details comments will have to be sacrificed. It won’t make any functional difference to how the code works though, so you can leave it as-is if you prefer. I’m going to shorten it a little, because I find that easier to read and edit.

perl
< view plain text >
1. use warnings;
2. use strict;
3. use lib '/usr/local/xkpasswd';
4. use XKpasswd;
5.  
6. # build up the config for the XKpasswd object,
7. # starting with the required fields
8. my $xkpasswd_config = {
9.     dictionary_file => '/usr/local/xkpasswd/dict.txt',
10.     min_word_length => 4,
11.     max_word_length => 8,
12.     num_words => 3,
13.     case_transform => 'CAPITAL',
14.     custom_separator => '+',
15.     prepend_numbers => 0,
16.     append_numbers => 3,
17.     pad_char => '!',
18.     pre_pad => 5,
19.     post_pad => 5,
20. };

We are now ready to create an XKpasswd object, which can then be used to generate passwords as specified by the config we’ve assembled.

perl
< view plain text >
1. # instantiate an XKpasswd object with the above config
2. my $xkpasswd = XKpasswd->new($xkpasswd_config);

Finally, with the object created, we can generate a password, and print it. The way the Run Shell Script Automator Action works is that what ever the script prints out is used as the output of the action and passed on as the input to the next Action in the workflow. Note that we are using the word print in a software sense, not a hardware sense. We’re talking about printing to the standard output stream, not a piece of paper.

perl
< view plain text >
1. # generate a random password and print it
2. print $xkpasswd->generate_password();

Our final assembled code now looks like this:

perl
< view plain text >
1. use warnings;
2. use strict;
3. use lib '/usr/local/xkpasswd';
4. use XKpasswd;
5.  
6. # build up the config for the XKpasswd object,
7. # starting with the required fields
8. my $xkpasswd_config = {
9.     dictionary_file => '/usr/local/xkpasswd/dict.txt',
10.     min_word_length => 4,
11.     max_word_length => 8,
12.     num_words => 3,
13.     case_transform => 'CAPITAL',
14.     custom_separator => '+',
15.     prepend_numbers => 0,
16.     append_numbers => 3,
17.     pad_char => '!',
18.     pre_pad => 5,
19.     post_pad => 5,
20. };
21.  
22. # instantiate an XKpasswd object with the config we built
23. my $xkpasswd = XKpasswd->new($xkpasswd_config);
24.  
25. # generate a random password and print it
26. print $xkpasswd->generate_password();

Before copying and pasting this code into the Run Shell Script Action back in Automator, we should test it. To do that we’re going to make use of the so-called proxy icons (the little icons next to the file name in the window’s title bar) native OS X documenting editing apps like TextEdit and Smultron provide. Before trying to test your code, be sure to save your document! Start by opening a terminal window (Applications→Utilities→Terminal) and typing the word perl with a single space after it (DO NOT HIT ENTER YET) into the Terminal window. Then, drag and drop the proxy icon at the top of your TextEdit/Smultron window onto your Terminal window, this should copy the full path to your Perl script into the Terminal.



Once the path is in your Terminal window, you can go ahead and hit enter. If there are no errors in your code, a random password should be printed, and then you should get a fresh prompt directly after it on the same line (this is because we are not including a newline character when printing the password to keep automator happy). It should look something like this (I’ve underlined the password in red):



The two most likely errors you might encounter would be caused by getting one of the two paths wrong. If you get the include path for the XKpasswd module wrong you’ll see an error starting with Can't locate XKpasswd.pm in @INC (I’ve underlined the important part of the error in the screenshot below).



If, on the other hand, you made a mistake with the path to the dictionary file you’ll see an error that starts with Required config option dictionary_file is invalid (again, I’ve underlined the important part of the error in the screenshot below).



Aside – if you didn’t know about proxy icons, they’re one of OS X’s coolest power features – want to email someone a document you have open, just drag and drop the proxy icon onto the mail.app icon in your dock, want to go to a folder you have open in the Finder in the Terminal, just type cd followed by a space into the Terminal window, drag and drop the proxy icon from the Finder window onto the Terminal window, and hit enter, want to copy a document you have open to a thumb drive, just drag and drop the proxy icon onto the icon for the thumb drive on the desktop, or onto a Finder window showing the contents of the thumb drive etc..

When you’re happy that the script is working, you can copy and paste the code into the Run Shell Script Action back in Automator.

Next we need to add an Automator Action into our workflow to copy the value passed to it into the clipboard. Unsurprisingly, this action is called Copy to Clipboard. Again, the easiest way to find it is to search for it, and when you have found it you need to drag and drop it into your workflow below the Run Shell Script.



At this stage your action should look something like the screenshot below, and you can test it by clicking on the Run button in the toolbar on the top right of the Automator window. You can see the contents being returned by each action if you click the Results button on the bottom of the Actions. If all is well, a green tick should show next to each Action. If there’s an error, the details will be in the log, which will appear at the bottom of the window.



At this stage, you can save the Service, and start using it. However, I like to add an action to show a Growl notification when the password is ready. Particularly the first time you use a Service, it can take a few seconds to load and run, so it’s very handy to see some form of visual feedback when the password is ready for you in the clipboard. Obviously this will only work if you have Growl installed on your Mac. Sadly, as of OS X 10.8.0 there is no Automator action to send a message to the new Notification Centre in OS X Mountain Lion, which strikes me as an annoying oversight by Apple.

The downloadable ZIP below contains two copies of the service, one with Growl, and one without. To install them, just copy them to your ~/Library/Services folder. If you’re using Lion or Mountain Lion you can get to your Library folder by holding down the Option key when you click the Go menu in the Finder, and then selecting the Library option which appears.



Finally, if you’d like to add a keyboard shortcut for your new Service, you can do this using the Keyboard Shortcuts tab in the Keyboard applet in System Preferences. In here you’ll need to select Services in the left panel, and then scroll until you find your new XKpasswd service listed (probably at the bottom) in the right pane, select it, click the add shortcut button that appears, and then enter your chosen shortcut (I use ctrl+x).



]]> http://www.bartbusschots.ie/blog/?feed=rss2&p=2357 2