{"id":9277,"date":"2014-09-07T13:04:47","date_gmt":"2014-09-07T13:04:47","guid":{"rendered":"https:\/\/www.bartbusschots.ie\/s\/?p=9277"},"modified":"2018-07-02T09:13:01","modified_gmt":"2018-07-02T09:13:01","slug":"taming-the-terminal-part-20-of-n-file-searches","status":"publish","type":"post","link":"https:\/\/www.bartbusschots.ie\/s\/2014\/09\/07\/taming-the-terminal-part-20-of-n-file-searches\/","title":{"rendered":"TTT Part 20 of n &#8211; File Searches"},"content":{"rendered":"<div class=\"pps-series-post-details pps-series-post-details-variant-classic pps-series-post-details-18702\" data-series-id=\"553\"><div class=\"pps-series-meta-content\"><div class=\"pps-series-meta-text\">This entry is part 20 of 39 in the series <a href=\"https:\/\/www.bartbusschots.ie\/s\/series\/taming-the-terminal\/\">Taming the Terminal<\/a><\/div><\/div><\/div><p>In the previous previous instalment we looked at using <code>egrep<\/code> to search for a particular piece of text in a stream or file. <code>egrep<\/code> is often a great tool for finding a file you are looking for, but only if the file is a plain text file, and only if you are searching for that file based on its content. What if you want to search for files based on other criteria, like the last time the file was edited, or the name of the file, or the size of the file, or the type of the file etc.? For that you need a different command, for that you need <code>find<\/code>.<\/p>\n<p><!--more--><\/p>\n<div class=\"podcast\">\n<p>Listen Along: Taming the Terminal Podcast Episode 20<\/p>\n<!--[if lt IE 9]><script>document.createElement('audio');<\/script><![endif]-->\n<audio class=\"wp-audio-shortcode\" id=\"audio-9277-1\" preload=\"none\" style=\"width: 100%;\" controls=\"controls\"><source type=\"audio\/mpeg\" src=\"http:\/\/media.blubrry.com\/tamingtheterminal\/archive.org\/download\/TTT20FileSearches\/TTT_20_File_Searches.mp3?_=1\" \/><a href=\"http:\/\/media.blubrry.com\/tamingtheterminal\/archive.org\/download\/TTT20FileSearches\/TTT_20_File_Searches.mp3\">http:\/\/media.blubrry.com\/tamingtheterminal\/archive.org\/download\/TTT20FileSearches\/TTT_20_File_Searches.mp3<\/a><\/audio>\n\n\t\t\t<script>\n\t\t\t\tpodcastData1 = {\"title\":\"Taming the Terminal\",\"subtitle\":\"By Bart Busschots & Allison Sheridan\",\"description\":\"Taming the Terminal is a podcast based on the tutorial created by Bart Busschots and explained to Allison Sheridan.\",\"cover\":\"https:\\\/\\\/www.bartbusschots.ie\\\/s\\\/wp-content\\\/uploads\\\/2014\\\/08\\\/TTT-Logo.jpg\",\"feeds\":[{\"type\":\"audio\",\"format\":\"mp3\",\"url\":\"http:\\\/\\\/podfeet.com\\\/ttt\\\/ttt-rss.xml\",\"variant\":\"high\"}]}\n\t\t\t<\/script>\n\t\t\t<script\n\t\t\t\tclass=\"podlove-subscribe-button\"\n\t\t\t\tsrc=\"https:\/\/cdn.podlove.org\/subscribe-button\/javascripts\/app.js\" data-size=\"medium auto\" data-style=\"big-logo\" data-format=\"rectangle\" data-color=\"#599677\" data-json-data=\"podcastData1\" data-hide=\"\" data-language=\"en\" >\n\t\t\t<\/script>\n\t\t<\/div>\n<h2>The Basics of the <code>find<\/code> Command<\/h2>\n<p>Regardless of the criteria you wish to use, the basic form of the <code>find<\/code> command is always the same, you first need to tell it where to look, then you tell it what criteria to use when searching:<\/p>\n<pre class=\"crayon:false\">\r\nfind <u>path<\/u> <u>criteria 1<\/u> [<u>criteria 2<\/u> ...]\r\n<\/pre>\n<p>The path is searched recursively (by default), so if you give a path of <code>~\/Documents<\/code>, it will search your documents folder, and all folders within your documents folder. To search your entire computer, and all mounted drives, use a path of just <code>\/<\/code>. To use the current folder as the base of your search use a path of <code>.<\/code> (which always means &#8216;the current folder&#8217; as we learned in <a href=\"https:\/\/www.bartbusschots.ie\/s\/2013\/05\/26\/taming-the-terminal-part-4-of-n\/\" title=\"Taming the Terminal \u00e2\u20ac\u201c Part 4 of n (Navigation)\" target=\"_blank\">instalment 4<\/a>).<\/p>\n<h2>Defining Search Criteria<\/h2>\n<p>To see a full list of all possible search criteria, you can of course read the manual entry for find with the command <code>man find<\/code>, but we&#8217;ll look at some of the more common criteria you&#8217;ll be most likely to need here.<\/p>\n<h3>Search by File Name<\/h3>\n<p>You can do simple file-name searches with the <code>-name<\/code> flag followed by a simple pattern. Note that these simple patterns are NOT regular expressions, they use the same syntax as wild card expansion in BASH, i.e. <code>*<\/code> means <em>any number of any characters<\/em>, and <code>?<\/code> means <em>exactly one of any character<\/em>.<\/p>\n<p>A lot of the time you really don&#8217;t need the added power and complexity of regular expressions, because a lot of the time all you really want is the good old fashioned DOS pattern <code>*.extension<\/code>.<\/p>\n<p><strong>IMPORTANT<\/strong> &#8211; remember that <code>*<\/code> and <code>?<\/code> have meanings in BASH, so you need to escape them in some way to get reliable results. It&#8217;s ugly and hard to read <code>\\*<\/code> all over the place, so my suggestion is to get into the good habit of ALWAYS quoting your patterns when using <code>-name<\/code>.<\/p>\n<p>Let&#8217;s start with a really simple example, you know the full name of the file you&#8217;re looking for, but, you have no idea where it is. This is something you often come across when someone asks you to take a look at their server. You know you need to edit, say, <code>php.ini<\/code>, but you have no idea where their version of PHP is installed (this is very common when using web server packages for OS X like MAMP or XAMPP), the command below will find all files called <code>php.ini<\/code> anywhere in your computer:<\/p>\n<pre class=\"crayon:false\">\r\nfind \/ -name 'php.ini'\r\n<\/pre>\n<p><strong>NOTE<\/strong> &#8211; if you&#8217;re going to search your whole computer (like we did above), you&#8217;ll see a lot of &#8216;permission denied&#8217; errors. To avoid this, run the command with <code>sudo<\/code>, or, if you want to just ignore the errors, redirect <code>STDERR<\/code> to <code>\/dev\/null<\/code> with <code>2>\/dev\/null<\/code> like we learned in <a href=\"https:\/\/www.bartbusschots.ie\/s\/2014\/02\/08\/taming-the-terminal-part-16-of-n-crossing-the-streams\/\" title=\"Taming the Terminal \u00e2\u20ac\u201c Part 16 of n (Crossing the Streams)\" target=\"_blank\">instalment 16<\/a>.<\/p>\n<p>Something else you very often want to do is find all files of a given extension in a given location. For example, the command below will list all text files in your home directory:<\/p>\n<pre class=\"crayon:false\">\r\nfind ~ -name '*.txt'\r\n<\/pre>\n<p>You can get more creative by looking for all text files with names starting with a <code>b<\/code>:<\/p>\n<pre class=\"crayon:false\">\r\nfind ~ -name 'b*.txt'\r\n<\/pre>\n<p>Or all text files with a four letter name with <code>a<\/code> as the second letter:<\/p>\n<pre class=\"crayon:false\">\r\nfind ~ -name '?a??.txt'\r\n<\/pre>\n<p>JPEG files are an interesting case, you&#8217;ll very often see them with two different extensions, <code>.jpg<\/code> or <code>.jpeg<\/code>, how can we search for those without using regular expressions? The key is the <code>-or<\/code> flag. So, to look for files ending in either <code>.jpg<\/code> or <code>.jpeg<\/code> in our home directory, we could use the following:<\/p>\n<pre class=\"crayon:false\">\r\nfind ~ -name '*.jpg' -or -name '*.jpeg'\r\n<\/pre>\n<p>It looks like we&#8217;ve cracked it, but actually, we haven&#8217;t fully. Some cameras, for reasons I simply cannot fathom, use the extension <code>.JPG<\/code> instead of <code>.jpg<\/code> or <code>.jpeg<\/code>. To get around this we could either add two sets of <code>-or -name<\/code> criteria, or, we could do a case-insensitive name-search by replacing <code>-name<\/code> with <code>-iname<\/code>. This gives us a final JPEG-finding command of:<\/p>\n<pre class=\"crayon:false\">\r\nfind ~ -iname '*.jpg' -or -iname '*.jpeg'\r\n<\/pre>\n<p>Easily 90% of the time <code>-name<\/code> and <code>-iname<\/code> will be all you need to achieve your goals, but, sometimes, you really do need the power of full regular expressions. When this is the case, you can use the <code>-regex<\/code> or <code>-iregex<\/code> flags (<code>-iregex<\/code> being the case-insensitive version of <code>-regex<\/code>). There are two very important caveats when using regular expressions with <code>find<\/code>.<\/p>\n<p>Firstly, unlike with <code>-name<\/code>, <code>-regex<\/code> and <code>-iregex<\/code> do not match against just the file name, they match against the entire file path. It&#8217;s important that you remember this when constructing your patterns, or you&#8217;ll get unexpected results, either false positives or false negatives.<\/p>\n<p>Secondly, by default <code>-regex<\/code> and <code>-iregex<\/code> use Basic POSIX Regular Expressions (BREs), rather than the newer Extended POSIX Regular Expressions (EREs) we learned about in instalments <a href=\"https:\/\/www.bartbusschots.ie\/s\/2014\/04\/27\/taming-the-terminal-part-16-of-n-regular-expressions\/\" title=\"Taming the Terminal \u00e2\u20ac\u201c Part 17 of n (Regular Expressions)\" target=\"_blank\">17<\/a> and <a href=\"https:\/\/www.bartbusschots.ie\/s\/2014\/05\/10\/taming-the-terminal-part-18-of-n-more-res\/\" title=\"Taming the Terminal \u00e2\u20ac\u201c Part 18 of n (More REs)\" target=\"_blank\">18<\/a>. Don&#8217;t worry, you can make find use EREs, you just need to add a <code>-E<\/code> flag to the start of the command (before the path).<\/p>\n<p>Given our knowledge of regular expressions, we could re-write our JPEG search as follows:<\/p>\n<pre class=\"crayon:false\">\r\nfind -E ~ -iregex '.*[.]jpe?g'\r\n<\/pre>\n<p>Again, like with <code>egrep<\/code> in the previous instalment, notice that we are quoting the RE to stop BASH wrongly interpreting any of the special characters we may need like <code>*<\/code> and <code>?<\/code> in the above example. Also notice the position of the <code>-E<\/code> flag, and that we don&#8217;t need to use <code>^<\/code> or <code>$<\/code> because the ENTIRE path has to match for the result to validate. This also means that without the <code>.*<\/code> at the start of the pattern no files will be returned.<\/p>\n<h3>Searching Based on Modified Time<\/h3>\n<p>Very often, we need to find something we were working on recently, and the key to finding such files is to search based on the time elapsed since a file was last modified. We can do just that with the <code>-ctime<\/code> flag (for changed time).<\/p>\n<p>By default <code>-ctime<\/code> works in units of days, however, we can explicitly specify the units of time we&#8217;d like to use by appending one of the following after the number:<\/p>\n<dl>\n<dt><code>s<\/code><\/dt>\n<dd>seconds<\/dd>\n<dt><code>m<\/code><\/dt>\n<dd>minutes<\/dd>\n<dt><code>h<\/code><\/dt>\n<dd>hours<\/dd>\n<dt><code>d<\/code><\/dt>\n<dd>days<\/dd>\n<dt><code>w<\/code><\/dt>\n<dd>weeks<\/dd>\n<\/dl>\n<p>Unless you specify a sign in front of the number, only files modified EXACTLY the specified amount of time in the past will be returned. That&#8217;s not usually useful. Instead, what you generally want are all files modified less than a certain amount of time ago, and to do that you add a minus sign before the number.<\/p>\n<p>So, to find all files in your Documents folder that have been updated less than an hour ago you could use:<\/p>\n<pre class=\"crayon:false\">\r\nfind ~\/Documents -ctime -1h\r\n<\/pre>\n<h3>Searching Based on File Size<\/h3>\n<p>Another criteria we may want to search on is file size. We can do this using the <code>-size<\/code> flag. The default units used by <code>-size<\/code> are utterly unintuitive &#8211; 512k blocks! Thankfully, like <code>-ctime<\/code>, <code>-size<\/code> allows you to specify different units by appending a letter to the number. The following units are supported:<\/p>\n<dl>\n<dt><code>c<\/code><\/dt>\n<dd>Characters (8-bit bytes)<\/dd>\n<dt><code>k<\/code><\/dt>\n<dd>KiB = 1024 bytes<\/dd>\n<dt><code>M<\/code><\/dt>\n<dd>MiB = 1024KiB (notice the case &#8211; must be upper!)<\/dd>\n<dt><code>G<\/code><\/dt>\n<dd>GiB = 1024MiB (notice the case &#8211; must be upper!)<\/dd>\n<dt><code>T<\/code><\/dt>\n<dd>TiB = 1024GiB (notice the case &#8211; must be upper!)<\/dd>\n<dt><code>P<\/code><\/dt>\n<dd>PiB = 1024TiB (notice the case &#8211; must be upper!)<\/dd>\n<\/dl>\n<p>Note that this command uses the old 1024-based sizes, not the 1,000 based <a href=\"http:\/\/en.wikipedia.org\/wiki\/International_System_of_Units\" target=\"_blank\">SI units<\/a> used by OS X and hard drive manufacturers (and scientists and engineers and anyone who understands what kilo and mega etc. actually mean).<\/p>\n<p>Also Like with <code>-ctime<\/code>, if you don&#8217;t pre-fix the number with a symbol, only files EXACTLY the size specified will be returned.<\/p>\n<p>For example, the following command shows all files in your downloads folder that are bigger than 200MiB in size:<\/p>\n<pre class=\"crayon:false\">\r\nfind ~\/Downloads -size +200M\r\n<\/pre>\n<p>Similarly, the following command shows all files in your documents folder smaller than 1MiB in size:<\/p>\n<pre class=\"crayon:false\">\r\nfind ~\/Downloads -size -1M\r\n<\/pre>\n<h3>Filtering on File &#8216;type&#8217;<\/h3>\n<p>When I say file type, I mean that in the POSIX sense of the word, not the file extension sense of the word. In other words, I mean whether something is a regular file, a folder, a link, or some kind of special file.<\/p>\n<p>The type of a file can be filtered using the <code>-type<\/code> flag followed by a valid file type abbreviation. The list below is not exhaustive, but it covers everything you&#8217;re likely to need:<\/p>\n<dl>\n<dt><code>f<\/code><\/dt>\n<dd>a regular file<\/dd>\n<dt><code>d<\/code><\/dt>\n<dd>a directory (AKA folder)<\/dd>\n<dt><code>l<\/code><\/dt>\n<dd>a symbolic link<\/dd>\n<\/dl>\n<p>This flag will almost always be used in conjunction with one or more other search flags. For example, the following command finds all directories in your documents folder that contain the word <code>temp<\/code> in their name in any case:<\/p>\n<pre class=\"crayon:false\">\r\nfind ~\/Documents -type d -iname '*temp*'\r\n<\/pre>\n<h3>Inverting Search Parameters<\/h3>\n<p>In most situations it&#8217;s easiest to express what it is you want to search for, but sometimes it&#8217;s easier to specify what you don&#8217;t want. In situations like this it can be very useful to be able to invert the effect of a single search parameter. You can do this with the <code>-not<\/code> flag.<\/p>\n<p>For example, you may have a folder where you keep your music, and it should only contain MP3 files and folders. To be sure that&#8217;s true you could search for all regular files that do not end in <code>.mp3<\/code> and are not hidden (like those ever-present <code>.DS_Store<\/code> files) with a command like:<\/p>\n<pre class=\"crayon:false\">\r\nfind ~\/Music\/MyMP3s -type f -not -iname '*.mp3' -not -name '.*'\r\n<\/pre>\n<h3>Limiting Recursion<\/h3>\n<p>By default the find command will drill down into every folder contained in the specified path, but, you can limit the depth of the search with the <code>-maxdepth<\/code> flag. To search only the specified folder and no deeper use <code>-maxdepth 1<\/code>.<\/p>\n<p>Note that limiting the depth can really speed up searches of large folders if you know what you want is not deep down in the hierarchy. For example, if you have a lot of documents in your documents folder it can take ages to search it, but, if you are only interested in finding stuff at the top level you can really speed things up. Lets say we are the kind of person who makes lots of temp folders at the top level of their Documents folder (guilty as charged), and you want to find them all so you can do a bit of house keeping, you could search your entire Documents folder with:<\/p>\n<pre class=\"crayon:false\">\r\nfind ~\/Documents -type d -iname '*temp*'\r\n<\/pre>\n<p>When I do this it takes literally minutes to return because I have over a TB of files in my Documents folder. I can get that down to fractions of a seconds by telling <code>find<\/code> that I&#8217;m only interested in the top level stuff with:<\/p>\n<pre class=\"crayon:false\">\r\nfind ~\/Documents -type d -iname '*temp*' -maxdepth 1\r\n<\/pre>\n<h2>Combining Search Criteria (Boolean Algebra)<\/h2>\n<p>We&#8217;ve already seen that we can use the <code>-or<\/code> and <code>-not<\/code> flags, but there is also a <code>-and<\/code> flag. In fact, if you don&#8217;t separate your criteria with a <code>-or<\/code> flag, a <code>-and<\/code> flag is implied.<\/p>\n<p>The following example from above:<\/p>\n<pre class=\"crayon:false\">\r\nfind ~\/Music\/MyMP3s -type f -not -iname '*.mp3' -not -name '.*'\r\n<\/pre>\n<p>Is actually interpreted as:<\/p>\n<pre class=\"crayon:false\">\r\nfind ~\/Music\/MyMP3s -type f -and -not -iname '*.mp3' -and -not -name '.*'\r\n<\/pre>\n<p>We can even take things a step further and add sub-expressions using <code>(<\/code> and <code>)<\/code> to start and end each sub expression (they can even be nested). Note that <code>(<\/code> and <code>)<\/code> have meaning in BASH, so they need to be either escaped or quoted. Since I find escaping makes everything hard to read and understand, I recommend always quoting these operators.<\/p>\n<p>As a final example, the following command will find large powerpoint presentations in your Documents folder, i.e. all files bigger than 100MiB in size that end in <code>.ppt<\/code> or <code>.pptx<\/code>.<\/p>\n<pre class=\"crayon:false\">\r\nfind ~\/Documents -size +100M '(' -iname '*.ppt' -or -iname '*.pptx' ')'\r\n<\/pre>\n<h2>Conclusions<\/h2>\n<p>In this instalment we&#8217;ve seen that we can use the <code>find<\/code> command to search for files based on all sorts of criteria, and that we can combine those criteria using boolean algebra to generate very powerful search queries. In the next instalment we&#8217;ll discover that you can use the <code>find<\/code> command not only to search for files, but to apply an action to every file it finds.<\/p>\n<p>The <code>find<\/code> command is common to all POSIX operating systems, so it works on Linux, Unix, and OS X. OS X maintains an index of your files allowing quick searching in the Finder and via Spotlight. Because this index is kept up to date by the OS, it makes searching with Spotlight much quicker than searching with <code>find<\/code>. In the next instalment we&#8217;ll also discover that OS X ships with a terminal command that allows you to use the power of Spotlight from the command line!<\/p>\n","protected":false},"excerpt":{"rendered":"<div class=\"pps-series-post-details pps-series-post-details-variant-classic pps-series-post-details-18702 pps-series-meta-excerpt\" data-series-id=\"553\"><div class=\"pps-series-meta-content\"><div class=\"pps-series-meta-text\">This entry is part 20 of 39 in the series <a href=\"https:\/\/www.bartbusschots.ie\/s\/series\/taming-the-terminal\/\">Taming the Terminal<\/a><\/div><\/div><\/div><p>In the previous previous instalment we looked at using egrep to search for a particular piece of text in a stream or file. egrep is often a great tool for finding a file you are looking for, but only if the file is a plain text file, and only if you are searching for that [&hellip;]<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[12,446],"tags":[414,406,412,26,516,458],"series":[553],"class_list":["post-9277","post","type-post","status-publish","format-standard","hentry","category-computers-tech","category-sysadmin","tag-bash","tag-commandline","tag-linux","tag-os-x","tag-taming-the-terminal","tag-tutorial","series-taming-the-terminal"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p7t9xK-2pD","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.bartbusschots.ie\/s\/wp-json\/wp\/v2\/posts\/9277","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.bartbusschots.ie\/s\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.bartbusschots.ie\/s\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.bartbusschots.ie\/s\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.bartbusschots.ie\/s\/wp-json\/wp\/v2\/comments?post=9277"}],"version-history":[{"count":8,"href":"https:\/\/www.bartbusschots.ie\/s\/wp-json\/wp\/v2\/posts\/9277\/revisions"}],"predecessor-version":[{"id":15339,"href":"https:\/\/www.bartbusschots.ie\/s\/wp-json\/wp\/v2\/posts\/9277\/revisions\/15339"}],"wp:attachment":[{"href":"https:\/\/www.bartbusschots.ie\/s\/wp-json\/wp\/v2\/media?parent=9277"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.bartbusschots.ie\/s\/wp-json\/wp\/v2\/categories?post=9277"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.bartbusschots.ie\/s\/wp-json\/wp\/v2\/tags?post=9277"},{"taxonomy":"series","embeddable":true,"href":"https:\/\/www.bartbusschots.ie\/s\/wp-json\/wp\/v2\/series?post=9277"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}