Learn How to Use the Command Line: 99 Ways Workshop #88

The Software Testing Club recently put out an eBook called "99 Things You Can Do to Become a Better Tester". Some of them are really general and vague. Some of them are remarkably specific.

My goal for the next few weeks is to take the "99 Things" book and see if I can put my own personal spin on each of them, and make a personal workshop out of each of the suggestions. 

Suggestion #88: Learn how to use the command line. Shell and batch scripts too.

I have to admit, these are getting a little more difficult to write in this stretch. Suggestions 87 through 90 could keep people busy for years in and of themselves, and coming up with a workshop idea to put in a blog post is proving to be quite daunting. Still, it's worth a shot, and I'm going to give it my best.

Workshop #88: Spend some quality time inside the terminal program of your operating system. Learn its ins and outs, practice typing in commands and parameters that can help control and direct them. Gather commands that can be executed in sequence together (i.e. batch them). For extra credit, do it for both UNIX/Linux and Windows.

Really, this is not a simple "do this in an afternoon" endeavor. Windows may have a pretty graphical gloss to wrap itself in (Mac, too) but at their heart are command lines that can be executed. The command line provides a lot of power, flexibility, and some syntactical requirements that have to be learned. 

Command lines (and the various tools that interact with them), primarily handle four functions. Those four things are:

- variable interpretation and substitution
- quoting
- syntax interpretation
- redirection

That's about it, but "that's about it" covers a really large field. Still, by looking at it from that perspective, it makes a lot of things easier to understand, and thus, it becomes easier to see what we can do with those options.

When we see something like:

"cat file1 file2 file 3 | grep '[0-9]\{1,2\}\/[0-9]\{1,2\}\/[0-9]\{2,4\}' | uniq | sort > file4"

we are, effectively, calling on all four of the functions (though it may not be so obvious). 

In the above example we are saying the following:

- take the three files I have named, and concatenate their output (show the contents of the three files one after another) together as one output stream. 

- "pipe" that output stream (take the output of one program and direct it as the input to another program) to "grep", and have "grep" look for a string. If grep finds the string, it will keep it. if it doesn't, that line will be removed.

[Yes, that quoted option after grep is a regular expression, or regex, like we talked about in the last suggestion. They are hugely helpful in examples like this. In this case, all that it means is "look for any date string that is formatted as "mm/dd/yy" or "mm/dd/yyyy"]

- take the output that was parsed by grep, and pipe it to a program called "uniq". 

-  uniq will receive each of the lines, examine them, keep only the lines that are "unique" and remove the lines that are not. 

- uniq will then pipe its output to the input of another program, called "sort" which, as its name indicates, will reorder the lines so that they are in alphabetical order.

- finally, sort will take its output and redirect (write) it into a file (file 4), where the input stream will be stored and saved.

So there you go, redirection, quoting, and syntax. Variable substitution is going on in this example, too, albeit under the covers. How? Well, the file names we provide are variables in their own right, as far as the commands are concerned. 

If we wanted to make it a bit more apparent, we could save the same command as a script file, and provide variables like this:

file1="$1"
file2="$2"
file3="$3"
cat ${file1} ${file2} ${file3} | grep '[0-9]\{1,2\}\/[0-9]\{1,2\}\/[0-9]\{2,4\}' | uniq | sort > file4


If we were to save this simple script in a file like "strip-out-dates", make it executable, and run with the names of three files, the script above would execute and work on them, like this:

% strip-out-dates example1 example2 example3

When it finishes, you'll see a file called "file4", and that file will have a combination of all the lines from the three files that you fed to the script that had valid dates in them.

This, again, is just a little bit of what you can do, but suffice it to say that, if you wanted to take a variety of commands like I have here and run them, or stack a bunch of commands you use over and over, then you can build up scripts that can allow you to run commands in the order you have placed them in the file.

Bottom Line:

The command line can feel really daunting, but in truth, it only does what it is directed to do and what it has the ability to do. The shell itself is rather limited. The commands that it can run, and the permutations as to how those commands can be formatted, now that can go on forever. The good news is that, even there, there are also some basic rules that we can use that can help us. Those rules and examples, though, will have to wait for the next post.