Advanced Windows Shell Tutorial

Review of the Windows Shell

In the first lab, you learned the basics of using the Windows command shell. In this lab you will learn how to automate shell commands. You will create batch files consisting of sequences of commands that can be executed again and again. First, however, we will review the basic shell commands.

Exercise 1

Open a command shell window. Describe what you did to open it.

Exercise 2

In the command shell window, change to your personal directory. (If necessary, create it.) What commands did you issue?

Exercise 3

Inside your personal directory, create a subdirectory named advanced. What command did you issue?

Next, download this ZIP file and place it into the c:\yourname\advanced directory. (As always in this tutorial, c:\yourname denotes your personal directory. Be sure to substitute d:\jqsmith or c:\users\jqsmith or whatever your personal directory is called.)

To unzip a file, you can use the jar command. The jar command takes an option that describes the action you want to take, then the name of the compressed file. For example,

jar -xf bank.zip

extracts the files from the file bank.zip into the current directory. (The xf denotes extraction from a file.)

Exercise 4

Unzip the file bank.zip now. How many files were stored inside it? (Hint: dir)

NOTE: If you get a "command not found" error message when executing the jar command, then you need to specify the full path name of the jar.exe program, such as c:\j2sdk1.4.2\bin\jar.exe. You may need to spend some time looking for the jar.exe file, or ask a lab assistant. If you do this exercise on your own computer, make sure that the Java SDK is installed. You can download it from http://java.sun.com/j2se.

Batch Files

When you submit your homework by email, your instructor will probably ask you to zip up all homework files. A beginner would use a  program such as WinZip for this task. You know the drill. Click on the WinZip icon. Click, click, click, until you have added each file. Click, click, click, type the zip file name, click, click, and you are done. Did you need to make a last-minute change to a file? Do it all over again. And again.

You can do better than that. The command

jar -cf homework.zip *.java

makes a zip file called homework.zip that contains all Java files in the current directory. (The cf option denotes compression to a file.)

Maybe you also need to include some other files (such as Javadoc documentation and UML diagrams). Then the command gets longer.

jar -cvf homework.zip *.java *.html *.jpg

(The v option produces verbose output, listing the actions that the jar command takes.)

Rather than typing the command every time, you can place it into a batch file, and run the batch file as a single command. We will start with a simple version of such a batch file.

Start the Notepad program. Type in the following text:

@echo off 
rem Zip up source, HTML, images 
jar -cvf homework.zip *.java *.html *.jpg

Then use the File -> Save as menu option to save the file as c:\yourname\advanced\zipup.bat.

NOTE: Depending on the Windows configuration, the Notepad program may add a .txt extension to the file name, that is, zipup.bat.txt. (This evil and stupid behavior may be hard to spot unless you changed the Windows Explorer's evil and stupid default of "hiding known file extensions". You learned how to make that change in lab #1.) The remedy is to surround the file name in double quotes: "zipup.bat".

To execute the batch file, simply type its name:

zipup

In general, you execute the commands in a batch file by typing the name of the file (without the extension .bat).

Exercise 5

Run the zipup batch file now, in the c:\yourname\advanced directory. Before running the batch file, make sure that the directory contains a few Java files (from the preceding exercise). What is the result of running the batch file?

The first two commands of the start batch file are new commands. @echo off suppresses the display of each command on the screen as it is executed. A line starting with rem contains comments that are ignored by DOS.

Exercise 6

Comment out the line @echo off (by prefixing it with rem). Run the batch file. What is the difference?

Your instructor may ask you to run the Javadoc program to extract document comments before you submit your homework. The command is

javadoc *.java

Exercise 7

Enhance the zipup.bat file to automatically execute the javadoc program. Call the file zipup2.bat. What is the contents of the file now?

As you make more changes to the file, keep changing the name (zipup2, zipup3, etc.) Then you can submit the various versions for grading.

Batch File Arguments

Suppose you sometimes want to zip up your homework to hw1.zip, then to hw2.zip, or even to hw1_1728.zip. This can be easily done by modifying zipup.bat.

Change the filename homework.zip to %1.zip:

jar -cvf %1.zip *.java *.html *.jpg

When executing the batch file, you now need to supply the name of the zip file (without the .zip extension which is appended in the batch file). For example,

zipup3 hw1
 

Then the %1 is replaced by hw1. %1 is the first argument of the batch file, %2 is the second, and so on.

Exercise 8

How do you run the batch file to save your homework to hw1_1728.zip?

Exercise 9

Type zipup3 without an argument. What happens?

Let's modify the batch file to give an error message if no command line argument is specified.

Add the following lines to the batch file and name it zipup4.bat.

@echo off 
if "%1" == "" goto error
rem Zip up source, HTML, images 
jar -cvf %1.zip *.java *.html *.jpg
goto end
:error
echo Usage: zipup4 filename     

Exercise 10

Now type zipup4 without an argument. What happens?

The if "a" == "b" command tests whether the strings a and b are equal. The goto command goes to a label. Each label starts with a colon (:). The echo command displays a message on the screen.

Environment Variables

The zipup batch file is quite useful, and you may want to put it to work for many assignments. But you now have a problem. Suppose you have worked on your homework in the directory c:\yourname\hw1. But the batch file is located in c:\yourname\advanced.

Exercise 11

Create a directory c:\yourname\hw1. Change to that directory. Copy all Java files (but not the batch file) from the c:\yourname\advanced directory, so that you can pretend that you just completed a homework assignment. What shell commands did you use?

Exercise 12

Now type zipup4 hw1. What happens?

The operating system cannot find the zipup4.bat file. After all, that file is in a rather obscure directory. The operating system only looks at certain places to locate executable files. These places include

• the current directory
• the c:\windows and c:\windows\system32 directories
• any other directories that are referenced in the PATH environment variable

An environment variable is a variable that is used by the command shell to identify certain important settings.

Exercise 13

Type the command set. Look for the definition of the PATH variable. What result do you get?

The command set lists all environment variables. The PATH variable contains a list of directories, separated by semicolons. Minimally, the PATH will contain

c:\windows;c:\windows\system32

If you work in a computer lab, the lab administrator probably has added the directory for the Java commands (c:\j2sdk1.4.2\bin or something similar).

NOTE: Depending on your version of Windows, the environment variable may be listed as Path, not PATH. That's ok--in Windows, environment variables are not case-sensitive. However, it is considered good style to use uppercase letters for environment variables.

You sometimes want to add directories to the PATH so that the operating system can locate executable files in them. For example, let us enhance the PATH so that the command shell finds the zipup batch file.

Create a batch file myenv.bat that has the following contents:

@echo off
set PATH=c:\yourname\advanced;%PATH%
echo The PATH is now %PATH%

Be very careful with the spacing. There must be no spaces around the symbols = ; %. Only set and PATH are separated by a space.

The expression %PATH% denotes the current value of the environment variable PATH.

The command

set PATH=...

sets the PATH environment variable to the expression on the right hand side.

Exercise 14

Execute the myenv.bat batch file. What result do you get?

NOTE: In Windows XP, you can also execute the command set PATH=...;%PATH% directly on the command line. However, in older versions of Windows, the %PATH% expression works only inside a batch file.

Exercise 15

Make sure that you are currently in the c:\yourname\hw1 directory. Now type zipup4 hw1. What happens? Why does the zipup command work now?

Another important environment variable is the CLASSPATH variable. It contains a list of locations in which the Java compiler and interpreter look for class files. Particularly for advanced Java programming, you will need to set the CLASSPATH variable.

We will show one example how this is done. The CheckStyle program checks whether a Java program follows good style conventions (no missing Javadoc comments, no public instance fields, etc.) CheckStyle itself is a Java program, so you start it by specifying its main class name. You specify a rule file with the -c option. Then specify the Java files that you want to check as additional parameters. For example,

java com.puppycrawl.tools.checkstyle.Main -c sjsu.xml BankAccount.java

Here, com.puppycrawl.tools.checkstyle.Main is the Main class in the com.puppycrawl.tools.checkstyle package. That class has a method public static void main(String[] args) that kicks off the CheckStyle program.

Exercise 16

Change to the c:\yourname\advanced directory. Download the file checkstyle-all-3.1.jar into that directory. Also download the file sjsu.xml and save it in the same directory. That file contains rules for the SJSU CS department style guide. Now type java com.puppycrawl.tools.checkstyle.Main -c sjsu.xml BankAccount.java What happens?

Unfortunately, the Java interpreter can't find the com.puppycrawl.tools.checkstyle.Main class. That class isn't in one of the standard places. By default, the Java interpreter looks for classes

• in the rt.jar file of the Java runtime directory
• in all jar files contained in the ext subdirectory of the Java runtime directory
• in the directories listed in the CLASSPATH environment variable
  
• or, if there is no CLASSPATH environment variable, in the current directory
  

But the com.puppycrawl.tools.checkstyle.Main file is contained inside the checkstyle-all-3.1.jar file. The remedy is to set the CLASSPATH to include that file:

set CLASSPATH=.;c:\yourname\advanced\checkstyle-all-3.1.jar

Note that the CLASSPATH contains the current directory (.).

Exercise 17

Execute the command for setting the CLASSPATH. Now type java com.puppycrawl.tools.checkstyle.Main -c sjsu.xml BankAccount.java What happens? Why does the command now work?

In general, you will want to set the CLASSPATH variable in a batch file such as myenv.bat.

Of course, the CheckStyle command is way too long to type each time you want to check on the quality of a source file. Let's turn it into a batch file.

Exercise 18

Write a batch file check.bat so that you can check the style of any Java file. For example, check BankAccount.java In your batch file, you need to specify the full path name for the jar file and the sjsu.xml file. Otherwise, your batch file won't work when it is called from another directory. Because the same directory name occurs twice, use an environment variable: SET CHECKSTYLEHOME=c:\yourname\advanced SET CLASSPATH=.;%CHECKSTYLEHOME%\checkstyle-all-3.1.jar ... What is the contents of your batch file?

NOTE: You can avoid the use of the CLASSPATH environment variable by using the -classpath option for the javac and java commands. For example, java -classpath c:\yourname\advanced\checkstyle-all-3.1.jar com.puppycrawl.tools.checkstyle.Main -c c:\yourname\advanced\sjsu.xml BankAccount.java Of course, such a long command only makes sense inside a batch file.

You have just learned the best method for manipulating the PATH and CLASSPATH variables. Make yourself a batch file that sets the variables to the settings that you want. Then execute the batch file immediately after opening the command shell.

Alternatively, you can make permanent changes to the PATH and CLASSPATH variable. However, in a lab environment, you probably don't have the authority to do this. Moreover, when you work on different projects, you may want to have different settings, not one global setting. Simply make a separate batch file for each project and execute the appropriate file after opening a command shell.

NOTE: Occasionally, it does make sense to make permanent additions to the PATH. For example, after downloading the Java SDK, you should add the directory for the Java commands to the PATH. In Windows XP, open the control panel. Make sure it is in classic mode. Click the System icon, then on the Advanced tab, then on the Environment Variables button. Edit the PATH variable in the System Variables section. The same method works for Windows NT and Windows 2000.

NOTE:  For Windows 95, 98, and ME, you use a completely different (and more sensible) approach to change the PATH environment variable. Edit the file c:\autoexec.bat (or create it if it doesn't already exist). Then place the set PATH=...;%PATH% command inside that batch file. It is executed whenever your computer boots up.

NOTE: Globally setting the CLASSPATH variable (in the control panel or autoexec.bat) will only bring you grief in the long run. Don't do it. Set CLASSPATH in a project batch file, or use the -classpath option.

NOTE: If you use Windows 95/98/ME, you may run out of "environment space" if you set too many environment variables.  The remedy: In the shell window, click on the Properties button, then pick the Memory tab and set the Initial Environment to the maximum value (4096).

Redirection

Suppose you want to email a directory listing, perhaps to ask for help with a technical problem. How can you do it? The dir command shows the display of the directory on the screen. If it happens to fit on one screen, you can copy and paste it into your email program. But what do you do if the directory listing doesn't fit on one screen?

You can capture terminal output to a file, with the output redirection operator. The construct

command > file

sends the output of a command to a file.
 

Exercise 19

Execute the command dir c:\windows > out.txt Then look inside the out.txt file. What do you see? What is the last line in the file?

One useful application for redirection is automated grading of homework. Imagine you work as a grader for your professor. You get a bunch of zip files, one from each student. For each of the zip files, you want to go through the same steps:

• Unzip the file
• Compile the program
• Run the program
• Capture the output in a file

Why capture the output? You can then look at it when it is convenient, or email it to the professor or student.

Let's design such a batch file. We'll call it grade.bat. It takes two parameters: the name of the zip file (without the extension), and the class with the main method. For example,

grade hw1_1728 BankAccountTest

Exercise 20

What are the commands to unzip the file hw1_1728.zip, compile the Java source, run the BankAccountTest program, and capture the output in the file hw1_1728.txt? Hint: Use redirection: java ... > ...

Exercise 21

Now write the grade.bat file. Simply take the results from the preceding exercise and replace hw1_1728 with %1 and BankAccountTest with %2.  What is the contents of grade.bat?

Exercise 22

Make sure that both grade.bat and bank.zip are in the c:\yourname\advanced directory. Then run grade bank BankAccountTest What output file was created? What is the contents of that file?

Another useful redirection operator is >>. It appends the output of a program to a file. For example, you may first want to run CheckStyle on the submitted programs and capture the output in the report file, then append the output of the program run.

java -jar checkstyle-all-3.1.jar -c sjsu.xml *.java > homework.txt
javac *.java
echo ===Program Run=== >> homework.txt
java BankAccountTest >> homework.txt

Note the echo command. It produces a line of output, ===Program Run===. That line is appended to the homework.txt file.

Exercise 23

Add this enhancement to the grade.bat file and call the result grade2.bat. As before, make sure the file can be executed from any directory. (Note that we use a different method for invoking CheckStyle, with the -jar option. This option does not require you to set the class path.) What is the contents of grade2.bat now?

Finally, let us capture the compiler output in the homework report as well. Unfortunately, now we have a problem. Compiler errors are reported to the standard error stream, not the standard output stream. The > and >> commands only redirect  the standard output stream. The remedy is to use the 2> operator which redirects the standard error stream. (For historical reason, that stream is also known as stream #2.) 

javac *.java 2> homework.txt

There is also a 2>> operator to append the standard error stream output.

NOTE: The 2> operator does not exist in Windows 95/98/ME. For these versions of Windows, compile the following Java class: import java.io.*; public class Errout { public static void main(String[] args) throws IOException { Process p = Runtime.getRuntime().exec(args); BufferedReader err = new BufferedReader(new InputStreamReader(p.getErrorStream())); String line; while ((line = err.readLine()) != null) System.out.println(line); } } Then execute the following command: java Errout javac MyProg.java > homework.txt

You can also supply the input to programs in a file rather than keying it in. This is useful when supplying program input for testing. The < operator redirects console input to a file. That is, whenever the program wants to read input, it grabs it from the file, not the keyboard.

Download and unzip the interest.zip file. Open the SavingsAccountTest program with the Notepad program and have a glance at it.

The test program asks the user to supply several values.

Exercise 24

Compile and run the SavingsAccountTest program. What balance do you get after 10 years with an initial deposit of $10,000 at 5% interest?

As a grader, you would not want to type in the same input values over and over. Therefore, let us place the program inputs into a file input1.txt.

Exercise 25

Create an input.txt file with three lines, one for each program input. Run the command java SavingsAccountTest < input1.txt Note that you no longer provide any keyboard input. What output do you get?

Input redirection only works with console input, not with input into dialog boxes. For that reason, the SavingsAccountTest program uses the BufferedReader class, not the JOptionPane class.

Some commands are specifically designed to work with input from the keyboard (thus allowing redirected input). The sort program is an example.

Exercise 26

Try the following: dir c:\windows > out.txt sort < out.txt What is the result?

The sort program reads from standard input (that is, the keyboard or a redirected file). It is possible (although not very useful) to have sort sort some lines of keyboard input.

Exercise 27

Type sort and then the following lines. Mary had a little lamb Hit Enter after each line. When you are done, hit Ctrl-Z and Enter. Ctrl-Z is a special key that tells the shell to stop sending keyboard input to the program. What is the result?

As you just saw, sorting a directory listing can be achieved in two steps. First capture the listing in a file (dir > out.txt), then sort the captured output (sort < out.txt). Because this combination is so frequent, there is a shortcut for it, called a pipe.

dir | sort 

This pipe, written with a vertical bar, runs dir, stores its output in a temporary file, then runs sort and redirects its input from that temporary file, and then automatically deletes the temporary file.

Another useful program that reads from standard input is more. The more program stops every screen full and waits for you to hit a key before continuing.

Exercise 28

Type the command dir c:\windows | more What happens?

You can pipe any sequence of commands together.

Exercise 29

Type the command dir c:\windows | sort | more What happens?

Error Level

Let's look at the grade.bat file again. Suppose a student submitted a file that doesn't even compile. Then we should abort the batch file after the call to javac, without trying to execute the program.

Every program returns an integer value, called the error level, to the operating system. By convention, a program returns the value 0 when it is successful and a non-zero value to indicate some kind of error. In your own Java programs, you can set the error level by calling

System.exit(level);

The Java compiler sets the error level to 0 when a program compiled successfully, and to a non-zero value otherwise.

You can test the error level of a program in a batch file.

javac *.java
if errorlevel 1 goto error 
...
goto end
:error
echo Program didn't compile > %1.txt
:end

Exercise 30

Add this enhancement to your grading batch file and call the result grade3.bat. Test it by introducing an error in the BankAccount.java file: Change class to lass. Then create a new zip file bad.zip that contains the file with the error. What is the contents of grade3.bat What commands did you use to create the file bad.zip? What was the contents of bad.txt after you ran grade bad BankAccountTest?

The if errorlevel n command tests whether the error level is >= n. Thus, if you have a program that returns 1 for a mild error and 2 for a fatal error, then you want to first carry out a check if errorlevel 2, then if errorlevel 1.

Loops

You know how to make simple branches in batch files with commands such as

if "%1" == "" ...
if errorlevel 1 ...
if exist input1.txt ...

As you can see, you can do simple programming in batch files. For more advanced programming, you need the for construct.

There are two separate versions of for:

for %f in (*.java) do javac %f
for %d in (A,C,D) do dir %d:

The first for loop lets the variable %f run through all files that match the pattern *.java. The second for loop lets the variable %d run through the three choices A, C, D.

Variables in a for loop must always start with a % sign. However, inside a batch file, you must use two% signs, such as

rem this is inside a batch file
for %%f in (*.java) do javac %%f

Now let us put this technique to work in our grading batch file. We want to feed all input files of the form input*.txt to the program to be graded. That way, the grader can prepare an arbitrary number of files input1.txt, input2.txt, etc. Ideally, we would like to issue the command

for %%f in (input*.txt) do java %2 < %%f >> %1.txt

Sadly, this command does not work in all versions of Windows. In some Windows versions, you cannot mix the redirection operators with the for loop.

NOTE: Incompatibilities between Windows versions are a vexing problem for authors of sophisticated batch files. Most shell users eventually give up in disgust and migrate to another shell such as the bash shell. In Windows, install the excellent Cygwin package to get a bash shell and a wealth of powerful command-line tools with well-defined behavior.

In this situation, you can solve the compatibility problem with trick. Write a second batch file gradehelper.bat that contains as its  single command the body of the for loop.

rem this is gradehelper.bat
rem %1 is the name of the output file (without extension)
rem %2 is the name of the program file (without extension)
rem %3 is the name of the input file (with extension)
java %2 < %3 >> %1.txt

Then the original batch file can call gradehelper:

rem this is grade4.bat
rem %1 is the name of the output file (without extension)
rem %2 is the name of the program file (without extension)
. . .
for %%f in (input*.txt) do call gradehelper %1 %2 %%f
. . .

(You use of call command to call a batch file from inside another batch file.)

Exercise 31

Add this enhancement to your grading batch file. Call the result grade4.bat. Test it by producing two files input1.txt and input2.txt with different inputs for the SavingsAccountTest program. What is the contents of grade4.bat? What was the contents of interest.txt after you ran grade interest SavingsAccountTest? How can you improve gradehelper.bat so that the outputs that belong to different program runs are separated from each other? (Hint: echo)

If you have trouble with this exercise, try using the echo command for debugging. For example,

echo gradehelper.bat is called with parameters %1 %2 %3

or

for %%f in (input*.txt) do echo Need to call gradehelper %1 %2 %%f

Exercise 32

The grade4.bat file grades a single homework submission. Suppose you are the grader, and you have to grade dozens of submissions, each of which has the form hw1_xyzw.zip, where xyzw is a four-digit number. Write a file gradeall.bat that grades all files and produces a combined report for all of them. Hints: Modify the grading batch file so that the reports are all appended to a single report file. Call the result grade5.bat. Typical usage: grade5 hw1_xyzw.zip BankAccountTest report.txt. (Note that you will need to make some minor changes in grade5.bat because now %1 includes the .zip extension. You should use the same gradehelper.bat file as in the preceding exercise.) In gradeall.bat, loop through all zip files in the current directory and call grade5.bat for each of them. Typical usage: gradeall BankAccountTest report.txt What are your gradeall.bat,and grade5.bat files?

Hopefully, these exercises have given you a feel for the power of automation. While it is undeniably challenging to automate a task for the first time, the effort is  repaid handsomely. It is fun to watch the computer do the same boring tasks over and over, particularly if you consider how much time it would have taken you to do it by hand.

In your programming and testing process, you carry out lots of repetitive steps. Automate them, and you will become more productive. You will also find that you would never attempt certain tasks without automation. For example, consider the task of testing your programs. Whenever you change a program, you should really test it again with a bunch of inputs. Do you do that? Probably not. What could be more tedious than typing in the same inputs over and over again? You now know that you can automate that task. Put a bunch of test inputs into files and write a batch file that automatically feeds them into your program. Test automation leads to higher quality programs.

For those reasons, all professional programmers are serious about automating their build and test processes. You have just learned how to use the command shell for basic automation tasks.

Blogged with the Flock Browser