Understand Computer Grader

"Being lucky" does not mean "it is working."

Two Stage Grading

Your programs are graded in two stages. The first stage is a computer grader and the second is a human grader. The purposes of this two-stage grading has many advantages.

The computer grader can

• give you quick feedback, usually within a few seconds, after each submission.
• allow you to submit multiple times and still give you quick feedback.
• execute your program using many test cases.
• provide performance, coverage, and memory usage information.
• check memory access violation even though a program seems "working"

The computer grader is objective and consistent but it has limitations.

The human grader can

• evaluate whether a program is "close" to work but fails to pass computer grading.
• determine whether a program is designed and implemented well.
• detect whether a program tries to "trick" the computer grader.
• You have to submit programs without any syntax error. If your programs have one or multiple syntax errors, the programs will not be graded by a human grader.

The human grader is a person making judgment; thus, the decision is subjective.

There are also disadvantages. The computer grader

• can perform only well-defined testing. In most case, the computer grader compares your program's output against the expected output using the diff command in Linux. If there is any difference, the computer grader thinks your program is wrong. This means that your program may fail the test if the output has additional space or some other types of "minor" differences. We try to avoid this problem by giving you the code for formatting the output and asking you to provide correct values. You should not modify the code given to you.
• stops your program when it takes too long. The computer grader cannot wait for your program indefinitely (this is call the halting problem and it is mathematically proven unsolvable). We set the time limit to be 100 times of the execution time of the sample solution. If your program is too slow, it is not graded. The computer grader also stops your program if it uses too much memory.
• stops testing if your program terminates abnormally. The most common problem is segmentation fault, when your programs manage memory incorrectly.

The computer grader is meant for you to submit your final solutions, not as a way to debug your programs. When computer grading was adopted the first time, a few students submitted many times and hoped to get lucky. We observed that a few students always received zero even after many submissions. Moreover, these students submitted multiple times within minutes. These students overloaded the computer grader and caused problems to the other students. Thus, we limit the number of submissions.  A student survey suggests that 10 submissions would be sufficiently. The limited submissions encourage students to seek help from teaching staff after several attempts. This is much better than blindly submitting meaningless code to test luck.

It has been observed that the computer grader is less forgiving than some other computers, even though the grading computer is a standard Linux computer. You have to run your programs on Linux machines. If your programs fail computer grading, we will provide as much information as possible. It is your responsiblity to fix your programs. We will never consider whether your programs "work" on your own computers, because we do not know what is on your computers.  The instructors will not accept any regrading requests because "my program works on my computer."  Your programs have to work on ECN-managed Linux and have to pass all testing tools. 

Please be aware that being lucky does not mean it is working.  Your program may have inconsistent behavior. The most common problem is uninitialized variables and accessing invalid memory addresses. When this occurs, the program's behavior is undefined.  Sometimes, it "works"; sometimes, it does not. If your program fails when it is graded, it fails.  It does not matter if it works when you test it. It does not matter how many times you test it. You need to fix the program.

Current State of Computer Grader

After you submit your program, you will see a message on the web site. A grade report is sent to your email address. The grade report contains much more information than the message on the web browser.  Our goal is to help you learn. Thus, if your program fails a test case, the test case is included in your grade report. You should study the grade report carefully. For each exercise or programming assignment, we have several hundred test cases. Every time you submit, a set of test cases are randomly chosen. It is very likely that each submission is graded by using different test cases. It is possible that you receive a lower score even if your program is improved, because different test cases are used.

It would not be wise to believe that your program is perfect and to blame the computer grader for failing to give you the full score. The computer grader has been used in several courses for several years. The computer grader is stable and quite accurate, even though bugs are still possible. There are many instances when students claim their programs to be "perfect" or "100% working" (based on students' words), but the teaching staff find serious problems in these students' code. If you believe your program is perfect and refuse the consider the possibility of mistakes, you will not find any problem in your program.

Common Problems in Submissions

Before claiming that your program is perfect and the computer grader is wrong, please check the following common mistakes.

• Incomplete submission. Do you include all needed files? If you miss a file, the computer grader cannot produce an executable and cannot grade your program. Move the zip file to an empty directory and unzip it. Check whether you can create the executable.
• Uninitialized variables or pointers. This is the most common mistake making a program behavior inconsistent. If a program uses unitialized variables or pointers, the program's behavior is undefined. Such a program sometimes "works" on the student's computer and fails by the computer grader. It has been observed that sometimes Windows + Cygwin (Windows + MinGW or Windows + Virtualbox + Linux) initializes variables or pointers to zero. However, this is not true in Linux. You should test your program on native Linux (such as the machines in EE206).  You should also check your program by using valgrind. You should initialize every variable or pointer to an invalid value (such as NULL for a pointer). Why? You guarantee that the program fails if you do not correct the values later. This is better than relying on your luck.
• Infiite loop. The computer grade stops your program after timeout.
• Non-stopping recursive calls and runs out of the stack memory.
• Using an excessive amount of memory. For some assignments, the computer grader limits the amount of memory your program can use. If your program uses too much memory, the computer grader will stop your program.

How to Prevent and Detect Bugs

Some people can finish each assignment within a few hours. For some people, an assignment can take forever. What is the difference?

• Write down your solution on paper before writing code in computer. If you do not have a solution, your program will not solve the problem magically.
• Have a clean design of your program. Don't write messy code. If it is hard to read, it is hard to be correct.
• Read your code before testing. Read your code line by line and carefully convince yourself that the program is doing what you want. If you do not understand what is written, your program will not work.
• Remove all gcc warnings. Some students ignore gcc warnings and debug forever. Before testing, correct your program and remove all warning messages.
• Use valgrind to detect memory errors. Some errors will not terminate your program and give you the impression that the program is working.
• Do not use "mysterious numbers."  It is easy to write down "4" if you think there are only 4 possible scenarios. The problem is that some time later, you discover the fifth case. You change one place to 5 and forget to change the others. Now, you have a program is very hard to debug because sometimes it works (if it is one of the first cour cases) and sometimes it fails (if it is the fifth case). What should you do? Create a symbol using

              #define NUMCASES 4

You will use only this symbol in the whole program. If you need to add the fifth case, you need to change only one place.