Creating and grading assignments

This guide walks an instructor through the workflow for generating an assignment and preparing it for release to students.

Accessing the formgrader extension

The formgrader extension provides the core access to nbgrader's instructor tools. After the extension has been installed, you can access it through the tab in the notebook list:

Creating a new assignment

From the formgrader

To create a new assignment, open the formgrader extension and click the "Add new assignment..." button at the bottom of the page. This will ask you to provide some information such as the name of the assignment and its due date. Then, you can add files to the assignment and edit them by clicking the name of the assignment:

From the command line

To simplify this example, two notebooks of the assignment have already been stored in the source/ps1 folder:

• source/ps1/problem1.ipynb
• source/ps1/problem2.ipynb

Developing assignments with the assignment toolbar

Note: As you are developing your assignments, you should save them into the source/{assignment_id}/ folder of the nbgrader hierarchy, where assignment_id is the name of the assignment you are creating (e.g. "ps1").

Once the toolbar has been installed, you should see it in the drop down "View -> Cell Toolbar" menu:

Selecting the "Create Assignment" toolbar will create a separate toolbar for each cell which by default will be a dropdown menu with the "-" item selected. For markdown cells, there are two additional options to choose from, either "Manually graded answer" or "Read-only":

For code cells, there are four options to choose from, including "Manually graded answer", "Autograded answer", "Autograder tests", and "Read-only":

The following sections go into detail about the different cell types, and show cells that are taken from a complete example of an assignment generated with the nbgrader toolbar extension:

• source/ps1/problem1.ipynb
• source/ps1/problem2.ipynb

"Manually graded answer" cells

If you select the "Manually graded answer" option (available for both markdown and code cells), the nbgrader extension will mark that cell as a cell that contains an answer that must be manually graded by a human grader. Here is an example of a manually graded answer cell:

The most common use case for this type of cell is for written free-response answers (for example, which interpret the results of code that may have been written and/or executed above).

Note: the blue border only shows up when the nbgrader extension toolbar is active; it will not be visible to students.

“Manually graded task” cells

If you select the “Manually graded task” option (available for markdown cells), the nbgrader extension will mark that cell as a cell that contains the description of a task that students have to perform. They must be manually graded by a human grader. Here is an example of a manually graded answer cell:

The difference with a manually graded answer is that the manually graded tasks cells are not edited by the student. A manually or automatically graded cell ask students to perform a task in one cell. A manually graded task asks students to perform a task with cells.

The common use case for this type of cell is for tasks that require the student to create several cells such as "Process the data and create a plot to illustrate your results." or to contain notebook-wide tasks such as "adhere to the PEP8 style convention."

Note: the blue border only shows up when the nbgrader extension toolbar is active; it will not be visible to students.

“Manually graded task” cells with mark scheme

A mark scheme can be created through the use of a special syntax such as === BEGIN MARK SCHEME === and === END MARK SCHEME ===. The section of text between the two markers will be removed from the student version, but will be visible at the grading stage and in the feedback.

"Autograded answer" cells

If you select the "Autograded answer" option (available only for code cells), the nbgrader extension will mark that cell as a cell that contains an answer which will be autograded. Here is an example of an autograded graded answer cell:

Unlike manually graded answers, autograded answers aren't worth any points: instead, the points for autograded answers are specified for the particular tests that grade those answers. See the next section for further details.

Note: the blue border only shows up when the nbgrader extension toolbar is active; it will not be visible to students.

"Autograder tests" cells

If you select the "Autograder tests" option (available only for code cells), the nbgrader extension will mark that cell as a cell that contains tests to be run during autograding. Here is an example of two test cells:

The lock icon on the left side of the cell toolbar indicates that the tests are "read-only". See the next section for further details on what this means.

Note: the blue border only shows up when the nbgrader extension toolbar is active; it will not be visible to students.

"Autograder tests" cells with hidden tests

Tests in "Autograder tests" cells can be hidden through the use of a special syntax such as ### BEGIN HIDDEN TESTS and ### END HIDDEN TESTS, for example:

"Read-only" cells

If you select the "Read-only" option (available for both code and markdown cells), the nbgrader extension will mark that cell as one that cannot be modified. This is indicated by a lock icon on the left side of the cell toolbar:

This functionality is particularly important for test cells, which are always marked as read-only. Because the mechanism for autograding is that students receive full credit if the tests pass, an easy way to get around this would be to simply delete or comment out the tests. This read-only functionality will reverse any such changes made by the student.

Validating the instructor version

From the validate extension

Ideally, the solutions in the instructor version should be correct and pass all the test cases to ensure that you are giving your students tests that they can actually pass. To verify this is the case, you can use the validate extension:

If your assignment passes all the tests, you'll get a success pop-up:

If it doesn't pass all the tests, you'll get a message telling you which cells failed:

From the command line

You can also validate assignments on the command line using the nbgrader validate command:

Generate and release an assignment

From the formgrader

After an assignment has been created with the assignment toolbar, you will want to generate the version that students will receive. You can do this from the formgrader by clicking the "generate" button:

This should succeed with a pop-up window containing log output:

From the command line

{course_directory}/source/{assignment_id}/{notebook_id}.ipynb

Note: The student_id is not included here because the source and release versions of the assignment are the same for all students.

After running nbgrader generate_assignment, the release version of the notebooks will be:

{course_directory}/release/{assignment_id}/{notebook_id}.ipynb

As a reminder, the instructor is responsible for distributing this release version to their students using their institution's existing student communication and document distribution infrastructure.

When running nbgrader generate_assignment, the assignment name (which is "ps1") is passed. We also specify a header notebook (source/header.ipynb) to prepend at the beginning of each notebook in the assignment. By default, this command should be run from the root of the course directory:

Preview the student version

After generating the student version of assignment, you should preview it to make sure that it looks correct. You can do this from the formgrader extension by clicking the "preview" button:

Under the hood, there will be a new folder called release with the same structure as source. The release folder contains the actual release version of the assignment files:

• release/ps1/problem1.ipynb
• release/ps1/problem2.ipynb

If you are working on the command line, you may want to formally verify the student version as well. Ideally, all the tests should fail in the student version if the student hasn't implemented anything. To verify that this is in fact the case, we can use the nbgrader validate --invert command:

If the notebook fails all the test cases, you should see the message "Success! The notebook does not pass any tests."

Releasing files to students and collecting submissions

submitted/{student_id}/{assignment_id}/{notebook_id}.ipynb

Please note: Students must use version 3 or greater of the IPython/Jupyter notebook for nbgrader to work properly. If they are not using version 3 or greater, it is possible for them to delete cells that contain important metadata for nbgrader. With version 3 or greater, there is a feature in the notebook that prevents cells from being deleted. See this issue for more details.

To ensure that students have a recent enough version of the notebook, you can include a cell such as the following in each notebook of the assignment:

import IPython
assert IPython.version_info[0] >= 3, "Your version of IPython is too old, please update it."

Autograde assignments

In the following example, we have an assignment with two notebooks. There are two submissions of the assignment:

Submission 1:

• submitted/bitdiddle/ps1/problem1.ipynb
• submitted/bitdiddle/ps1/problem2.ipynb

Submission 2:

• submitted/hacker/ps1/problem1.ipynb
• submitted/hacker/ps1/problem2.ipynb

From the formgrader

You can autograde individual submissions from the formgrader directly. To do so, click on the the number of submissions in the "Manage Assignments" view:

This will take you to a new page where you can see all the submissions. For a particular submission, click the "autograde" button to autograde it:

After autograding completes, you will see a pop-up window with log output:

And back on the submissions screen, you will see that the status of the submission has changed to "needs manual grading" and there is now a reported score as well:

From the command line

We can run the autograder for all students at once from the command line:

When grading the submission for Bitdiddle, you'll see some warnings that look like "Checksum for grade cell correct_squares has changed!". What's happening here is that nbgrader has recorded what the original contents of the grade cell correct_squares (when nbgrader generate_assignment was run), and is checking the submitted version against this original version. It has found that the submitted version changed (perhaps this student tried to cheat by commenting out the failing tests), and has therefore overwritten the submitted version of the tests with the original version of the tests.

You may also notice that there is a note saying "ps1 for Bitdiddle is 21503.948203 seconds late". What is happening here is that nbgrader is detecting a file in Bitdiddle's submission called timestamp.txt, reading in that timestamp, and saving it into the database. From there, it can compare the timestamp to the duedate of the problem set, and compute whether the submission is at all late.

Once the autograding is complete, there will be new directories for the autograded versions of the submissions:

autograded/{student_id}/{assignment_id}/{notebook_id}.ipynb

Autograded submission 1:

• autograded/bitdiddle/ps1/problem1.ipynb
• autograded/bitdiddle/ps1/problem2.ipynb

Autograded submission 2:

• autograded/hacker/ps1/problem1.ipynb
• autograded/hacker/ps1/problem2.ipynb

Manual grading

After running nbgrader autograde, the autograded version of the notebooks will be:

autograded/{student_id}/{assignment_id}/{notebook_id}.ipynb

We can manually grade assignments through the formgrader as well, by clicking on the "Manual Grading" navigation button. This will provide you with an interface for hand grading assignments that it finds in the directory listed above. Note that this applies to all assignments as well -- as long as the autograder has been run on the assignment, it will be available for manual grading via the formgrader.

Generate feedback on assignments

autograded/{student_id}/{assignment_id}/{notebook_id}.ipynb

Creating feedback for students is divided into two parts:

• generate feedback
• release feedback

Generating feedback will create HTML files in the local instructor directory. Releasing feedback will copy those HTML files to the nbgrader exchange.

We can generate feedback based on the graded notebooks by running the nbgrader generate_feedback command, which will produce HTML versions of these notebooks at the following location:

feedback/{student_id}/{assignment_id}/{notebook_id}.html

The nbgrader generate_feedback is available by clicking the Generate Feedback button on either the Manage Assignments view (to generate feedback for all graded submissions), or on the individual student's Manage Submission page (to generate feedback for that specific individual).

We can release the generated feedback by running the nbgrader release_feedback command, which will send the generated HTML files to the nbgrader exchange.

The nbgrader release_feedback is available by clicking the Release Feedback button on either the Manage Assignments view (to release feedback for all generated feedback), or on the individual student's Manage Submission page (to release feedback for that specific individual).

Workflow example: Instructor returning feedback to students

In some scenarios, you may not want to (or be able to) use the exchange to deliver student feedback. This sections describes a workflow for manually returning generated feedback.

In the following example, we have an assignment with two notebooks. There are two submissions of the assignment that have been graded:

Autograded submission 1:

• autograded/bitdiddle/ps1/problem1.ipynb
• autograded/bitdiddle/ps1/problem2.ipynb

Autograded submission 2:

• autograded/hacker/ps1/problem1.ipynb
• autograded/hacker/ps1/problem2.ipynb

Generating feedback is fairly straightforward (and as with the other nbgrader commands for instructors, this must be run from the root of the course directory):

Once the feedback has been generated, there will be new directories and HTML files corresponding to each notebook in each submission:

Feedback for submission 1:

• feedback/bitdiddle/ps1/problem1.html
• feedback/bitdiddle/ps1/problem2.html

Feedback for submission 2:

• feedback/hacker/ps1/problem1.html
• feedback/hacker/ps1/problem2.html

If the exchange is available, one would of course use nbgrader release_feedback. However if not available, you can now deliver these generated HTML feedback files via whatever mechanism you wish.

Getting grades from the database

In addition to creating feedback for the students, you may need to upload grades to whatever learning management system your school uses (e.g. Canvas, Blackboard, etc.). nbgrader provides a way to export grades to CSV out of the box, with the nbgrader export command:

After running nbgrader export, you will see the grades in a CSV file called grades.csv: