Specifications-Grading: An Example

Sometimes Specifications-Grading (Nilson, 2015) can feel like cooking – I may have all the ingredients, but it doesn’t mean I can turn it into an edible product.  Bouncing ideas off other colleagues has been extremely beneficial.  In this post, I will discuss an implementation I used for an intermediate statistics course.

Setting up the Hurdles

When introducing the framework to students, I emphasize that substandard statistical analyses can have significant consequences (from financial loss to loss of life).

The course is designed to reflect the importance of high-quality work.  Each assignment is graded as “successfully completed” or not (no partial credit is awarded).  To help you successfully complete each assignment, very clear expectations will be provided.

The course consists of four modules, each covering a different modeling framework (Linear Models, Nonlinear Models, Repeated Measures, Survival Analysis).  The grade in the course is tied to the proficiency level within each module; the proficiency scale (adapted from the NIH Competencies Proficiency Scale) is given below:

1. Fundamental Awareness: 
   • Successfully complete all Homework Assignments for the module.
   • Successfully complete the Article Review.
2. Novice:
   • All requirements for Fundamental Awareness proficiency.
   • Successfully complete the Concept Check.
3. Intermediate:
   • All requirements for Novice proficiency.
   • Successfully complete the Module Quiz.
4. Advanced:
   • All requirements for Intermediate proficiency.
   • Successfully complete the Analysis Task.

Course grades are then earned according to the following definitions:

Grade	Requirements
A	Advanced proficiency in all 4 modules. Successfully complete the Case Study
B	Advanced proficiency in 3 modules; Intermediate proficiency in the remaining. Successfully complete the Case Study.
C	Advanced proficiency in 1 module; Intermediate proficiency in 2; Novice proficiency in the remaining.
D	Intermediate proficiency in 1 module; Novice proficiency in 2; Fundamental Awareness in the remaining.
F	Fail to meet requirements for D

In this structure, the assignments support the course learning objectives.  Homework assesses lower-level learning (group work, very directed) while an Analysis Task or Case Study supports higher-level learning (individual work, open-ended).  Higher grades are earned by completing more work and work which is at a higher level (“more hurdles, higher hurdles”).  Note that this structure also requires students to have some exposure to all material in the course, even if they do not demonstrate mastery of all material.  An alternative would be to define a D as Advanced proficiency in 1 module, a C as Advanced proficiency in 2, etc.  This emphasizes depth over breadth in the course.  As this course is not a pre-requisite for others, I have fewer constraints.  With courses in a pre-requisite chain, a D should represent the minimum knowledge required to continue in the sequence.

Implementing Specifications-Grading is often easiest if you divide the course into modules and create a “bundle” of work for each.  While this class has four modules, my introductory course has 9 (3 for one-sample inference, 3 for regression, 3 for ANOVA).

Defining “Successful” Completion

When defining specifications for each assignment, it is helpful to think about the role the assignment plays.  In this class, homework is meant to expose students to the terminology and practice the skills from lectures; assessment happens on Concept Checks (explaining a single key concept from the class), Module Quizzes (auto-graded with multiple choice and computation questions to assess fundamental skills), and Analysis Tasks (analyze provided data to address a research question).  The specifications reflect the Homework’s role as learning from feedback:

Successful completion is awarded if all questions are attempted, and the solution makes use of content from the course.

Similarly, the learning for Article Reviews happens with in-class discussions.  The specifications ensure students read the article in advance:

Successful completion will be awarded if the article is scored on each section of the provided rubric and at least one comment is made justifying the score, citing specific evidence from the paper.

The minimal requirements encourage all students to complete these assignments as they establish literacy with the course topics.  For the Module Quiz, the specifications require that students answer 100% of the quiz correctly; however, they are provided two attempts to do so.  Administered via the LMS, the quiz informs the students of which questions were missed to encourage them to discuss the content with me before their second attempt.

A Case Study is reserved for distinguishing higher grades in the course and asks students to choose an article in the literature with data provided and replicate an aspect of the analysis associated with our course.  This assignment has 31 specifications covering everything from the formatting and number of allowable grammatical errors to ensuring the analysis provided is adequate for addressing the question of interest and describing the conclusion in the context of the problem.  These thorough specifications reflect the level of learning students are demonstrating.

The tricky part is an Analysis Task, where there are a variety of problems to solve.  In that case, I have general specifications for each type of problem that might be present.  For example, the following specifications are for questions asking students to state the hypotheses:

• Define the parameter(s) in the context of the problem and state the associated mathematical symbol representing the parameter.
• Use the correct equality/inequality in each of the null and alternative hypotheses.
• The hypotheses should correspond to the research objective.

For each set of specifications, I provide at least 3 examples: 1 which meets all specifications and 2 which do not.  Each example is annotated to explain how specifications are met/violated.  The bulk of my preparation when first teaching a course is spent developing these examples, but it helps in clearly communicating the specifications to students.  I then reuse these when building future courses.

I am fortunate to teach at most 80 students in a single term across all courses.  If I needed to scale these ideas to larger classes, I would heavily rely on peer review.  I also find it helpful to leverage the course LMS for keeping track of tasks performed and revisions.

For as many fears of pass/fail grading as we may have, students will have more. To ease these fears, I use the phrases “successfully complete”/“do not successfully complete” instead of pass/fail when describing the framework to students. I also emphasize that “successfully complete” does not mean perfect, which brings us to the heart of Specifications-Grading – defining “successful completion.”

Flexibility

I recognize life happens.  We each have individual priorities, and they may not align with the course schedule, at which point, the grading structure may feel overwhelming.

I introduce “outliers” using the above language.  I emphasize the importance of beginning work early, but I allow students to declare 4 assignments as “outliers” which may be revised and resubmitted provided a good-faith effort is given on the initial submission as revisions are a chance to learn from mistakes; they are not a license to submit sub-standard work. 

I use a very similar framework in many of my upper-level courses, and I find that students in these courses have a much easier time adjusting to a Specifications-Grading system.  The hardest part for me is accepting that not all students want an A in the course.  That said, I work with each student on their mastery of the content they have chosen to complete, and that is a conversation I am willing to have anytime.

Resources

If you’re interested in getting started with specifications grading, check out my specifications grading website where I have posted my presentations on the topic, along with course design elements for an introductory statistics course, an intro to data science type course, an intermediate biostatistics course, an advanced mathematical statistics course, and a capstone experience.

References

National Institutes of Health. Competencies Proficiency Scale. https://hr.nih.gov/working-nih/competencies/competencies-proficiency-scale

Nilson, L. B. (2015).  Specifications Grading: Restoring Rigor, Motivating Students, and Saving Faculty Time.  Stylus.