Students can solve programming exercises but not explain their solutions. What to do?

Aivar 08/25/2017. 7 answers, 10.928 views
curriculum-design resource-request grading assessments

I've seen too many cases when a student has completed a programming exercise (without cheating) but can't say, for example, what type of values a variable contains during program runtime (in case of Python programs).

It looks like the ability to solve exercises should not be the only (or even main?) criteria for assessment.

Do you know any research papers (or books or posts) that tackle this issue and/or recommend other assessment techniques / exercise types?

4 Tomáš Zato 08/24/2017
Well at my college, most programming courses involve submitting homeworks to automated validation system. This produces students who basically randomly type code until it passes. Only few subjects involve code review by the teacher.

7 Answers

ShawnMartin 08/24/2017.

If I understand your problem correctly, it's that students can create programs that behave correctly without understanding why they behave correctly. I assume that they do this by some combination of brute force trying things and SO search.

Using program correctness for grading has the desirable qualities of being objective and automation friendly.

So how could you keep these qualities while testing for actual understanding? While I agree that good code structure, variable naming, and possibly comments can offer proof of understanding, they are all highly subjective.

What I would do instead is construct exam exercises that are relatively small requirements changes to homework assignments that students have completed earlier, ideally at least 2 weeks earlier. Then give each student access to the code (s)he submitted for the homework as a starting point for completing the exam exercises. Tune the original homework size and the exam requirements changes so that a student who understands the code and has well-written code can complete the exam exercise quickly, but one that started from scratch would not likely be able to complete the work. This accomplishes 4 things:

  1. It solves your original problem of testing whether students understand the code they have written.
  2. It maintains the objectivity and automation friendliness of grading on program behavior.
  3. It rewards production of readable, maintainable code.
  4. It prepares students for the world of coding outside of a class environment, where the majority of time spent is not on isolated exercises, but extending an existing code base.

Disclaimer: I work in industry and have no research to support this.

Jay 08/25/2017.

I have seen my share of this 'program gets output' but the programmer has no clue how she/he got there. It's funny how that happens so many times.

This is what I have done to at least handle the issue.

Before assessments, I break down the evaluation to include the following.

  • Simply getting the output gets them the bare passing grade.
  • Those who can use meaningful names for their variables, function calls get extra.
  • Those that can back up their loops, method calls and variable declarations with expansive ( 2 to 3 lines at least) comments get extra points.
  • In addition to that, those that can write psuedocode above the program code (perhaps in a big block of comments) get some more.

Obviously, this makes me their least favourite faculty in terms of evaluation and gets me a lot of complaints but I stick with it.

Update : Please note that, during evaluation, I don't expect all of them to be done. Say, a student is already using meaningful names everywhere, I would not mark her/him down for not writing comments.

35 vacip 08/23/2017
"'program gets output' but the programmer has no clue how she/he got there" - Sounds like a successful day at the office to me.
9 zebonaut 08/23/2017
@vacip Hmmm - Sounds like a successful day for that one co-worker I had who became an Agile Coach some months ago. ("Hey I'm even documenting my work because I kick that Excel sheet long enough to get a PWM duty cycle of 180 % which sounds close enough to being OK!") Everyone is happy there is now less of a mess to clean up because the person now stays away from code and has decided it would be better to just ruin whiteboards by juggling around colorful sticky notes with meaningless buzzwords on them.
37 richard 08/23/2017
@Jay when I was in industry, I spent so much time trying to teach people not to fill there code with comment noise. Just use good names, your 2nd point. As for psuedocode. It was useful before we had compilers. And I suspect it could be useful when learning to code, if we do it first. It was useful for me, as it allowed me to write as fast as I was thinking, without having to worry about syntax, keywords, and library method names. But once you are thinking in a programming language it is time to drop it.
6 Neil Slater 08/23/2017
@richard: I think (hope) that the explanatory comments being marked here are in lieu of an interview or separate document explaining the student's thinking. Long pseudo-code comments would not get past code review where I work currently (in fact nearly any comment that explained what the code does as opposed to why). But the explanations and pseudo-code surely have a place for evaluating student submissions. The question is then, is adding marks for them supplied as comments suitable? It seems pragmatic solution to me, if the current problem is getting no explanation at all.
15 Tim 08/23/2017
I think 2 lines of comments for a variable declaration is wrong, quite simply. I like comments at the top of a function if the name doesn’t explain it (which it should) and wherever the code is confusing

richard 08/23/2017.

I am wondering how much of this is because they can not express in natural language (don't know terminology). How much is because of just fiddle until it works programming.

Learning to express in natural language

This is important, to allow them to communicate with a larger team, to allow them to look stuff up on an internet search, and to answer some of the exam questions.

Programming by fiddling / tinkering

This is an important technique to use some of the time. However it should then be followed by evaluation.

For example: Trying to find the angle of a triangle, when drawing using turtle graphics.

  • Made a first guess (common first guesses: 30, 60, 45, 90, 100 )
    • Try it.
    • Decide higher of lower (this is often decided wrong, but that is ok)
    • Guess a new value.
  • repeat until you find the answer.
  • record answer with result for square.

Table of angles:

| number of sides | angle |
|        3        |       |
|        4        |  90   |
|        5        |       |
|        6        |       |
|        7        |       |

You can discuss what the best strategy is [binary search]. One you have a completed table (4 or 5 entries), you can start to look for a pattern. It there a general formula for a polygon. Also getting pupils to pretend to be the turtle can help, or teacher is turtle. In ether case another pupil is the instructor, that tells turtle to move and turn. While doing this they gain a deeper understanding of how the code works.

Buffy 08/24/2017.

I'm afraid my answer here will suggest that you completely revamp how you teach.

The sort of problems that result in issues like this, seem to me to be problems that treat the computer as a fancy calculator. Problems given to students are of the "math-y" type. Some require tricky thinking, of course, but they are unlike the sorts of problems that people in the real world write programs to solve.

My suggestion, is to, instead, use simulation as the basis of your teaching. There are many ways to do this, but one of the best and easiest to introduce is the Greenfoot system. It provides both an IDE (for Java) and a simulation framework.

There is also an organization for users, Greenroom, who contribute simulation frameworks that you can start with and modify. There are hundreds of such simulations, some with teaching materials, even videos. Note that the Greenroom is a membership site. You will need to join.

Here are a few examples. I have used some of them, but not all.

Create Flappy Bird Game

Fuel Depot Question from APCS-A

Greeps - A Programming Competition One of the originals.

Karel J Robot meets Greenfoot A robot simulation - from the book.

2d Platformer Similar to Mario.

There are hundreds more, both at the main Greenfoot site and the Greenroom. Only the Greenroom requires (free) membership. With these sorts of frameworks and the programming that it involves, the problem you discuss simply won't arise.

It has been noted that I often promote Greenfoot. I'm not affiliated with the site or its developers, nor have I ever been. I know some of the developers and I have produced materials for use with Greenfoot. I also occasionally submit bug reports and feature requests to the site.

But Greenfoot aligns well with my general teaching philosophy and makes Object-Oriented Programming easy to teach to novices in an interesting and engaging way.

Talha Irfan 08/23/2017
Thanks a lot sir! Really helped.
Andrew T. 08/24/2017
Hi! I noticed that you mentioned and linked to 'Greenfoot' on some of your answers. Are you affiliated with it? If yes, you need to disclose the affiliation. Otherwise, you may need to put a disclaimer to prevent a misunderstanding of over-promotion. Thanks!
Buffy 08/24/2017
@AndrewT. I've added the disclaimer to the answer itself. Thanks for the suggestion.

ncmathsadist 08/23/2017.

It's time for a little "code review." Have a student present his code in front of class and talk about how he made it work. Hey, this happens in the professional world. There is no time like the present to begin learning this vital skill.

Talha Irfan 08/23/2017.

I also encounter this issue (just encountered it yesterday in a lab exam). This is how I differentiate between someone who has done his work and just not been able to explain it and someone who has crammed/cheated:

  1. Deleting some part of code and asking student to rewrite it
  2. Asking student to modify a snippet of code
  3. Explain this line

Usually Point 2 is enough to differentiate. Kindly ensure to let him/her sit calmly in order to not let nerves get hold of him/her - it works for me!

JarHead0844 08/23/2017.

Isn't this what lab reports are for. Computer Science is a science and should come with some basic rigors. Copying some code and hacking away might get an output, but forcing students to do analysis, flow charts, etc will prove how much they really understand the process. Even simple things like writing a paragraph on the theory and another on the basic algorithm to solve the problem would go a long way. - Download Hi-Res Songs

1 Alan Walker

Diamond Heart flac

Alan Walker. 2018. Writer: Alan Walker;Sophia Somajo;Mood Melodies;James Njie;Thomas Troelsen;Kristoffer Haugan;Edvard Normann;Anders Froen;Gunnar Greve;Yann Bargain;Victor Verpillat;Fredrik Borch Olsen.
2 Sia

I'm Still Here flac

Sia. 2018. Writer: Sia.
3 Cardi B

Taki Taki flac

Cardi B. 2018. Writer: Bava;Juan Vasquez;Vicente Saavedra;Jordan Thorpe;DJ Snake;Ozuna;Cardi B;Selena Gomez.
4 Little Mix

Woman Like Me flac

Little Mix. 2018. Writer: Nicki Minaj;Steve Mac;Ed Sheeran;Jess Glynne.
5 Halsey

Without Me flac

Halsey. 2018. Writer: Halsey;Delacey;Louis Bell;Amy Allen;Justin Timberlake;Timbaland;Scott Storch.
6 Lady Gaga

I'll Never Love Again flac

Lady Gaga. 2018. Writer: Benjamin Rice;Lady Gaga.
7 Bradley Cooper

Shallow flac

Bradley Cooper. 2018. Writer: Andrew Wyatt;Anthony Rossomando;Mark Ronson;Lady Gaga.
8 Bradley Cooper

Always Remember Us This Way flac

Bradley Cooper. 2018. Writer: Lady Gaga;Dave Cobb.
9 Kelsea Ballerini

This Feeling flac

Kelsea Ballerini. 2018. Writer: Andrew Taggart;Alex Pall;Emily Warren.
10 Mako

Rise flac

Mako. 2018. Writer: Riot Music Team;Mako;Justin Tranter.
11 Dewain Whitmore

Burn Out flac

Dewain Whitmore. 2018. Writer: Dewain Whitmore;Ilsey Juber;Emilio Behr;Martijn Garritsen.
12 Avril Lavigne

Head Above Water flac

Avril Lavigne. 2018. Writer: Stephan Moccio;Travis Clark;Avril Lavigne.
13 Khalid

Better flac

Khalid. 2018. Writer: Charlie Handsome;Jamil Chammas;Denis Kosiak;Tor Erik Hermansen;Mikkel Stoleer Eriksen;Khalid.
14 Lady Gaga

Look What I Found flac

Lady Gaga. 2018. Writer: DJ White Shadow;Nick Monson;Mark Nilan Jr;Lady Gaga.
15 Deep Chills

Run Free flac

Deep Chills. 2018.
16 Dynoro

In My Mind flac

Dynoro. 2018. Writer: Georgi Kay;Feenixpawl;Ivan Gough.
17 Charli XCX

1999 flac

Charli XCX. 2018. Writer: Charli XCX;Troye Sivan;Leland;Oscar Holter;Noonie Bao.
18 NCT 127

Regular (English Version) flac

NCT 127. 2018.
19 Lukas Graham

Love Someone flac

Lukas Graham. 2018. Writer: Don Stefano;Morten "Rissi" Ristorp;Morten "Pilo" Pilegaard;Jaramye Daniels;James Alan;David LaBrel;Lukas Forchhammer Graham.
20 Rita Ora

Let You Love Me flac

Rita Ora. 2018. Writer: Rita Ora.

Related questions

Hot questions


Popular Tags