Takeaways From My Latest Project

I recently shipped a new exercise on Khan Academy called Visualizing Derivatives. The exercise seeks to test students’ ability to intuit and visualize how a derivative relates to a function.

To users, the exercise appears fairly straightforward, but I found it to be an interesting and nicely-scoped engineering challenge. I was fortunate enough to get some things at least somewhat right from the beginning, which I’ll share here. Plenty of things didn’t go so nicely. I write about one big lesson I learned in the next post.

1. Iterate the design on paper before writing a line of code

Iterating on paper isn’t a particularly new or sexy concept, but I’ve found it incredibly helpful every time I’ve done it. 15 minutes of sketching can save me hours of coding. I choose paper instead of computer-based tools because (1) most easy-to-use mock-up tools aren’t designed for the “interactive exercise” use-case, and (2) drawing low-fidelity sketches is fast and intuitive – I can try out lots of different ideas really quickly.

Another reason I like to sketch a design on paper is that it sometimes helps me visualize and keep track of different constraints and edge cases that I’ll deal with later. In the case of Visualizing Derivatives, my sketching helped me narrow down the different types of function segments I’d include.

Sketch of function segments

Sal (Salman Khan, the founder/executive director/headmaster of Khan Academy) had previously made a video spec of the exercise, which was a really helpful start.

A screenshot from Sal’s video spec

Through discussions with Sal and exercises guru Ben Eater, the design evolved into one using an interactive “sliding window” containing the graph to be matched. While it might have been easier to generate the multiple-choice version, we thought the new version might be more fun and less game-able. Which brings me to my next point…

2. Think in advance, “How can this be gamed?”

When I started working on this exercise, I had recently visited one of our pilot schools, Summit, a charter school in San Jose, CA. It was my first school visit and was a really eye-opening view into the messy real-world conditions in which Khan Academy is being used.

For one, many students were using Internet Explorer or Firefox on laptops or netbooks with crappy, dimly-lit (to preserve power) screens. A far cry from using Chrome on the crisp, bright Macbook screen I have the distinct privilege to develop on. KhanUtil.ORANGE looked more like yellow, and text using that color was hard to read.

Summit San Jose (Photo courtesy Bilal)

I observed how some students made progress in exercises without necessarily demonstrating understanding of the underlying concepts. The practice of “pattern matching” is something that Ben Eater and Sal had mentioned on several occasions, but seeing some of it happening firsthand made a deeper impression on me. On the product side, we do have some features that reduce the risk of pattern matching (a context-switching feature we call “power mode” that draws problems from multiple exercises) and our analytics team is developing others (e.g., using Item Response Theory to determine how well an exercise discriminates between proficient and non-proficient users). But on the content side, we still need to try to develop difficult-to-game exercises.

For many multiple-choice questions, that can mean adding a “no solution” and/or “there is not enough information to solve the problem” option, inspired by the data sufficiency questions in the GMAT math section. In the case of Visualizing Derivatives, I worried that if a valid solution was guaranteed to be present, users would memorize the derivatives of just a few segments and have an easy time of guessing the answer from the handful that they knew.

Applying this to Visualizing Derivatives

So, I set out to make a “bogus” problem graph that partially matched up with the full graph. For Visualizing Derivatives, making a convincing “no solution” option was easier said than done. I ended up building a first pass of the exercise that didn’t have this feature (although it was always part of the plan), then realized that a not-insignificant refactor would allow me to use the same code for generating the original piecewise function and the bogus problem graph, helping me stay DRY.

On Sal’s suggestion, I also made the exercise harder by switching up the task for the user. In one “problem type”, I ask the user to match up a piece of the derivative with a full graph of the original function. In another, I ask the user to match up a piece of the original function with a full graph of the derivative.

Thinking in advance about how the exercise could be gamed helped me make design decisions that simplified the process of adding these anti-gaming features.

Got this far? Congrats! Still not totally bored? Check out the next post. :)