gradebook automation with TextExpander and Keyboard Maestro

in

i'm teaching my writing seminar course for the fifth time this semester. you would think that would mean that the class is running at peak efficiency. in terms of what goes on in the classroom, it is. but there's one big difference behind the scenes: the last time i taught the course (in 2011), i hadn't yet been converted to TextExpander, Hazel, or Keyboard Maestro. and now i have them all at my disposal.

the old system: a sliding rubric spreadsheet

gradebook-slider.gif

i teach a writing seminar, so the bulk of the assignments are essays. i give letter grades on these, but don't assign points to them. however, by the time the semester is done, students have written 6 or 7 short essays. i could eyeball 6 letter grades and try to come up with a fair average for everybody, but i know it wouldn't come out equal. so for equanimity and ease of computation, i convert all the letter grades to numbers on the back end. i could just apply a fixed rubric and be done, but i like to have a little bit more control over the grade distribution — again, without inadvertently creating bias towards one student or another.

to accomplish this, i set up a two-table spreadsheet in Numbers, one with students' scores, and one with a rubric. last year i waited until the entire semester was over and then ran a bunch of manual find-and-replace commands to link each grade cell to the corresponding rubric cell. the result is the only actually useful implementation of Numbers' slider cells that i've ever found. as you can see easily from the conditional formatting that i've put on the "paper average" column, tweaking individual paper grades triggers related changes in the course grades.

the automation: enter TextExpander

this semester, i decided that i didn't want to wait until all the papers had been collected and graded to be able to see the quantitative results. to do that, i needed a fast way to create a reference to the rubric cells. there's a mousey way (select the grade cell, type =, click the rubric cell, type enter), but it's slow and error-prone. instead, i decided to write a quick TextExpander script snippet that would give me a fill-in field where i could just type in the grade and have the cell reference generated automatically. it's not the prettiest thing ever — just a manual lookup in a hash — but it's fast and effective. if i accidentally mistype a grade, it returns no output, which is better than mis-clicking a rubric cell or bringing down the wrath of Numbers' formula editor in more creative ways. now i just type "gbg" (for gradebook grade) and i get this form:

gradebook snippet.gif

in testing, i can enter a full column of grades for my 18 students in 60–90 seconds. way better than waiting until May to sort it out.

more column-filling: attendance

i also keep a separate Numbers spreadsheet for attendance. i admittedly print this spreadsheet out (blasphemy, i know!) because making tickmarks with a pencil as my students enter the room is still the quickest, least intrusive way of keeping track. at the end of each week, i transfer the information back to my spreadsheet, which counts up absences and tardies and counts them against my students' allowable absences (3 per semester; tardies count as 1/3 of an absence).

again i was looking for the fastest way to transfer this information. that involves marking absences X and tardies T and then filling in the rest with checkmarks ✔. the simplest manual solution is to fill the whole column with checkmarks and then insert the X's and T's, but all the extra symbols are pretty visually distracting. then i wind up doing a lot of copy-pasting and working with Numbers' very finicky drag-to-fill operation.

it's almost certainly overkill, but i decided to build a Keyboard Maestro macro to do the filling for me. the method is simple:

  1. grab the contents of the current cell
  2. see if it matches T or X
  3. if not, fill it with ✔
  4. move down one cell
  5. repeat until it hits the bottom (current cell = ✔)

the format of the macro is below. i'm sure there's a better way (perhaps involving Applescript?), but i was happy for the practice building some more complex macros. and i'm guessing that it could be adapted to several other data-entry scenarios where the application isn't directly scriptable.

km attendance macro.gif

it's all about repetition

teaching and learning are all about repetition. for the instructor, there are always tasks that have to be iterated across students, or across class meetings. the best we used to be able to do was build muscle memory. now we can build tools.

a couple other things that i've automated so far this year that i may post on later include exporting Keynote slides to PDFs and sharing them, and batch-processing assignment files downloaded from course management software. even if i'm not coming out time ahead, it keeps the process fresh.

we've already lost the filesystem

in ,

or: How I Haven't Learned to Stop Worrying and Love the Cloud.

there's a persistent background of fear in the Apple community. even when rationality ought to prevail (as it most certainly did in John Gruber's written and spoken coverage of the Mountain Lion announcement), paranoia rushes in to fill the void. "it's only a matter of time", the devil perched on your shoulder says, "before they take away the filesystem. for good. everywhere."

an aside for all those about to go Van Hœt on this: what we're talking about is the filesystem metaphor, i.e. that the stuff on your computer is stored in files which get organized into folders. this is separate from the filesystem qua filesystem, the method by which a computer actually stores, organizes, and finds data on disk. Hypercritical e56 is a nicely comprehensive review of these, and from 1:42:00 to the end deals with the actuality vs. metaphor distinction. to disambiguate, John Siracusa uses the term "file hierarchy", which i think is certainly apt, and points out that one of the major reasons that its demise might be imminent and even welcome is the fact that most people simply cannot grok it. this is certainly true. ten years on, my parents still haven't quite grasped where /Applications is in OS X. they had no trouble with the file hierarchy in Classic Mac OS — just bung stuff wherever you want it, as long as you don't touch the System Folder. so the question is, if the file hierarchy ceases to exist, will they lose anything? i think the answer is yes, but perhaps less than most people think, and for a different reason than most people think: namely, that we've already lost a good bit of the meaningful hierarchy.

the case of the missing calendars

last fall, like many people, i updated to iOS 5. also, i wasn't an early adopter of Lion — i waited until 10.7.2 came out and then did one epic afternoon of software updates. i mostly did this to try to ensure a smooth transition to iCloud. despite having to actually set up iCloud from the bleachers at a hockey game because i didn't believe that the full wipe and restore of my phone would take as long as iTunes warned me it would, everything seemed to go perfectly smoothly. until a couple months later.

one day i was idly playing around with the ⌘4 / heat map / year view in iCal and noticed that 2009 and 2008 looked a little … sparse. (my iCal universe begins on April 6, 2008, due to massive a hard drive catastrophe that happened then. back up, kids!) after digging a little deeper, i noticed that some calendars, despite professing that they were merrily synced to the cloud, had zero events in them. oof. for whatever reason, iCloud had been benevolent and only disappeared the contents of calendars that were no longer in active use; otherwise it would have been a code red disaster. as it was, it was an inconvenience that left a hole in my personal digital history, which i now try to preserve a little better than i did in March 2008. i filed it away as a problem to be dealt with later. perhaps i did that because i knew it would be no simple task to get them back.

backups are only as good as the files they contain

what i was up against was another concern of power users everywhere: (1) the cloud is canon, (2) the cloud is now, (3) the cloud is only now. in this case, iCloud was very sure that those calendars weren't supposed to have any events. that's how things are supposed to be now, because those are the instructions the cloud received, be it in error or good faith. and iCloud has no idea if that's how things used to be, or whether those calendars ever used to have any events in them. the fact that i had a human, fleshy, grey matter memory that i definitely put things there and used to be able to look at them was really of no concern to the cloud. as i learned in the 2008 hard drive crash, wistful memories don't bring data back from the dead, they just make you feel like crap.

but all was not lost. those now-cloudy calendars were once local calendars, back when the canonical version lived inside my apartment, instead of in North Carolina. and i have Time Machine backups, so they must be preserved. it's just a matter of … finding them.

sidebar: i have considered myself a power user for a long, long time — my entire adult life, certainly. restoring a few files from a backup should be trivial, not throw me into a rage. but when there are as many layers of pain as there were between me and those files, there was no other possible reaction. bear in mind that this task would be impossible for an average user, and as i'll point out along the way, near impossible for a third party trying to help an average user. for a power user, it was merely maddeningly difficult.

ok, let's do this! first, where do those calendars live? must be somewhere in ~/Library …which is of course now hidden in Lion. no problem, ⌘⇧G will get me there in no time. from there it'll just be a quick hop into the Time Machine, rescue the appropriate calendars, and— oh, hell. ~/Library/Calendars looks like this:

~/Library/Calendars

~/Library/Calendars

we have just exited the universe of the filesystem metaphor. there is next to nothing here that is human-readable. perhaps i was supremely naïve to believe that my Typology calendar would be named something sensible like Typology.ics, or at least be a folder called Typology.calendar. instead i was staring down 32-character hex hashes. (note the Finder's hilariously awful attempt to sort these by name. since it tries to use logical base-10 numerical sorting, clearly a file that starts with 127 should come before one that starts 0259!)

i was not deterred. i went digging and found that each folder had an Events subfolder, which contained each event in its own self-contained .ics file. an empty Events folder indicated that i'd found one of my raptured calendars. there was also an Info.plist file. i started cracking these open, figuring that they had to include important metadata like, you know, the name of the bloody calendar. rinse and repeat for my 33 calendars. some time and scratch paper later, i'd compiled a list of dead calendars matched with the first four digits of their hash. now, certainly, i could step into the Time Machine.

since i access my Time Capsule over wifi, this is excruciating in and of itself. two minutes "Connecting to backup volume …". another three minutes remembering that any previous states of this folder exist, during which i can't do anything else with my Mac. (a necessary design choice when Time Machine was first introduced. but please, Apple, make this thing behave like a proper full-screen app in Lion.) all the while i'm not being soothed by my slow drift through some sort of nebula or star cluster, and the all-important navigation bar on the right side of the screen has rendered improperly on my external display. this is well past the point of giving up for most people.

stepping backward through backups proves incredibly slow and massively unhelpful — my calendars are modified between nearly every backup. so i get to play guess and check with when it was that i made the switch to iCloud. i finally find it! here we go, time to bring back those calendars when i realize, son of a bitch all the local calendar hashes are different. 33 new sequences of hex digits, totally unrelated to my notes and the previous work i've done. i am no longer surprised that iCloud lost track of some of my data.

this is also the stage where the endeavor becomes impossible even for a user being assisted by someone else, say a Mac Genius. first, the user would have to take their Mac and Time Capsule to the Apple store. then the Genius would have to get to this point. then, together, they would have to go through every calendar folder through the Time Machine interface, which is about as fast as working with floppy disks, handcuffed by only being able to use quick look to ascertain what's what. i found out that quick look-ing the .ics files was faster than the .plist files, so i did that. i wrote down another set of 4-digit hex prefixes on my scrap sheet, selected those folders and finally, at long last, hit Restore. after clearing the iCal cache and forcing it to rebuild its database, my events were back. not in the cloud, but back, on my hard drive…somewhere.

i don't want to have to be lucky

there's no question, i got lucky in this scenario. the snafu happened in the transition from local to cloud copies of my calendars. i had backups. i had knowledge and a hell of a lot of persistence. but what about the person who buys their first Mac today and always lives in the cloud? when their data goes missing, who's going to be able to save it? it'd be like waking up one morning to find that your car was missing from your garage. you call the police to report it, but once you give them your license plate number and VIN, they tell you that they have no record of the registration and that in fact, the vehicle had never even been produced by the manufacturer. this is nightmare, Kafka-type stuff.

everybody, power-user or not, should want to be able to lay a hand on their data, in both the literal and metaphorical sense: "look here, i hold in my hand a drive that contains my data" and "look here, i'm pointing with my cursor at the representation of that data". if all our data lives inside apps, all our apps have to be able to talk directly to our incremental backups. a couple Apple apps do this — Mail.app, for one — but the vast majority don't.

losing a file hierarchy might be a pain for some power users. but plenty of those power users use a flattened system for the one place they have more data go in and out on a daily basis than any other: their gmail archive. gmail tags are a flat storage system (ironically, to be compatible with IMAP, which uses folder hierarchies, google had to hack a flattening mechanism on top of the hierarchy. actually, the same is true for iCloud — anything saved on an HFS+ drive necessarily lives in a directory, even if it's invisible, unaccessible, and miserably named.) i am all for organizing my stuff in the most efficient way. even though i'm a file hierarchy junkie, i usually cut through most of the layers by using LaunchBar. my files live in a semi-flat representation in my mind already.

fortunately, i don't have to conclude this by predicting only doom and gloom. iCloud is gravely unsatisfactory in terms of keeping my data safe, because it's dumb about backups and previous states. thankfully, we have an alternative: Dropbox. no matter what happens in Cupertino, it's a relatively safe bet that Dropbox is not going anywhere in the foreseeable future. and they provide two very, very important features. first, file hierarchy is their bread and butter. even on a device where the local file hierarchy is totally hidden, like in iOS, any app with Dropbox access lets you browse cloud-based files and folders. but most of all, they preserve previous states of everything, for a full 30 days even on free accounts. if something gets disappeared — through a bug, through human error, whatever — they will not deny that it ever existed.

so is Apple protecting our data? my answer on a file level, and John Siracusa's answer on a filesystem level, is a resounding no. this is sad, but it may change. at least, for now, there are many clouds in the sky.

fence me in

in ,

one of the highly touted features of iOS 5 at its introduction was Reminders. the feature that made the WWDC crowd ooh and ah wasn't just an official Apple to-do list, or a fancy timed alarm system (neither of those would be particularly innovative), but the ability to set location-based reminders. buy milk next time you're at the grocery store? get there and *buzz*. or so we thought. then this came across the twitter: [blackbirdpie url="https://twitter.com/#!/waferbaby/status/124518378076520448"]

[blackbirdpie url="https://twitter.com/#!/dokas/status/124520733496979456"]

hold on. i have to go to the grocery store before i can remind myself to buy something there? what if i want to remind myself to call my family when i arrive at an airport halfway across the country that i've never been to before? well, there's one other option: you can set reminders up for addresses that are tied to one of your contacts. i guess that would be fine if it didn't impede your productivity, but even stranger, it tempts you to violate Apple's aesthetic.

the garden of Apple

anyone who's ever watched an Apple product keynote knows about the perfectly cultivated, mythical world that exists within their demo devices. every photo is in focus with smiles. basic word processing tasks become graphic design projects and suitable-for-framing posters. every item is tagged down to the last iota of metadata. everything is beautiful. in one sense, this is just to put on a good face and show the product in its best possible light. in another sense, you get the idea that Apple actually expects you to use the product this way.

stevejobswwdc2011liveblogkeynote0674

stevejobswwdc2011liveblogkeynote0674

on e57 of The Talk Show, John Gruber and Merlin Mann talked about the similarities between Apple and Disney, particularly in re: the insane attention to detail and upkeep that's required to keep Disney World the most magical place on earth. they mentioned the fact that the Magic Kingdom is never empty; it's staffed 24/7, even if it's only open to the public during the day. keeping everything just so is a literally constant effort, and as such is only possible when done in shifts by a team of workers. Apple may or may not be a 24/7 company (at least not on the design end; who knows when you take into account overseas production by OEM manufacturers). nevertheless, they've shown us that with their own product in their own hands, they can keep up the illusion of perfection, down to a perfectly manicured garden of contacts, faux and real, social and business.

however, Apple isn't in the theme park business. they are in the computing business, which inevitably means handing over their precious product to be maintained by us, the users. we have different tastes. we want to tinker, or hack, or just be plain lazy. we would suck at running Disney World, for the most part. nevertheless, by force of implementation or by mere suggestion, we are still pushed towards The Apple Way.

breaking the illusion

the paint begins to crumble and the facades of Main Street USA fall down to reveal the fluorescent-lit office buildings behind them when the inutility of iOS 5 location reminders comes to bear. the odds of gaining value by setting up a reminder to trigger at the location you're already in, or at the home or business of a friend or colleague, are very slim; chances are a plain or timed reminder would work just as well. what's the workaround? well, you can put that address in a new contact. but then Grocery Store is in your master contact list along with Your Mom (the lack of good nickname support and the notion that you would want to list your close family members as Firstname Lastname is a separate rant).

people do this kind of stuff all the time. people create contacts with first names only; people deliberately put where they know somebody from instead of their last name; people list their best friend as aaaSteve so he's re-alphabetized to the top. but you would never, ever see such a one-off, hacky way of squeezing utility out of the OS put on stage for demonstration. it would be quintessentially un-Apple.

building a better fence

geo-fencing is nothing new, but Apple is just now dipping their toe into it. and, if you still use iPhoto (i'm sorry), you know that Apple's idea of efficient place management is anything but. still, i'm a bit disappointed that i can't set up a reminder of the following type:

remind me to buy bagels at CTB on the second Tuesday of every month only if i'm in Ithaca, NY

i could set up an iOS 5 reminder at my apartment in Ithaca, which would probably trigger on the appropriate day, but that seems like a slightly odd way of doing it. or, if i target the actual bakery, i may never stumble into the arbitrarily sized (and presumably rather small) geofence that surrounds it. i might even have to get close enough to see the sign out on the sidewalk advertising the monthly special before my reminder would be triggered. in that case, a decidedly 20th century reminder would be more effective than my 21st century one.

fortunately, there is hope that we won't have to just build kludgy reminders and faux contacts, while we wait for Apple to catch wind of the fact that we're dissatisfied with the project. the technology that underlies the location-based reminders is a new OS service in iOS 5 that does low-power GPS tracking. and, perhaps surprisingly, it's not a proprietary API call. that means that third party developers are free to use it and put apps in the official App Store that use it. Foursquare already announced a new feature called Radar that uses it to send you notifications based on friend activity in your vicinity. this means that similar functionality, including arbitrary geofencing, ought to be able to be merged into a reminder app. Remember The Milk has already added location services to their Android app; their first priority for iOS 5 and the iPhone 4S was to get Siri working with RTM, but i can only imagine that they'll be adding location support next. and then i will gladly buy their app and subscription service over using a wonky Apple implementation. will i be upset if Apple steals their feature for the next version of Reminders? no, in fact i expect it; it should have been there in the first place. if they can build a better geo-fence than the competition, they will win back my fence-building business.

thanks, Steve

in ,

i just read Steve Jobs' resignation letter on my iPhone.  then i turned my phone over, looked at the Apple logo on the back, and just stared at it for a minute straight.  and for the first time, i couldn't believe that an object that i carry in my pocket every day even exists. i became an Apple fanboy shortly after my parents bought a Power Mac 6100 in 1993.  i lived the dark days of Apple as a user.  i saw just how bad, desperate, and utterly without guidance the Mac world was.  this was pre-iMac, pre-iPod, pre-iPhone…and the iPad?  forget science fiction.

then the second coming of Steve happened.  a modest turnaround at first, to be sure.  the fact that the iMac turned a modest success in 1997 didn't presage any of what Apple has become in terms of products.  on the business side, they're now the biggest company in the world by market cap.  insanely great has become just plain insane.

for the long-term Apple diehards, this is a scary time.  i haven't updated to Lion, and there are things about it that make me squirm a little bit (what John Siracusa called "geek panic" in his Lion review).  i don't want my iPhone to take over my computer.  but i have my iPhone.  my second iPhone, actually.  how is that even possible?

in 2001 and 2002 i had a Palm V.  it was kinda good for keeping track of my calendar and playing the odd game.  but i wanted more.  i wanted an uber-device.  i sketched PDA/camera/phone hybrids in a little notepad.  they were idle wishes.  pipe dreams.  now i have one in my pocket.

thanks, Steve.  stay well, and be insanely great for as long as you can.