Obstructionist vs. Intensifying Recording, Novices vs. Experts, and Google Glass

Point: Don Norman on Google Glass. Read also I Go To A Sixth Grade Play, which is spot on. In theory, you can record everything and live totally seamlessly and not just miss a large portion of your life. In practice, we'll keep futzing with cheap, poor imitations, destroying the experience itself to get a recording nobody will ever watch. We're obstructing the experience by recording; it is possible to intensify it, as by a master photographer or artist, but we are usually not doing that.

Really killer point:

Probably we've all seen a wedding reception, an event meant to be full of spontaneous expressions of joy, transformed by the photographer into a series of staged events. “Kiss the bride.” “Again, please.” “Cut a piece of the wedding cake.” “Each of you feed the other.” “All you spectators, move out of the way of the camera.” It is amazing how tolerant we have become of this manipulation of the experience: the act of recording taking precedence over the event.

Interesting side question: why do we want all these recordings? why do we cling so hard to keeping certain moments? Fairly certain this is a Deep Question. (or a question with a simple answer, but a difficult problem to solve.)

Counterpoint: Thad Starner explains it himself. Farhad Manjoo agrees. The computer can get fully out of your way, allowing you to experience and record in real time. (we've always been able to experience OR record; the AND is the real trick.)

But look at Thad's devices vs. Glass. He's got a Twiddler one-handed keyboard, he's been taking notes and pulling things up on the fly for 20 years, he is an expert at wearable computing. If Glass becomes a mainstream thing, we'll run out to the Google Store and buy it to show off to our friends tomorrow. He's an experienced photographer with a DSLR; most of us will be chumps with point-and-shoots. (or, chumps with DSLRs, pretty much the same thing.) Which means we'll have obstructionist artifacts, not intensifying ones. And they'll be on our faces!

"Not a math person" in HCI

"Math is the area where America's 'fallacy of inborn ability' is the most entrenched."

"I'm not a math person" is an unfortunate sentiment that lots of kids echo when they're grumbling through homework, explaining bad grades, or choosing majors. Other people (like the above) have pointed out why this is Not a Great Thing more eloquently than I.

HCI kind of lives between CS, Design, and Psych/Cognitive Science. It's so new, nobody's totally clear on what the whole scope of the field is, and indeed, maybe that's a dumb question. It's also so broad; anything where humans and computers interact. What's submitted to UIST will be totally different than something submitted to CSCW. As a result, everyone (at least at CMU) is an HCI student, but also (and sometimes primarily) a computer scientist, a designer, or a psychologist.

Point: this influences people to stay in their "major", rather than approach problems cross-disciplinarily.

Counterpoint: well, we humans need to categorize things somehow, in order to understand, for example, what certain professors or students work on.

Point: Fine, I guess. But be very clear when you're pigeonholing people, and do it as rarely as possible.

New work directions: smartphone tensions

This is a question I've been interested in for a long time, and feel like maybe I'm finally assembling the tools, people, and mental energy to tackle it. To begin to attempt to tackle it.

What is it about smartphones that stresses people out, and what can we do about it?

Okay, a lot of things stress people out. Your friend's using his phone while you're talking to him. Your boss is calling you at night. Your family expects you to text them when your plane lands, and you forget. That lady in the car next to you is texting while she's driving. You keep feeling an itch to check on your Facebook. You keep feeling a literal itch, because your Facebook is buzzing you until you check it. You don't know what it is, but you feel a little scatterbrained.

A lot of issues! Ways we could approach them:
- pick a problem that is well-defined (like texting-while-driving) and develop targeted solutions to that. (like SafeCell, which stops you from texting while driving).
- pick a measurable dimension to address a slightly less well-defined problem.
- just start from the top and tackle the whole thing.

I think the last is most interesting. And I guess it leads to a multi-step approach:
1. understand the problem. What are the tensions involved here? Why do people want to use their phones so much? What about this becomes problematic?
2. address the problem.

For part 1, I'm thinking interview people and review log data to get at what people are actually doing and why. For #2, it's more prototypes/probes than actually functional ideas. Build apps that get at the causes of these stresses, not apps that change their behavior.

Because the goal here is not to build another app that helps you slow down/de-stress/be more present. If we build a thing you've got to use, we've already lost. But it'd be great if we could uncover some of the underlying design guidelines that should be built into phones and apps. Tell developers something like: "infinite scrolls are technically cool, but will cause users the following stresses: ..." or "if you notify people more than once a day, they'll start to get antsy about it" or whatever. Instead of building an app to help you de-stress, make your phone not stress you in the first place.

UIST 2013 highlights

(names are presenter or first author; of course they're all "et al". well, almost all. google scholar the paper titles to get links.)

PneUI: Pneumatically Actuated Soft Composite Materials for Shape Changing Interfaces, Lining Yao
- exploring what you can do with air-filled interfaces.
- examples: a soft material with a bubble on the back curls up as the bubble is filled, or a tower expands and contracts as air is added/removed.
- cool thing: you could have a soft phone that just morphs into a wristband.

Controlling Widgets with One Power-up Button, Daniel Spelmezan
- prox sensor + pressure sensor makes one physical button you can do six gestures with.
- it's a lot easier to put one button than a bunch of buttons on many small devices.

Haptic Feedback Design for a Virtual Button, Sunjun Kim
- a soft button that feels like a clicky mechanical keyboard button. this is aesthetically pleasing.

Transmogrification: Causal Manipulation of Visualizations, John Brosz
- select a section of a graphic, morph it into another shape (e.g. square -> trapezoid, or even square -> circle)
- snapping to paths, so you can e.g. straighten out rivers on a map
- cool thing: you can make a chain of interactive infographics that all depend on the previous one.

Visualizing Web Browsing History with Barcode Chart, Ningxia Zhang (poster)
- looking at browsing to see how often you switch, maybe.

StickEar: Making Everyday Objects Respond to Sound, Kian Peen Yeo
- little sticky gadgets that each sense sounds and can be configured to do things

uTrack: 3D Input Using Two Magnetic Sensors, Ke-Yu Chen
- you wear two magnetometers on your ring finger and one magnet on your thumb, and then you can do free-space 3D gestures. Neat!

FingerPad: Private and Subtle Interaction Using Fingertips, Liwei Chan
- similarly, you wear a sensor grid on your fingernail and a magnet on your thumbnail, and now you can draw on your fingertip. The privacy of it is kind of cool.

BitWear: A Platform for Small, Connected, Interactive Devices, Kent Lyons (poster)
- little fingernail-sized buttons with Bluetooth and LEDs that you can configure on the internet.
- I would really like to play with these.

Imaginary Reality Gaming: Ball Games without a Ball, Patrick Baudisch
- you can play basketball without the ball. QR-ish codes on people's heads let an overhead camera know who is where, and a speaker says who has the ball.
- I mean, that's cool in itself.
- even cooler: imagine real world games with power-ups!

inFORM: Dynamic Physical Affordances and Constraints through Shape and Object Actuation, Sean Follmer, Daniel Leithinger
- tangible table, a lot of little square things that can move up and down.
- you can make really non-"computery" controls like the ball answering machine. Or it can move things around on the table. It looks like it has a personality.

Traxion: A Tactile Interaction Device with Virtual Force Sensation, Jun Rekimoto
- little metal thing with a moving magnet, feels like it's pulling left or right. The demo was a big hit. Pretty weird that it can fool your mind like that.

There were a few cool mixed-initiative things there too. Cobi: A Community-Informed Conference Scheduling Tool, AttribIt: Content Creation with Semantic Attributes, SeeSS: Seeing What I Broke - Visualizing Change Impact of Cascading Style Sheets (CSS), A Mixed-Initiative Tool for Designing Level Progressions in Games, A Colorful Approach to Text Processing by Example. I might be making up a theme out of nothing (I guess the part-computer part-human system thing is just all of HCI) but there's something that feels pleasingly interactive out of these.

Something about addiction and flow

So flow is good, right?

Well, sometimes, yes, unless you're addicted to slot machines. Or online games. Or Facebook, or e-commerce sites, or really anything else unproductive.

So if you were, say, starting out on some research that could probably be pretty well defined by "make your computing experience more flowy"... how can you motivate that better so it's actually about making your life better, and not just faster? Google gets away with worshipping speed and fluidity because faster page loads means more internet use, which means more money for them, but the rest of us are not optimizing for money.

To Save Everything, Click Here

by Evgeny Morozov. This was quite a read. Not often I painfully jot down 11 pages of notes in my ebook reader. I guess it's because he has strong opinions about most everything I'm interested in.

Quick overview: railing against "internet-centrism" and "solutionism." Internet-centrism is the modern trend to ascribe magic powers or moral judgments to "The Internet." (Piracy should be tolerated (instead of DRM) because The Internet! We should change government to be more like Wikipedia because The Internet!) Solutionism, a broader topic, is the tendency to see any phenomenon as a problem to be solved.

Caveats: he's kind of a jerk. I'll call him out on it explicitly later. It's really frustrating; I want to like this book more, but often he just resorts to bullying. And his caustic writing style is probably calculated just so people like me will get all fired up about his book! In that case, sir, good job. (I hope I never meet you.)

Anyway, solutionism. A good example of something that's maybe not a problem to be solved is cooking. He makes fun of all these kitchen technologies that help you cook "more perfectly" - which is to say, more accurately or more efficiently, silently assuming that accuracy or efficiency is what we want. "Here is modernity in a nutshell: we are left with possibly better food but without the joy of cooking."

Internet-centrism: people complain about Apple creating a closed ecosystem. But just why is open better? In Apple's case, for selling you some sweet apps, maybe closed is a better model. (Morozov's thoughts, not mine.) Arguments for openness often (not always) resort to "because openness! The Internet!"

Other things that Morozov says that I agree with him on, or am open to considering:

- Silicon Valley libertarianism is bad: hell, we wouldn't even have The Internet without public financing.

- "Open government" isn't a goal - or at least, that openness isn't a goal in itself. For a few reasons: 1. it's a huge pain to have to document every damn thing you do; 2. it opens the doors to the public to nitpick every last little spending decision (some of which are unintuitive but turn out well). Efficiency is great, but not necessarily our #1 goal in our government.

- Technological solutions often have unintended consequences. If you publish crime stats by neighborhood, sure, that helps home buyers/renters to find safe neighborhoods, but it also hurts sellers and therefore might make them less likely to report crimes.

- You can't rely on Yelp-style crowds for everything. (hell, you can't even rely on Yelp-style crowds for Yelp.) I don't want crowds telling me where to eat, much less what to vote.

- If you publish some metrics (like senator attendance records) then people will optimize for them. (this can be problematic. maybe one senator has more important things to do one day.)

- We shouldn't take Google search results as gospel; they can be manipulated. (Interesting question then: given that we do take them as gospel, what should we do about it?)

- "Internet-centrism is at its most destructive when it recasts genuine concerns about the mismatch between what new digital tools and solutions have to offer and the problems they are trying to solve as yet more instances of Luddite and conservative resistance."

- Complicated computer algorithms, like any other decision making tool, reflect the biases of their creators. But complicated algorithms offer a (sometimes real, sometimes fake) appearance of objectivity. (for example, for police work.) (sounds like a call for intelligibility.)

- Oh man, great stuff about SCP, the law enforcement philosophy that says you should make it impossible to commit crimes, rather than just punishing people who do. This is really interesting. For example, if I decide I'm not going to eat cookies, I want SCP-style prevention there! I want to make it impossible for me to eat cookies! He agrees: as long as you make the decision yourself, there's no problem "shifting registers" (term from Roger Brownsword) from the "moral" register (x is good or bad) to the "practicable" register (x is easy or hard) or the "prudential" register (x helps me or hurts me).  So he's against "nudges". I can dig it. When we shift things out of the moral register, though, we might never even think about them again.

- Excessive quantification in the research world is a mess. People are gaming metrics, counting publications and citations too much, etc.

- Food is a good example of solutionism gone wrong. We decide fat is bad, so everyone counts grams and does all sorts of nasty tricks to call their products "low fat", only to later discover that fat's not so bad after all.

- Furthermore, Quantified-Self solutions can backfire by putting the onus back on the individual, rather tahn the broken system. ("why didn't you just count your calories?")

- Memory != preservation. And we shouldn't assume that we should preserve everything.

- We talk about "information nutrition" (e.g. The Economist is healthy, tabloids are junk food), but we really have no idea what we're talking about.

- You can't really design for serendipity.

- Gamification via points and badges is dumb. (okay, duh.)

- Check out Albert Hirschman's futility-perversity-jeopardy trio as a set of common reactions to new things.

- We should try to change people's behavior mostly by reflection, not by paternalistically "nudging" them into making the right decisions. I don't know, though; his example, using some complicated parking meter thing, seems like those 5 cent nickels at Whole Foods when you bring your own bags, and then you have to decide where to donate your 5 cent credit. I don't want to think about those 5 cents. Just let me get on with my day.

- Sure, everyone's always manipulating us. But I want to know when it's happening.

Anyway, the overall feeling I got from reading this is that maybe my earlier research thoughts are misguided. I basically heard the Weiser ubiquitous computing story ("your computers will fade into the background" etc) and thought "yes, let's do that." Maybe perfect efficiency isn't always the goal!

Some things that Morozov says that are stupid: (if he's going to bully people, I can bully him back)

- somehow make "open government" data (say, campaign finance) only appear on the original source (like the FEC website) so people can't re-publish it and selectively alter or highlight it.

- the "Pirate party" in Germany is losing support, therefore their ideas are failing and should be mocked.

- LiquidFeedback, this tool that sounds like Google Moderator, is a "solution to a problem that doesn't exist" - we don't need more feedback from, say, politicians. 

- partisanship isn't necessarily a problem

- Amazon might start automatically generating books! That are creepily optimized to be exactly what you like! Never mind that this is probably AI-complete!

- We should start having all algorithms be audited by qualified third parties. (oh my god. what constitutes an algorithm? geez.)

- "Once everyone is wearing Google's magic glasses, the costs of subjecting friends to a mini lie detector... are trivial."

- Argh, he totally doesn't understand some of the systems he's writing about and mocking. (e.g. Matthew Kay's Lullaby)

- Quantified Self people are weird and gross. (seriously, this whole chapter is just straight up bullying. it's really offensive.)

- Quantified Self people are all into some weird woo-woo shit about revealing the deeper truth of who we are, with numbers! What a bunch of misguided weirdos!

- Self tracking for health purposes makes a mess for the insurance industry, so we shouldn't track things about ourselves.

- If you like quantifying something, then you must be a Super Quantifier who wants to quantify everything!

- "Even though Bell doesn't quite put it this way..." (... puts words into his mouth.)

- Gordon Bell is a weird guy. I can just dismiss anything he does by mocking him.

Some comical excerpts from my notes as I was reading this:

"no, you numbskull."

"sigh"

"this section sounds fearmongery"

"the bullying in this section makes me wonder about the rest of this book."

"not what he said. you clown."

(on chapter title "Gamify or die") "I hate this chapter already"

(when he starts talking about extrinsic vs. intrinsic motivation) "sigh I knew this was coming"

"ad hominem, you're a jerk, etc"

My "wearable computer" set up, June 2013

I don't (yet) have a Google Glass, but I wear computers, and it's great. Currently I have:
- Android phone (Nexus 4, if you must know)
- Pebble wristwatch
- LG Tone headphones
- Fitbit Ultra (no longer made; replaced by the Fitbit One)

The Fitbit counts my steps, and that's all. The neat thing about that is that I can compare days.
I know what it's like when my friend tells me she had an 8000-step day.
I know dancing for a couple hours is usually a good 10k steps at least.
I know that my 30k-step day in Dublin was seriously a lot of walking.
I haven't figured out why it's useful to know these things. Just cool.

The Android/Pebble combo gives me texts on my wrist. This is cool. I've received, understood, and dismissed a message on a bike. Yes, you can do this safely. I've received, understood, and dismissed a message in mid sentence. Yes, you can do this politely. (trust me; your conversational partner will mind much less than if you get your phone out and fiddle with it.)

Is it good that you can do these things? Good question.

But the Android/Pebble/Tone combo! This is the coolest part. Connect everything and start my music app, and then I can start/stop/forward/back the music from either my Pebble or the main Tone hardware. AND I can see the title of the currently playing song on my Pebble. I can almost-fully control my music on my bike. (yes, I can do this safely.)

The fact that it used to take ~25 seconds to start listening to music (unwind headphones, plug in, etc), and it now takes 3, means I listen to a lot more music. The fact that I can see the song playing means I remember it a lot better.

Is it good that I am listening to more music? Good question. But as a Real Actual DJ, who's constantly trying to take in new music, I appreciate it.

What do I take from this? Not a lot, because it's just me. Nevertheless:
- reducing the time it takes to do something really does make me do it more.
- I have no idea if there are negative effects from, say, the fact that I rarely walk more than 5 minutes in silence anymore. I have no idea how to measure this.
- the form factor has to be good and sort of invisible, but it doesn't have to be all that invisible, if there's clear benefit. (the Tone is a new one; it's "around-the-neck" style, and noticeable but usually not problematic.)
- we have enough ways to control music.

Voice of the Coffeepot

Well, this was kind of fun:

A class project. Used a force sensor to tell when the carafe was present/empty/full/being pressed, and either thanked you for refilling the coffee or wished you to enjoy your coffee via a text-to-speech chip and speakers. Enjoy your weekend!

Visualizations, video/audio, and ML for time series data: which platform?

I want interactive visualizations of a bunch of time series, as well as some audio and video, scrub through them and keep them all synced, and then generate features from them and feed those into a machine learning model. No need to be a web app.

What environment do we do this in? Cross-platform choices seem to be javascript/browser, python/installed, and java/installed.

Visualization:
Javascript: d3 (gallery), cubism, Google Charts, some others like rickshaw, even processing.js. This is maybe the best reason to pick JS.
Python: matplotlib - mostly static visualizations (gallery) or Bokeh? (see this post) Bokeh outputs to an html5 canvas in the future (or a Chaco plot currently).
Java: ...?

Playing audio/video:
Javascript in a browser: HTML5 video and audio?
Python/Java: beats me. Codec hell?

Machine learning:
Javascript: ...?
Python: scikit-learn (and others)
Java: Weka (I hear the API's a pain, though.)

The path forward seems to be to start building an HTML/JS app, even if it's only client side, and figure something out for the machine learning. Perhaps compile scikit-learn to JS with pyjs? Perhaps (this sounds kind of painful) just send all the features to a server and use weka or scikit-learn there to do the real ML and send back results? But I'd welcome any input.

It's about how you use it

"I'm still here: back online after a year without the internet"- in which he leaves the internet, and all's well, until it's not anymore; all his bad habits catch up with him offline too. Also, you can't just quit the internet alone. It's part of all our life together. 

Don't be a gargoyle with Google Glass. Don't be with people but not actually with them. Don't tolerate other people being gargoyles.

Is Glass an anti-distraction tool? Could be. So long as you can ruthlessly and efficiently curate what goes into it (or if the algorithms it uses are good at doing that for you). Given people's abilities with their email inboxes, I'm not sure, but I think this writer's on to something: if people have a low cost way to get into your visual field, that'll be a problem.

So what are we left with? "The Internet, or Google Glass, or whatever technology you want, isn't good or bad in itself; it depends on how people use it. (which depends partially on design decisions of the people making it.)" Xkcd is right on.

Three kinds of multitasking

I've got to start posting one thought at a time. Shorter posts more often.

Let's get our words straight: here's multitasking vs. switch-tasking; multitasking is doing a lot of related things, switch-tasking is doing a lot of unrelated things.
Maybe this points to the polychronicity puzzle: how is it that some people can prefer to do many different things, even though multitasking doesn't work? On the other hand, examples of polychronicity sound like switch-tasking, so that actually doesn't help answer that question.
A third term: continuous partial attention, or paying attention to many things at a surface level.

I guess it's like this?

Thinking about time and speed

Half formed ideas and links here.

Been reading a lot of blogs and listening to talks, going to a workshop, thinking about how we view time and how fast it goes. In Motion has a bunch of thoughts about time, in relation to travel; while a lot of the book is frustrating (what is "Deep Travel"? is it just "traveling while paying attention"? why does it need a name?), it makes me think about time as experienced vs. time on the clock.

Monochronic vs. polychronic time - first pointed out to me in two papers by Gloria Mark and Laura Dabbish. Monochronic is what we're used to: one thing at a time, time matters, be on time. Polychronic time sense is the approximate kind of time; do a couple things at once, value relationships more than the time. (but multitasking doesn't work! what's going on here?)

Contemplative Computing points me to a few things: an article about speed mentioning Google Glass cutting down the picture taking time from 12 seconds to 1 second, an interview with Linda Stone on time management vs. attention management, a group interview where Neema Moraveji talks of (among other things) "speed bumps" put into typical tasks. We're putting speed bumps into one task and taking them out of others (and indeed, as texts-on-the-Pebble has shown, removing speed bumps is surprisingly awesome). The answer is probably not "speed is good" or "speed bumps are good", but something more subtle. What is it?

Another thing I want to jot down here: calendars are to time as maps are to space. Ponder that.

CHI 2013

... was, of course, great. I student-volunteered, which meant I didn't get to see a ton of talks; nevertheless, here are a few things I particularly liked:

Stories of the Smartphone in Everyday Discourse: Conflict, Tension & Instability, by Ellie Harmon and Melissa Mazmanian. They looked at the stories people tell themselves about their mobile phone use; it oscillates between the "integration" story ("get a smart phone, be a super connected techno future hero") and the "dis-integration" story ("unplug, de-stress, get back to the real world"). This causes tension and makes people uncomfortable. Interestingly, these are kind of the only stories being told, and they're overly simplistic.

Indoor weather stations: investigating a ludic approach to environmental HCI through batch prototyping, by Bill Gaver et al. They put little devices in people's homes that would playfully reflect indoor environment conditions (like slight wind, etc); people didn't find them practical or particularly fun, but still felt some attraction to them; "it's like there's a ghost in the house."

Slow Design for Meaningful Interactions, by Barbara Grosse-Hering et al. They described principles they used to design a juicer. Most interesting: it's okay to make some things slower. It can be good, in fact, as long as they're the key parts of the process. People don't mind spending a little more time juicing; they mind spending a long time cleaning the thing afterwards.

Some cool gadgets you can wear: WatchIt (watch band interaction) and NailDisplay (goes on your fingernail; I wasn't sold at first, but then it grew on me throughout the presentation).

Some cool info-vis-related stuff: The Challenges of Specifying Intervals and Absences in Temporal Queries. Interactive Horizon Graphs. Alternate Glyphs (color and spirals). And, fNIRS for evaluation!

Finally, and most entertainingly, don't use seven segment displays. See you all next year in Toronto!

Canabalt 8x8

You might think I spend all my time researching Really Hard Problems in Human-Computer Interaction. This is not entirely true. Sometimes I take classes, and sometimes classes have projects, and sometimes I get to make fun things like this:

(real Canabalt, which is a lot more fun and I cannot take any credit for making)
(more details on github)

Hypothesis Generation Systems

Imagine you're tracking something about your health. Say it's your weight. You have weight measurements from every day and you want to generate some hypotheses to test, like "my weight this month is significantly higher than last month." Or say you're tracking how many steps you take each day, and want to know some things about how that correlates with when you woke up, or any number of other data streams you've got.

I'm picturing some kind of visual analysis tool. Something that would help with hypothesis generation and feature selection. What's in this space? I started with the 150 most popular tools on the Quantified Self guide, as well as all 204 tools on Ian Li's personalinformatics.org. Some things I found:

Systems that pull in data from a lot of places and generate visualizations
Notch pulls in data streams from (currently) Fitbit, Runkeeper, and Bodymedia. Then it generates some pretty basic (although definitely pretty) visualizations.
Cosm (formerly Pachube) lets you look at a bunch of graphs together (one example here). Doesn't do any statistics, though, AFAICT. Focuses on Arduino/Internet-of-Things applications.
Trackify seems nonexistent.
Fluxtream seems promising, but I don't have access yet.
Also The Locker Project (Memo to myself: they seem to be very connected with Singly, which looks like a way to easily integrate apps with lots of social and/or quantified-self applications), TheCarrot, DataDepot
These all would be much cooler if they did stats (although just visualization is useful too!)

Tools that let you track things yourself by hand (and then they generate graphs)
ChartMyself (meant for QS-type use), TallyZoo (for anything), Mycrocosm, Daytum (anything), MercuryApp (mood-ish things), Dayta, Tonic, DidThis (things you've done), rTracker, Graphomatic, DailyDiary, LifeMee (maybe? signup is broken), Limits, DailyTracker, Grafitter, your.flowingdata, lifemetric (for many people)
These all seem like not what I'm looking for. The data entry is manual, and the output is still mostly visualization.

There's Quantified Mind, which apparently shows you all the significant results in your mental-test data automatically. Only for mental-test data, but still, this is the kind of thing I'm looking at.

There's media annotation and editing tools as jumping-off points, if you wanted to build this. Elan is one, but it's based on the quirky Java Media Framework; Pitivi is another, but for Linux only.

Then there's the whole field of information visualization, big-data type stuff (recent startup Trifacta comes to mind), which I'm just starting to dig in to.

More to come! There's a conference called IEEE VIS (formerly VAST) that might have some interesting stuff, I'll trawl through that. Also, any ideas you have would be quite welcome.

New Note. New Note. *sigh* New Note.

I'm interested in what you can do with audio if you're wearing a headset/microphone. As a baseline, I want to see what you can do with off the shelf components already with minimal frustration. The test task is making a note without taking the phone out of my pocket.

(tl;dr: this does not work at all. I am surprised at how much this doesn't work. Geez, I figured voice commands were solved.)

Headsets:
I've tried the Plantronics M50 and the LG Tone. The M50 is a standard bluetooth headset, the Tone is worn around the neck and has detachable magnetized earbuds that you can put in your ears. First impressions:
- I feel like a businessman and a tool wearing the M50. With the Tone I feel like a regular person. This is important.
- Music streaming to the ears works right out of the box on both. This is surprisingly great, especially when biking.
- Sound quality in my ears seems fine on both, although I've only used them a couple days each. The M50 is a little bit quiet even on max volume for listening to music, but okay for calls. Can't say much about voice recording quality.

Apps:
- the Google app that includes Voice Actions and Google Now (on Jelly Bean, I think) is so cool and so flawed and buggy. The cool: saying "note ____" records your voice and sends it to yourself in an email: both the audio file and an attempted transcription. No confirmations or anything. Exactly correct. However, when I associate the "bluetooth button" action with the Google Search app and press the button, nothing happens. And when I open the app on my own, sometimes it jumps right into recording, and sometimes it, says "initializing" forever and freezes. This is on my 2-year-old Nexus S, and the Galaxy S3 I'm working with doesn't have Jelly Bean. I'm guessing when they get the bugs worked out, this will be the reasonable way to talk to my phone.

- Voice Control (Full) works pretty well. Pressing the button on my bluetooth opens it and starts listening (IF you disable the Google app and then re-associate the "bluetooth button" action with Voice Control). I can say "Make a note" and it puts it in my Evernote account. The downside is that it's a 5-step process: "Make a note" What should the content be? "blah blah" Do you want to make a note with content 'blah blah'? "Yes" What should the title be? "title title" Do you want the title to be 'title title'? "Yes." Five steps is four steps too many (especially because all these steps can, and do, fail).

- Vlingo boasted at least somewhat-competent voice recognition but you can only do about 7 things, none of which I even want to do. (call people, text people, update your facebook status...)

- Utter looks like it's headed in the right direction, but right now doesn't accept the button on the bluetooth to trigger it.

- Samsung's S Voice: not so good. Pressing the button on my Bluetooth device starts it... sometimes... and sometimes it just opens the app so I can start it by pressing a button on the screen (which defeats the whole purpose of the bluetooth button). Also, when you're making a note, it asks you to confirm by saying "Save note"... and if it doesn't hear "save", it just hears "note", which throws away your old note and starts a new one. What!

- Skyvi ("Siri for Android") had poor voice recognition. Also, it doesn't support just taking notes. Am I the only one who never wants to call people if it's not very reliable that the service will call the right person?

- Iris ("Siri backwards") just didn't work on my Nexus S, and on the Galaxy S3, it looks like it's a press-an-on-screen-button app. (with lots of annoying advertising for some other app too.)

Finally, a couple of thoughts Absolutely Correct Ideas about how voice commands should work, based on a day of futzing with them:
- start on bluetooth button press. If I take my phone out of my pocket, you've lost me. Ideally ideally, we'd be starting on a wake-up word, but I assume the battery life isn't there quite there yet.
- don't rely on correct transcription, when possible. (taking notes shouldn't rely on correct transcription.)
- corollary: don't ask me to confirm stuff, unless it relies on correct transcription (like calling a person). I should say things once, maybe twice.

Memorizing Names

Say I was talking to you and I told you one fact (like my name) and you wanted to memorize it, but we kept conversing. How would you do it?

You should rehearse it at T+8 seconds, 14 seconds, 32 seconds, 86 seconds, and increasing intervals in a 5+3^n pattern. Here's why:

Spaced Repetition has been around a long time. The idea is that, if you're going to practice a fact N times to remember it, you should practice it over time, not all at once. (this is called the Spacing Effect.) You should be asked to reproduce the item, not simply shown it again. ("Testing Effect") Furthermore, these rehearsals should be in increasing intervals. ("Expanded Retrieval")

About the spacing effect: this has been shown repeatedly (see pretty much any link in this post where spaced practice beats massed practice)
About the testing effect: this has been shown repeatedly too, e.g. by Carpenter and DeLosh (2005).

About expanded retrieval: this is a little less clear. Pimsleur (1967) suggested exponentially increasing intervals. Landauer and Bjork (1978) found that increasing intervals (e.g. rehearse in 1-5-9 seconds) is better than equally-spaced intervals (like 5-5-5) if you're testing yourself, but neither Carpenter and DeLosh (linked above) nor Balota et al (2007) found much support for the "increasing intervals is best" argument. Indeed, Karpicke and Roediger (2007) found that increasing intervals helped short-term recall, but equally-spaced intervals helped long-term recall. But they found that this effect may be due to the equally-spaced intervals' lack of an immediate recall (the "1" in a 1-5-9 schedule). They showed that just delaying the first test by 5 seconds makes it harder, which helps long-term recall. So it seems like you should be able to get the best of all worlds by adopting an increasing-intervals schedule, but also delaying the first review.

Another consideration is that this is the real world, not a 3-repetition study in the lab, and increasing intervals scales better. If you start practicing every 5 seconds, by the time you're at repetition 10 you'll be fed up, whereas if you go with 3-9-27-81 etc the intervals will quickly become so infrequent that you're not bothered.

But what should the first interval be? Peterson and Peterson (1959) show that recall percentage at 3 seconds is better than 6 seconds, 6 better than 9, etc., and we want them to remember it at the first interval, so might as well make the first interval 3 seconds. But, as Karpicke and Roediger (linked above) mentioned, we don't want it to be too easy to remember it at the first interval. Well, the same Peterson et al (1963) found 8 seconds to be the best interval.

So, how about a 3-9-27-81 interval plus 5 second delay, so 8-14-32-86-etc? Whew! Well, whatever; increasing + delayed-first-item sounds like at least a pretty good way to go.

Hmm... but if you're not in a lab, talking to an experimenter, how will you remember to test yourself at all these intervals? Hey, what if we could do that with a system using instant, unconscious, subtle microinteractions...
Stay tuned!

Microinteractions: the book

Since I came across Daniel Ashbrook's thesis, I've been thinking about "microinteractions": "interactions with a device that take less than four seconds to initiate and complete." I'm interested in expanding the space of possible microinteractions that people use.

Now this fellow Dan Saffer is writing an O'Reilly book with the title Microinteractions. You can currently read a draft of the first chapter. Sounds like he's using a more general definition of the term, to include lots of non-mobile interactions: the "see translation" button on Facebook, the password entry form on Twitter, calendar apps including the duration and end time when you're scheduling a thing. I like it. It leaves me wondering: is this just "everything" now? Is "microinteractions" a synonym for "details"; something that of course we should focus on but nobody's going to have some big revolutionary ideas about? Or is this part of a big shift in thinking, now that we've got enough computing resources to actually make meaningful and positive microinteractions?

Incidentally, your potential microinteraction of the day: squeeze your phone as you pull it out of your pocket depending on how you want to use it.

UIST 2012 highlights

... in my humble opinion, based on my particular interests:

Tactile/hands/fingers:

Facet: a multi-segment wrist worn system, Kent Lyons (Nokia)

Watches are old news. How about having 6 watch screens in a bracelet shape around your wrist? It's clunky now, but who knows. They can detect pose and interact smoothly as needed.

iRing: Intelligent Ring Using Infrared Reflection, Masa Ogata (Keio U)

A ring for input with 4 sensors. Clever: recognizes IR reflection and skin deformation to tell whether you're clenching, bending, pushing, flipping, or rotating the ring. Detects position/rotation by melanin content (which varies around your finger. Wired currently (and a ring is so small it makes me wonder if the wire could be removed).

Magic Finger: Always-Available Input Through Finger Instrumentation, Xing-Dong Yang, Autodesk

Camera/LED on your finger for always-available input. The camera is 1x1mm, 148x148 pixels. ("NanEye") Read "fake textures"; ascii characters printed into patterns. Downsides: wired and ~1s latency on touches. Still, cool!

Digits: Freehand 3D Interactions Anywhere Using a Wrist-Worn Gloveless Sensor, David Kim, MSR UK

IR laser line and camera mounted on your wrist to detect your finger positions. Allows finger gesture prediction, 3d manipulation, etc. It's a bulky box now, but you could imagine it shrinking. Between 2-9 degrees of error, which is good enough for a lot of tasks.

Dynamic Tactile Guidance for Visual Search Tasks, Ville Lehtinen, Helsinki IIT

Wear gloves so when you're looking at a wall of stuff you can find the exact thing. Sounds like it could be useful. (the trick is finding a task where the computer knows where the right thing is, but you still have to find it with your hands.)

Pressages: Augmenting Phone Calls with Non-Verbal Messages, Eve Hoggan, U Helsinki

An addition to phone calls. You can squeeze the phone, and then it vibrates on the receiver's ear. Four intensity levels, from light to sharp. This might sound a little silly, but:

- they tried it with 3 couples for a month, and they all sent at least one Pressage every single call.

- they wanted to have it available other times too, and as another channel of communication (e.g. light buzz = "I'm coming home")

Assuming they didn't just get 6 quirky people in their study, people will use this. It's super intuitive and quick, and adds a layer of richness to phone conversations. (the channel during non-phone conversation is mostly a nice bonus, and is kind of tricky for a lot of reasons.)

Other Things:

Steerable augmented reality with the beamatron, Andy Wilson, MSR

Pan-tilt projector and kinect so things can project and interact all around the office. Quite a feat of engineering.

Designing for Low-Latency Direct-Touch Input, Albert Ng, MS

Our current displays have around 100ms latency on touch. You can see this yourself: draw a quick line with your finger; it lags an inch behind you. What if instead we had 1ms latency? I tried the demo, and it is much slicker. Feels like you're moving real objects. Remember when Google started targeting latency and all of a sudden Gmail became a viable non-painful application? Latency matters on tablets too.

Histomages: Fully Synchronized Views for Image Editing, Fanny Chevalier, Toronto

You know image histograms? What if you could select pixels by brightness or by blueness instead of by location, and edit them all based on that? Looks fun.

ConstraintJS: Programming Interactive Behaviors for the Web by Integrating Constraints and States, Stephen Oney, CMU

Ever made an iOS app with Interface Builder? You specify constraints (like "this text box aligns with the center of this image") and they are automatically maintained through resizes etc. ConstraintJS looks like a way you could do this on the web, and for more than UI layouts. You can make asynchronous calls and display all their states without the pain! http://cjs.from.so

DejaVu: Integrated Support for Developing Interactive Camera-Based Programs, Jun Keto, MSR Asia

An IDE for developing camera- (i.e. Kinect-) based applications. If I made a camera app, I think I would want this.

Low-cost Audience Polling Using Computer Vision, Andrew Cross, MSR India

Instead of expensive "clickers" which you might have used in university classes, just print everyone a QR-like card and have them hold it up. Cheap webcam takes a picture of the whole class.

Clui: A Platform for Handles to Rich Objects, Hubert Pham, MIT

What if the default unit on the web were a JSON object instead of a hyperlink? Sounds like web standards stuff, where it's awesome as much as everyone uses it, and good luck, but would be really useful in a ton of ways.

CrowdScape: interactively visualizing user behavior and output, Jeff Rzeszotarski, CMU

MTurk is great, but people cheat. Some tasks you can design around this, but some you can't. Crowdscape lets you visualize what people are doing as they do your task, easily weed out cheaters, use that as labeled input to bootstrap a machine learning system, and more importantly understand what work patterns lead to a good response. (Maybe tooting my school's horn a bit, but: won a best paper award!)

Some folks made a braille tutor out of the pressure-enabled touchpad that we got for the Student Innovation Contest. Awesome. They won 2nd place; I'd have given them first. Our ambient stress sensor was neat but did not win. Nor did it deserve to; there was a lot of great stuff.

Cool posters:

MMG armband, Shumpei Yamakura (like EMG, but resilient to sweat; not sure how they'd compare)

MISO, David Fleer, Bielefeld; point and snap at your electronics

Tongue-finding with Kinect (i.e. for rehabilitation), Masato Miyauchi

Breathwear (band to detect when you're breathing), Kanit Wongsuphasawat

Cool coffeeshops in Cambridge: Crema in Harvard Square for cappuccino and Voltage by Kendall for a fine Guatemala Buena Esperanza roasted by Barismo.

Yes! Another good conference. People asked me multiple times "So how's UIST going?" Look, of course it's fun and full of cool people doing exciting stuff!

Thinking about unconscious/micro interactions

Trying to define a research plan or story or something that I can both work on and apply for fellowships about. Right now a lot of work that I'm interested in feels related to me, but it's hard to explain to other people, which means it's not well-defined enough. In this post, I'm working on that.

All our interactions with computers/smartphones now are both intentional and slow. By "intentional" I mean that you have to think about getting out your computer (or phone), and by "slow" I mean at least on the order of seconds, if not minutes. Right now, get your phone out and check the weather forecast, and count seconds while you do it. (Just tried it; 23 seconds.) I want to break both of those constraints.

Why?

When you remove "intentional", you move from the slow brain to the fast brain. You get North Paw, Lumoback; systems that train you on a physical level. Ambient systems which help you change behavior: DriveGain, Breakaway, UbiFit (some slides). The Reminder Bracelet: ambient notifications. I guess it feels like, and I'm not sure how best to put this, when you do things unconsciously/unintentionally, you can learn procedural things or adapt physical movement without increased cognitive load. "Human attention is the scarce resource", and these systems give you something for free.

When you remove "slow", a lot more things become possible.

Thad Starner mentions the "two-second rule" in wearable interaction (IEEE article): people optimize their systems so they can start taking notes within two seconds of realizing the need. Daniel Ashbrook, in his thesis, defined microinteractions as interactions under four seconds, start to finish. At Google, speed was a big emphasis, and they're right: if something takes longer, people will use it less. (wish I had a good citation for this.)

Interactions are also overt; everyone can tell when you're computing. Breaking that constraint lets you interact with your computer without people knowing, which seems useful. Enrico Costanza has worked on "intimate interfaces" (EMG armband, eye-q glasses). ("intimate interfaces" is overloaded to also mean "interfaces that allow intimacy", e.g. among remote couples or family; not talking about that here.) Is this good? Detractors might argue that if there are social cues against something, they're there for a reason. Nobody wants you to be computing when you're trying to talk with them. However, two things: first, it's a tool just like anything else and should be used wisely; second, people already do these things with their phones. They get buzzed, answer texts, silence their phones, etc. But I'm not sure that I want to get rid of "overt", or at least not necessarily.

How?

Watches:

Mounting things on the wrist can cut down the action time up to 78%. You can use round touchscreen watches. Conveniently, the Pebble watch is now somewhere in production stages, and the Metawatch is... shipping? InPulse has been around for a couple years, but is a little clunky and doesn't have the battery life.

PinchWatch might not be what you think of as a watch (besides the display); a lot of the interaction is done by pressing fingers together. 

Nenya is a ring/watch system, getting really minimal. I like it. Reminds me of Abracadabra, due to the magnetic sensing, but now the ring just a regular-looking ring you would wear anytime.

Pockets:

You can touch your phone in your pocket, even do a Palm-Graffiti-style input (PocketTouch). More simply, for some tasks you could just hit your phone (Whack Gestures). Sami Ronkainen et al investigated this first, though they hit more false positives. (they also found taps to be more natural/accepted than gestures.)

Speech:

This seems obvious, right? But it's not. First, talking while walking around is weird. (ever play Bluetooth Or Crazy?) Second, it's not easy to get audio input into your phone in your pocket unless you're wearing a Bluetooth or something.

Sunil Vemuri et al's "Memory Prosthesis" was one approach, focusing on recording nearly-continuously and then searching; the search is less interesting to me, but the continuous recording is useful. Ben Wong experimented with dual-purpose speech: the user wouldn't give direct commands to a system; rather, the system would harvest information from things the user was saying. The Personal Audio Loop recorded your last 15 minutes and let you go back to search within it.