New work directions: smartphone tensions

This is a question I've been interested in for a long time, and feel like maybe I'm finally assembling the tools, people, and mental energy to tackle it. To begin to attempt to tackle it.

What is it about smartphones that stresses people out, and what can we do about it?

Okay, a lot of things stress people out. Your friend's using his phone while you're talking to him. Your boss is calling you at night. Your family expects you to text them when your plane lands, and you forget. That lady in the car next to you is texting while she's driving. You keep feeling an itch to check on your Facebook. You keep feeling a literal itch, because your Facebook is buzzing you until you check it. You don't know what it is, but you feel a little scatterbrained.

A lot of issues! Ways we could approach them:
- pick a problem that is well-defined (like texting-while-driving) and develop targeted solutions to that. (like SafeCell, which stops you from texting while driving).
- pick a measurable dimension to address a slightly less well-defined problem.
- just start from the top and tackle the whole thing.

I think the last is most interesting. And I guess it leads to a multi-step approach:
1. understand the problem. What are the tensions involved here? Why do people want to use their phones so much? What about this becomes problematic?
2. address the problem.

For part 1, I'm thinking interview people and review log data to get at what people are actually doing and why. For #2, it's more prototypes/probes than actually functional ideas. Build apps that get at the causes of these stresses, not apps that change their behavior.

Because the goal here is not to build another app that helps you slow down/de-stress/be more present. If we build a thing you've got to use, we've already lost. But it'd be great if we could uncover some of the underlying design guidelines that should be built into phones and apps. Tell developers something like: "infinite scrolls are technically cool, but will cause users the following stresses: ..." or "if you notify people more than once a day, they'll start to get antsy about it" or whatever. Instead of building an app to help you de-stress, make your phone not stress you in the first place.

UIST 2013 highlights

(names are presenter or first author; of course they're all "et al". well, almost all. google scholar the paper titles to get links.)

PneUI: Pneumatically Actuated Soft Composite Materials for Shape Changing Interfaces, Lining Yao
- exploring what you can do with air-filled interfaces.
- examples: a soft material with a bubble on the back curls up as the bubble is filled, or a tower expands and contracts as air is added/removed.
- cool thing: you could have a soft phone that just morphs into a wristband.

Controlling Widgets with One Power-up Button, Daniel Spelmezan
- prox sensor + pressure sensor makes one physical button you can do six gestures with.
- it's a lot easier to put one button than a bunch of buttons on many small devices.

Haptic Feedback Design for a Virtual Button, Sunjun Kim
- a soft button that feels like a clicky mechanical keyboard button. this is aesthetically pleasing.

Transmogrification: Causal Manipulation of Visualizations, John Brosz
- select a section of a graphic, morph it into another shape (e.g. square -> trapezoid, or even square -> circle)
- snapping to paths, so you can e.g. straighten out rivers on a map
- cool thing: you can make a chain of interactive infographics that all depend on the previous one.

Visualizing Web Browsing History with Barcode Chart, Ningxia Zhang (poster)
- looking at browsing to see how often you switch, maybe.

StickEar: Making Everyday Objects Respond to Sound, Kian Peen Yeo
- little sticky gadgets that each sense sounds and can be configured to do things

uTrack: 3D Input Using Two Magnetic Sensors, Ke-Yu Chen
- you wear two magnetometers on your ring finger and one magnet on your thumb, and then you can do free-space 3D gestures. Neat!

FingerPad: Private and Subtle Interaction Using Fingertips, Liwei Chan
- similarly, you wear a sensor grid on your fingernail and a magnet on your thumbnail, and now you can draw on your fingertip. The privacy of it is kind of cool.

BitWear: A Platform for Small, Connected, Interactive Devices, Kent Lyons (poster)
- little fingernail-sized buttons with Bluetooth and LEDs that you can configure on the internet.
- I would really like to play with these.

Imaginary Reality Gaming: Ball Games without a Ball, Patrick Baudisch
- you can play basketball without the ball. QR-ish codes on people's heads let an overhead camera know who is where, and a speaker says who has the ball.
- I mean, that's cool in itself.
- even cooler: imagine real world games with power-ups!

inFORM: Dynamic Physical Affordances and Constraints through Shape and Object Actuation, Sean Follmer, Daniel Leithinger
- tangible table, a lot of little square things that can move up and down.
- you can make really non-"computery" controls like the ball answering machine. Or it can move things around on the table. It looks like it has a personality.

Traxion: A Tactile Interaction Device with Virtual Force Sensation, Jun Rekimoto
- little metal thing with a moving magnet, feels like it's pulling left or right. The demo was a big hit. Pretty weird that it can fool your mind like that.

There were a few cool mixed-initiative things there too. Cobi: A Community-Informed Conference Scheduling Tool, AttribIt: Content Creation with Semantic Attributes, SeeSS: Seeing What I Broke - Visualizing Change Impact of Cascading Style Sheets (CSS), A Mixed-Initiative Tool for Designing Level Progressions in Games, A Colorful Approach to Text Processing by Example. I might be making up a theme out of nothing (I guess the part-computer part-human system thing is just all of HCI) but there's something that feels pleasingly interactive out of these.

Something about addiction and flow

So flow is good, right?

Well, sometimes, yes, unless you're addicted to slot machines. Or online games. Or Facebook, or e-commerce sites, or really anything else unproductive.

So if you were, say, starting out on some research that could probably be pretty well defined by "make your computing experience more flowy"... how can you motivate that better so it's actually about making your life better, and not just faster? Google gets away with worshipping speed and fluidity because faster page loads means more internet use, which means more money for them, but the rest of us are not optimizing for money.

To Save Everything, Click Here

by Evgeny Morozov. This was quite a read. Not often I painfully jot down 11 pages of notes in my ebook reader. I guess it's because he has strong opinions about most everything I'm interested in.

Quick overview: railing against "internet-centrism" and "solutionism." Internet-centrism is the modern trend to ascribe magic powers or moral judgments to "The Internet." (Piracy should be tolerated (instead of DRM) because The Internet! We should change government to be more like Wikipedia because The Internet!) Solutionism, a broader topic, is the tendency to see any phenomenon as a problem to be solved.

Caveats: he's kind of a jerk. I'll call him out on it explicitly later. It's really frustrating; I want to like this book more, but often he just resorts to bullying. And his caustic writing style is probably calculated just so people like me will get all fired up about his book! In that case, sir, good job. (I hope I never meet you.)

Anyway, solutionism. A good example of something that's maybe not a problem to be solved is cooking. He makes fun of all these kitchen technologies that help you cook "more perfectly" - which is to say, more accurately or more efficiently, silently assuming that accuracy or efficiency is what we want. "Here is modernity in a nutshell: we are left with possibly better food but without the joy of cooking."

Internet-centrism: people complain about Apple creating a closed ecosystem. But just why is open better? In Apple's case, for selling you some sweet apps, maybe closed is a better model. (Morozov's thoughts, not mine.) Arguments for openness often (not always) resort to "because openness! The Internet!"

Other things that Morozov says that I agree with him on, or am open to considering:

- Silicon Valley libertarianism is bad: hell, we wouldn't even have The Internet without public financing.

- "Open government" isn't a goal - or at least, that openness isn't a goal in itself. For a few reasons: 1. it's a huge pain to have to document every damn thing you do; 2. it opens the doors to the public to nitpick every last little spending decision (some of which are unintuitive but turn out well). Efficiency is great, but not necessarily our #1 goal in our government.

- Technological solutions often have unintended consequences. If you publish crime stats by neighborhood, sure, that helps home buyers/renters to find safe neighborhoods, but it also hurts sellers and therefore might make them less likely to report crimes.

- You can't rely on Yelp-style crowds for everything. (hell, you can't even rely on Yelp-style crowds for Yelp.) I don't want crowds telling me where to eat, much less what to vote.

- If you publish some metrics (like senator attendance records) then people will optimize for them. (this can be problematic. maybe one senator has more important things to do one day.)

- We shouldn't take Google search results as gospel; they can be manipulated. (Interesting question then: given that we do take them as gospel, what should we do about it?)

- "Internet-centrism is at its most destructive when it recasts genuine concerns about the mismatch between what new digital tools and solutions have to offer and the problems they are trying to solve as yet more instances of Luddite and conservative resistance."

- Complicated computer algorithms, like any other decision making tool, reflect the biases of their creators. But complicated algorithms offer a (sometimes real, sometimes fake) appearance of objectivity. (for example, for police work.) (sounds like a call for intelligibility.)

- Oh man, great stuff about SCP, the law enforcement philosophy that says you should make it impossible to commit crimes, rather than just punishing people who do. This is really interesting. For example, if I decide I'm not going to eat cookies, I want SCP-style prevention there! I want to make it impossible for me to eat cookies! He agrees: as long as you make the decision yourself, there's no problem "shifting registers" (term from Roger Brownsword) from the "moral" register (x is good or bad) to the "practicable" register (x is easy or hard) or the "prudential" register (x helps me or hurts me).  So he's against "nudges". I can dig it. When we shift things out of the moral register, though, we might never even think about them again.

- Excessive quantification in the research world is a mess. People are gaming metrics, counting publications and citations too much, etc.

- Food is a good example of solutionism gone wrong. We decide fat is bad, so everyone counts grams and does all sorts of nasty tricks to call their products "low fat", only to later discover that fat's not so bad after all.

- Furthermore, Quantified-Self solutions can backfire by putting the onus back on the individual, rather tahn the broken system. ("why didn't you just count your calories?")

- Memory != preservation. And we shouldn't assume that we should preserve everything.

- We talk about "information nutrition" (e.g. The Economist is healthy, tabloids are junk food), but we really have no idea what we're talking about.

- You can't really design for serendipity.

- Gamification via points and badges is dumb. (okay, duh.)

- Check out Albert Hirschman's futility-perversity-jeopardy trio as a set of common reactions to new things.

- We should try to change people's behavior mostly by reflection, not by paternalistically "nudging" them into making the right decisions. I don't know, though; his example, using some complicated parking meter thing, seems like those 5 cent nickels at Whole Foods when you bring your own bags, and then you have to decide where to donate your 5 cent credit. I don't want to think about those 5 cents. Just let me get on with my day.

- Sure, everyone's always manipulating us. But I want to know when it's happening.

Anyway, the overall feeling I got from reading this is that maybe my earlier research thoughts are misguided. I basically heard the Weiser ubiquitous computing story ("your computers will fade into the background" etc) and thought "yes, let's do that." Maybe perfect efficiency isn't always the goal!

Some things that Morozov says that are stupid: (if he's going to bully people, I can bully him back)

- somehow make "open government" data (say, campaign finance) only appear on the original source (like the FEC website) so people can't re-publish it and selectively alter or highlight it.

- the "Pirate party" in Germany is losing support, therefore their ideas are failing and should be mocked.

- LiquidFeedback, this tool that sounds like Google Moderator, is a "solution to a problem that doesn't exist" - we don't need more feedback from, say, politicians. 

- partisanship isn't necessarily a problem

- Amazon might start automatically generating books! That are creepily optimized to be exactly what you like! Never mind that this is probably AI-complete!

- We should start having all algorithms be audited by qualified third parties. (oh my god. what constitutes an algorithm? geez.)

- "Once everyone is wearing Google's magic glasses, the costs of subjecting friends to a mini lie detector... are trivial."

- Argh, he totally doesn't understand some of the systems he's writing about and mocking. (e.g. Matthew Kay's Lullaby)

- Quantified Self people are weird and gross. (seriously, this whole chapter is just straight up bullying. it's really offensive.)

- Quantified Self people are all into some weird woo-woo shit about revealing the deeper truth of who we are, with numbers! What a bunch of misguided weirdos!

- Self tracking for health purposes makes a mess for the insurance industry, so we shouldn't track things about ourselves.

- If you like quantifying something, then you must be a Super Quantifier who wants to quantify everything!

- "Even though Bell doesn't quite put it this way..." (... puts words into his mouth.)

- Gordon Bell is a weird guy. I can just dismiss anything he does by mocking him.

Some comical excerpts from my notes as I was reading this:

"no, you numbskull."

"sigh"

"this section sounds fearmongery"

"the bullying in this section makes me wonder about the rest of this book."

"not what he said. you clown."

(on chapter title "Gamify or die") "I hate this chapter already"

(when he starts talking about extrinsic vs. intrinsic motivation) "sigh I knew this was coming"

"ad hominem, you're a jerk, etc"

My "wearable computer" set up, June 2013

I don't (yet) have a Google Glass, but I wear computers, and it's great. Currently I have:
- Android phone (Nexus 4, if you must know)
- Pebble wristwatch
- LG Tone headphones
- Fitbit Ultra (no longer made; replaced by the Fitbit One)

The Fitbit counts my steps, and that's all. The neat thing about that is that I can compare days.
I know what it's like when my friend tells me she had an 8000-step day.
I know dancing for a couple hours is usually a good 10k steps at least.
I know that my 30k-step day in Dublin was seriously a lot of walking.
I haven't figured out why it's useful to know these things. Just cool.

The Android/Pebble combo gives me texts on my wrist. This is cool. I've received, understood, and dismissed a message on a bike. Yes, you can do this safely. I've received, understood, and dismissed a message in mid sentence. Yes, you can do this politely. (trust me; your conversational partner will mind much less than if you get your phone out and fiddle with it.)

Is it good that you can do these things? Good question.

But the Android/Pebble/Tone combo! This is the coolest part. Connect everything and start my music app, and then I can start/stop/forward/back the music from either my Pebble or the main Tone hardware. AND I can see the title of the currently playing song on my Pebble. I can almost-fully control my music on my bike. (yes, I can do this safely.)

The fact that it used to take ~25 seconds to start listening to music (unwind headphones, plug in, etc), and it now takes 3, means I listen to a lot more music. The fact that I can see the song playing means I remember it a lot better.

Is it good that I am listening to more music? Good question. But as a Real Actual DJ, who's constantly trying to take in new music, I appreciate it.

What do I take from this? Not a lot, because it's just me. Nevertheless:
- reducing the time it takes to do something really does make me do it more.
- I have no idea if there are negative effects from, say, the fact that I rarely walk more than 5 minutes in silence anymore. I have no idea how to measure this.
- the form factor has to be good and sort of invisible, but it doesn't have to be all that invisible, if there's clear benefit. (the Tone is a new one; it's "around-the-neck" style, and noticeable but usually not problematic.)
- we have enough ways to control music.

Voice of the Coffeepot

Well, this was kind of fun:

A class project. Used a force sensor to tell when the carafe was present/empty/full/being pressed, and either thanked you for refilling the coffee or wished you to enjoy your coffee via a text-to-speech chip and speakers. Enjoy your weekend!

Visualizations, video/audio, and ML for time series data: which platform?

I want interactive visualizations of a bunch of time series, as well as some audio and video, scrub through them and keep them all synced, and then generate features from them and feed those into a machine learning model. No need to be a web app.

What environment do we do this in? Cross-platform choices seem to be javascript/browser, python/installed, and java/installed.

Visualization:
Javascript: d3 (gallery), cubism, Google Charts, some others like rickshaw, even processing.js. This is maybe the best reason to pick JS.
Python: matplotlib - mostly static visualizations (gallery) or Bokeh? (see this post) Bokeh outputs to an html5 canvas in the future (or a Chaco plot currently).
Java: ...?

Playing audio/video:
Javascript in a browser: HTML5 video and audio?
Python/Java: beats me. Codec hell?

Machine learning:
Javascript: ...?
Python: scikit-learn (and others)
Java: Weka (I hear the API's a pain, though.)

The path forward seems to be to start building an HTML/JS app, even if it's only client side, and figure something out for the machine learning. Perhaps compile scikit-learn to JS with pyjs? Perhaps (this sounds kind of painful) just send all the features to a server and use weka or scikit-learn there to do the real ML and send back results? But I'd welcome any input.

It's about how you use it

"I'm still here: back online after a year without the internet"- in which he leaves the internet, and all's well, until it's not anymore; all his bad habits catch up with him offline too. Also, you can't just quit the internet alone. It's part of all our life together. 

Don't be a gargoyle with Google Glass. Don't be with people but not actually with them. Don't tolerate other people being gargoyles.

Is Glass an anti-distraction tool? Could be. So long as you can ruthlessly and efficiently curate what goes into it (or if the algorithms it uses are good at doing that for you). Given people's abilities with their email inboxes, I'm not sure, but I think this writer's on to something: if people have a low cost way to get into your visual field, that'll be a problem.

So what are we left with? "The Internet, or Google Glass, or whatever technology you want, isn't good or bad in itself; it depends on how people use it. (which depends partially on design decisions of the people making it.)" Xkcd is right on.

Three kinds of multitasking

I've got to start posting one thought at a time. Shorter posts more often.

Let's get our words straight: here's multitasking vs. switch-tasking; multitasking is doing a lot of related things, switch-tasking is doing a lot of unrelated things.
Maybe this points to the polychronicity puzzle: how is it that some people can prefer to do many different things, even though multitasking doesn't work? On the other hand, examples of polychronicity sound like switch-tasking, so that actually doesn't help answer that question.
A third term: continuous partial attention, or paying attention to many things at a surface level.

I guess it's like this?

Thinking about time and speed

Half formed ideas and links here.

Been reading a lot of blogs and listening to talks, going to a workshop, thinking about how we view time and how fast it goes. In Motion has a bunch of thoughts about time, in relation to travel; while a lot of the book is frustrating (what is "Deep Travel"? is it just "traveling while paying attention"? why does it need a name?), it makes me think about time as experienced vs. time on the clock.

Monochronic vs. polychronic time - first pointed out to me in two papers by Gloria Mark and Laura Dabbish. Monochronic is what we're used to: one thing at a time, time matters, be on time. Polychronic time sense is the approximate kind of time; do a couple things at once, value relationships more than the time. (but multitasking doesn't work! what's going on here?)

Contemplative Computing points me to a few things: an article about speed mentioning Google Glass cutting down the picture taking time from 12 seconds to 1 second, an interview with Linda Stone on time management vs. attention management, a group interview where Neema Moraveji talks of (among other things) "speed bumps" put into typical tasks. We're putting speed bumps into one task and taking them out of others (and indeed, as texts-on-the-Pebble has shown, removing speed bumps is surprisingly awesome). The answer is probably not "speed is good" or "speed bumps are good", but something more subtle. What is it?

Another thing I want to jot down here: calendars are to time as maps are to space. Ponder that.

CHI 2013

... was, of course, great. I student-volunteered, which meant I didn't get to see a ton of talks; nevertheless, here are a few things I particularly liked:

Stories of the Smartphone in Everyday Discourse: Conflict, Tension & Instability, by Ellie Harmon and Melissa Mazmanian. They looked at the stories people tell themselves about their mobile phone use; it oscillates between the "integration" story ("get a smart phone, be a super connected techno future hero") and the "dis-integration" story ("unplug, de-stress, get back to the real world"). This causes tension and makes people uncomfortable. Interestingly, these are kind of the only stories being told, and they're overly simplistic.

Indoor weather stations: investigating a ludic approach to environmental HCI through batch prototyping, by Bill Gaver et al. They put little devices in people's homes that would playfully reflect indoor environment conditions (like slight wind, etc); people didn't find them practical or particularly fun, but still felt some attraction to them; "it's like there's a ghost in the house."

Slow Design for Meaningful Interactions, by Barbara Grosse-Hering et al. They described principles they used to design a juicer. Most interesting: it's okay to make some things slower. It can be good, in fact, as long as they're the key parts of the process. People don't mind spending a little more time juicing; they mind spending a long time cleaning the thing afterwards.

Some cool gadgets you can wear: WatchIt (watch band interaction) and NailDisplay (goes on your fingernail; I wasn't sold at first, but then it grew on me throughout the presentation).

Some cool info-vis-related stuff: The Challenges of Specifying Intervals and Absences in Temporal Queries. Interactive Horizon Graphs. Alternate Glyphs (color and spirals). And, fNIRS for evaluation!

Finally, and most entertainingly, don't use seven segment displays. See you all next year in Toronto!

Canabalt 8x8

You might think I spend all my time researching Really Hard Problems in Human-Computer Interaction. This is not entirely true. Sometimes I take classes, and sometimes classes have projects, and sometimes I get to make fun things like this:

(real Canabalt, which is a lot more fun and I cannot take any credit for making)
(more details on github)

Hypothesis Generation Systems

Imagine you're tracking something about your health. Say it's your weight. You have weight measurements from every day and you want to generate some hypotheses to test, like "my weight this month is significantly higher than last month." Or say you're tracking how many steps you take each day, and want to know some things about how that correlates with when you woke up, or any number of other data streams you've got.

I'm picturing some kind of visual analysis tool. Something that would help with hypothesis generation and feature selection. What's in this space? I started with the 150 most popular tools on the Quantified Self guide, as well as all 204 tools on Ian Li's personalinformatics.org. Some things I found:

Systems that pull in data from a lot of places and generate visualizations
Notch pulls in data streams from (currently) Fitbit, Runkeeper, and Bodymedia. Then it generates some pretty basic (although definitely pretty) visualizations.
Cosm (formerly Pachube) lets you look at a bunch of graphs together (one example here). Doesn't do any statistics, though, AFAICT. Focuses on Arduino/Internet-of-Things applications.
Trackify seems nonexistent.
Fluxtream seems promising, but I don't have access yet.
Also The Locker Project (Memo to myself: they seem to be very connected with Singly, which looks like a way to easily integrate apps with lots of social and/or quantified-self applications), TheCarrot, DataDepot
These all would be much cooler if they did stats (although just visualization is useful too!)

Tools that let you track things yourself by hand (and then they generate graphs)
ChartMyself (meant for QS-type use), TallyZoo (for anything), Mycrocosm, Daytum (anything), MercuryApp (mood-ish things), Dayta, Tonic, DidThis (things you've done), rTracker, Graphomatic, DailyDiary, LifeMee (maybe? signup is broken), Limits, DailyTracker, Grafitter, your.flowingdata, lifemetric (for many people)
These all seem like not what I'm looking for. The data entry is manual, and the output is still mostly visualization.

There's Quantified Mind, which apparently shows you all the significant results in your mental-test data automatically. Only for mental-test data, but still, this is the kind of thing I'm looking at.

There's media annotation and editing tools as jumping-off points, if you wanted to build this. Elan is one, but it's based on the quirky Java Media Framework; Pitivi is another, but for Linux only.

Then there's the whole field of information visualization, big-data type stuff (recent startup Trifacta comes to mind), which I'm just starting to dig in to.

More to come! There's a conference called IEEE VIS (formerly VAST) that might have some interesting stuff, I'll trawl through that. Also, any ideas you have would be quite welcome.

New Note. New Note. *sigh* New Note.

I'm interested in what you can do with audio if you're wearing a headset/microphone. As a baseline, I want to see what you can do with off the shelf components already with minimal frustration. The test task is making a note without taking the phone out of my pocket.

(tl;dr: this does not work at all. I am surprised at how much this doesn't work. Geez, I figured voice commands were solved.)

Headsets:
I've tried the Plantronics M50 and the LG Tone. The M50 is a standard bluetooth headset, the Tone is worn around the neck and has detachable magnetized earbuds that you can put in your ears. First impressions:
- I feel like a businessman and a tool wearing the M50. With the Tone I feel like a regular person. This is important.
- Music streaming to the ears works right out of the box on both. This is surprisingly great, especially when biking.
- Sound quality in my ears seems fine on both, although I've only used them a couple days each. The M50 is a little bit quiet even on max volume for listening to music, but okay for calls. Can't say much about voice recording quality.

Apps:
- the Google app that includes Voice Actions and Google Now (on Jelly Bean, I think) is so cool and so flawed and buggy. The cool: saying "note ____" records your voice and sends it to yourself in an email: both the audio file and an attempted transcription. No confirmations or anything. Exactly correct. However, when I associate the "bluetooth button" action with the Google Search app and press the button, nothing happens. And when I open the app on my own, sometimes it jumps right into recording, and sometimes it, says "initializing" forever and freezes. This is on my 2-year-old Nexus S, and the Galaxy S3 I'm working with doesn't have Jelly Bean. I'm guessing when they get the bugs worked out, this will be the reasonable way to talk to my phone.

- Voice Control (Full) works pretty well. Pressing the button on my bluetooth opens it and starts listening (IF you disable the Google app and then re-associate the "bluetooth button" action with Voice Control). I can say "Make a note" and it puts it in my Evernote account. The downside is that it's a 5-step process: "Make a note" What should the content be? "blah blah" Do you want to make a note with content 'blah blah'? "Yes" What should the title be? "title title" Do you want the title to be 'title title'? "Yes." Five steps is four steps too many (especially because all these steps can, and do, fail).

- Vlingo boasted at least somewhat-competent voice recognition but you can only do about 7 things, none of which I even want to do. (call people, text people, update your facebook status...)

- Utter looks like it's headed in the right direction, but right now doesn't accept the button on the bluetooth to trigger it.

- Samsung's S Voice: not so good. Pressing the button on my Bluetooth device starts it... sometimes... and sometimes it just opens the app so I can start it by pressing a button on the screen (which defeats the whole purpose of the bluetooth button). Also, when you're making a note, it asks you to confirm by saying "Save note"... and if it doesn't hear "save", it just hears "note", which throws away your old note and starts a new one. What!

- Skyvi ("Siri for Android") had poor voice recognition. Also, it doesn't support just taking notes. Am I the only one who never wants to call people if it's not very reliable that the service will call the right person?

- Iris ("Siri backwards") just didn't work on my Nexus S, and on the Galaxy S3, it looks like it's a press-an-on-screen-button app. (with lots of annoying advertising for some other app too.)

Finally, a couple of thoughts Absolutely Correct Ideas about how voice commands should work, based on a day of futzing with them:
- start on bluetooth button press. If I take my phone out of my pocket, you've lost me. Ideally ideally, we'd be starting on a wake-up word, but I assume the battery life isn't there quite there yet.
- don't rely on correct transcription, when possible. (taking notes shouldn't rely on correct transcription.)
- corollary: don't ask me to confirm stuff, unless it relies on correct transcription (like calling a person). I should say things once, maybe twice.

Memorizing Names

Say I was talking to you and I told you one fact (like my name) and you wanted to memorize it, but we kept conversing. How would you do it?

You should rehearse it at T+8 seconds, 14 seconds, 32 seconds, 86 seconds, and increasing intervals in a 5+3^n pattern. Here's why:

Spaced Repetition has been around a long time. The idea is that, if you're going to practice a fact N times to remember it, you should practice it over time, not all at once. (this is called the Spacing Effect.) You should be asked to reproduce the item, not simply shown it again. ("Testing Effect") Furthermore, these rehearsals should be in increasing intervals. ("Expanded Retrieval")

About the spacing effect: this has been shown repeatedly (see pretty much any link in this post where spaced practice beats massed practice)
About the testing effect: this has been shown repeatedly too, e.g. by Carpenter and DeLosh (2005).

About expanded retrieval: this is a little less clear. Pimsleur (1967) suggested exponentially increasing intervals. Landauer and Bjork (1978) found that increasing intervals (e.g. rehearse in 1-5-9 seconds) is better than equally-spaced intervals (like 5-5-5) if you're testing yourself, but neither Carpenter and DeLosh (linked above) nor Balota et al (2007) found much support for the "increasing intervals is best" argument. Indeed, Karpicke and Roediger (2007) found that increasing intervals helped short-term recall, but equally-spaced intervals helped long-term recall. But they found that this effect may be due to the equally-spaced intervals' lack of an immediate recall (the "1" in a 1-5-9 schedule). They showed that just delaying the first test by 5 seconds makes it harder, which helps long-term recall. So it seems like you should be able to get the best of all worlds by adopting an increasing-intervals schedule, but also delaying the first review.

Another consideration is that this is the real world, not a 3-repetition study in the lab, and increasing intervals scales better. If you start practicing every 5 seconds, by the time you're at repetition 10 you'll be fed up, whereas if you go with 3-9-27-81 etc the intervals will quickly become so infrequent that you're not bothered.

But what should the first interval be? Peterson and Peterson (1959) show that recall percentage at 3 seconds is better than 6 seconds, 6 better than 9, etc., and we want them to remember it at the first interval, so might as well make the first interval 3 seconds. But, as Karpicke and Roediger (linked above) mentioned, we don't want it to be too easy to remember it at the first interval. Well, the same Peterson et al (1963) found 8 seconds to be the best interval.

So, how about a 3-9-27-81 interval plus 5 second delay, so 8-14-32-86-etc? Whew! Well, whatever; increasing + delayed-first-item sounds like at least a pretty good way to go.

Hmm... but if you're not in a lab, talking to an experimenter, how will you remember to test yourself at all these intervals? Hey, what if we could do that with a system using instant, unconscious, subtle microinteractions...
Stay tuned!

Microinteractions: the book

Since I came across Daniel Ashbrook's thesis, I've been thinking about "microinteractions": "interactions with a device that take less than four seconds to initiate and complete." I'm interested in expanding the space of possible microinteractions that people use.

Now this fellow Dan Saffer is writing an O'Reilly book with the title Microinteractions. You can currently read a draft of the first chapter. Sounds like he's using a more general definition of the term, to include lots of non-mobile interactions: the "see translation" button on Facebook, the password entry form on Twitter, calendar apps including the duration and end time when you're scheduling a thing. I like it. It leaves me wondering: is this just "everything" now? Is "microinteractions" a synonym for "details"; something that of course we should focus on but nobody's going to have some big revolutionary ideas about? Or is this part of a big shift in thinking, now that we've got enough computing resources to actually make meaningful and positive microinteractions?

Incidentally, your potential microinteraction of the day: squeeze your phone as you pull it out of your pocket depending on how you want to use it.

UIST 2012 highlights

... in my humble opinion, based on my particular interests:

Tactile/hands/fingers:

Facet: a multi-segment wrist worn system, Kent Lyons (Nokia)

Watches are old news. How about having 6 watch screens in a bracelet shape around your wrist? It's clunky now, but who knows. They can detect pose and interact smoothly as needed.

iRing: Intelligent Ring Using Infrared Reflection, Masa Ogata (Keio U)

A ring for input with 4 sensors. Clever: recognizes IR reflection and skin deformation to tell whether you're clenching, bending, pushing, flipping, or rotating the ring. Detects position/rotation by melanin content (which varies around your finger. Wired currently (and a ring is so small it makes me wonder if the wire could be removed).

Magic Finger: Always-Available Input Through Finger Instrumentation, Xing-Dong Yang, Autodesk

Camera/LED on your finger for always-available input. The camera is 1x1mm, 148x148 pixels. ("NanEye") Read "fake textures"; ascii characters printed into patterns. Downsides: wired and ~1s latency on touches. Still, cool!

Digits: Freehand 3D Interactions Anywhere Using a Wrist-Worn Gloveless Sensor, David Kim, MSR UK

IR laser line and camera mounted on your wrist to detect your finger positions. Allows finger gesture prediction, 3d manipulation, etc. It's a bulky box now, but you could imagine it shrinking. Between 2-9 degrees of error, which is good enough for a lot of tasks.

Dynamic Tactile Guidance for Visual Search Tasks, Ville Lehtinen, Helsinki IIT

Wear gloves so when you're looking at a wall of stuff you can find the exact thing. Sounds like it could be useful. (the trick is finding a task where the computer knows where the right thing is, but you still have to find it with your hands.)

Pressages: Augmenting Phone Calls with Non-Verbal Messages, Eve Hoggan, U Helsinki

An addition to phone calls. You can squeeze the phone, and then it vibrates on the receiver's ear. Four intensity levels, from light to sharp. This might sound a little silly, but:

- they tried it with 3 couples for a month, and they all sent at least one Pressage every single call.

- they wanted to have it available other times too, and as another channel of communication (e.g. light buzz = "I'm coming home")

Assuming they didn't just get 6 quirky people in their study, people will use this. It's super intuitive and quick, and adds a layer of richness to phone conversations. (the channel during non-phone conversation is mostly a nice bonus, and is kind of tricky for a lot of reasons.)

Other Things:

Steerable augmented reality with the beamatron, Andy Wilson, MSR

Pan-tilt projector and kinect so things can project and interact all around the office. Quite a feat of engineering.

Designing for Low-Latency Direct-Touch Input, Albert Ng, MS

Our current displays have around 100ms latency on touch. You can see this yourself: draw a quick line with your finger; it lags an inch behind you. What if instead we had 1ms latency? I tried the demo, and it is much slicker. Feels like you're moving real objects. Remember when Google started targeting latency and all of a sudden Gmail became a viable non-painful application? Latency matters on tablets too.

Histomages: Fully Synchronized Views for Image Editing, Fanny Chevalier, Toronto

You know image histograms? What if you could select pixels by brightness or by blueness instead of by location, and edit them all based on that? Looks fun.

ConstraintJS: Programming Interactive Behaviors for the Web by Integrating Constraints and States, Stephen Oney, CMU

Ever made an iOS app with Interface Builder? You specify constraints (like "this text box aligns with the center of this image") and they are automatically maintained through resizes etc. ConstraintJS looks like a way you could do this on the web, and for more than UI layouts. You can make asynchronous calls and display all their states without the pain! http://cjs.from.so

DejaVu: Integrated Support for Developing Interactive Camera-Based Programs, Jun Keto, MSR Asia

An IDE for developing camera- (i.e. Kinect-) based applications. If I made a camera app, I think I would want this.

Low-cost Audience Polling Using Computer Vision, Andrew Cross, MSR India

Instead of expensive "clickers" which you might have used in university classes, just print everyone a QR-like card and have them hold it up. Cheap webcam takes a picture of the whole class.

Clui: A Platform for Handles to Rich Objects, Hubert Pham, MIT

What if the default unit on the web were a JSON object instead of a hyperlink? Sounds like web standards stuff, where it's awesome as much as everyone uses it, and good luck, but would be really useful in a ton of ways.

CrowdScape: interactively visualizing user behavior and output, Jeff Rzeszotarski, CMU

MTurk is great, but people cheat. Some tasks you can design around this, but some you can't. Crowdscape lets you visualize what people are doing as they do your task, easily weed out cheaters, use that as labeled input to bootstrap a machine learning system, and more importantly understand what work patterns lead to a good response. (Maybe tooting my school's horn a bit, but: won a best paper award!)

Some folks made a braille tutor out of the pressure-enabled touchpad that we got for the Student Innovation Contest. Awesome. They won 2nd place; I'd have given them first. Our ambient stress sensor was neat but did not win. Nor did it deserve to; there was a lot of great stuff.

Cool posters:

MMG armband, Shumpei Yamakura (like EMG, but resilient to sweat; not sure how they'd compare)

MISO, David Fleer, Bielefeld; point and snap at your electronics

Tongue-finding with Kinect (i.e. for rehabilitation), Masato Miyauchi

Breathwear (band to detect when you're breathing), Kanit Wongsuphasawat

Cool coffeeshops in Cambridge: Crema in Harvard Square for cappuccino and Voltage by Kendall for a fine Guatemala Buena Esperanza roasted by Barismo.

Yes! Another good conference. People asked me multiple times "So how's UIST going?" Look, of course it's fun and full of cool people doing exciting stuff!

Thinking about unconscious/micro interactions

Trying to define a research plan or story or something that I can both work on and apply for fellowships about. Right now a lot of work that I'm interested in feels related to me, but it's hard to explain to other people, which means it's not well-defined enough. In this post, I'm working on that.

All our interactions with computers/smartphones now are both intentional and slow. By "intentional" I mean that you have to think about getting out your computer (or phone), and by "slow" I mean at least on the order of seconds, if not minutes. Right now, get your phone out and check the weather forecast, and count seconds while you do it. (Just tried it; 23 seconds.) I want to break both of those constraints.

Why?

When you remove "intentional", you move from the slow brain to the fast brain. You get North Paw, Lumoback; systems that train you on a physical level. Ambient systems which help you change behavior: DriveGain, Breakaway, UbiFit (some slides). The Reminder Bracelet: ambient notifications. I guess it feels like, and I'm not sure how best to put this, when you do things unconsciously/unintentionally, you can learn procedural things or adapt physical movement without increased cognitive load. "Human attention is the scarce resource", and these systems give you something for free.

When you remove "slow", a lot more things become possible.

Thad Starner mentions the "two-second rule" in wearable interaction (IEEE article): people optimize their systems so they can start taking notes within two seconds of realizing the need. Daniel Ashbrook, in his thesis, defined microinteractions as interactions under four seconds, start to finish. At Google, speed was a big emphasis, and they're right: if something takes longer, people will use it less. (wish I had a good citation for this.)

Interactions are also overt; everyone can tell when you're computing. Breaking that constraint lets you interact with your computer without people knowing, which seems useful. Enrico Costanza has worked on "intimate interfaces" (EMG armband, eye-q glasses). ("intimate interfaces" is overloaded to also mean "interfaces that allow intimacy", e.g. among remote couples or family; not talking about that here.) Is this good? Detractors might argue that if there are social cues against something, they're there for a reason. Nobody wants you to be computing when you're trying to talk with them. However, two things: first, it's a tool just like anything else and should be used wisely; second, people already do these things with their phones. They get buzzed, answer texts, silence their phones, etc. But I'm not sure that I want to get rid of "overt", or at least not necessarily.

How?

Watches:

Mounting things on the wrist can cut down the action time up to 78%. You can use round touchscreen watches. Conveniently, the Pebble watch is now somewhere in production stages, and the Metawatch is... shipping? InPulse has been around for a couple years, but is a little clunky and doesn't have the battery life.

PinchWatch might not be what you think of as a watch (besides the display); a lot of the interaction is done by pressing fingers together. 

Nenya is a ring/watch system, getting really minimal. I like it. Reminds me of Abracadabra, due to the magnetic sensing, but now the ring just a regular-looking ring you would wear anytime.

Pockets:

You can touch your phone in your pocket, even do a Palm-Graffiti-style input (PocketTouch). More simply, for some tasks you could just hit your phone (Whack Gestures). Sami Ronkainen et al investigated this first, though they hit more false positives. (they also found taps to be more natural/accepted than gestures.)

Speech:

This seems obvious, right? But it's not. First, talking while walking around is weird. (ever play Bluetooth Or Crazy?) Second, it's not easy to get audio input into your phone in your pocket unless you're wearing a Bluetooth or something.

Sunil Vemuri et al's "Memory Prosthesis" was one approach, focusing on recording nearly-continuously and then searching; the search is less interesting to me, but the continuous recording is useful. Ben Wong experimented with dual-purpose speech: the user wouldn't give direct commands to a system; rather, the system would harvest information from things the user was saying. The Personal Audio Loop recorded your last 15 minutes and let you go back to search within it.

Quantified Self 2012: some cool things

Quantified Self is a movement of researchers, business folks, and hobbyists who are interested in understanding themselves more deeply through data. Usually we track some data, either numeric (number of steps I took each day, number of hours I sleep) or more abstract (dream journal, photos taken every 5 minutes by a camera around my neck). Usually the meetings are local; a couple dozen people get together and share whatever projects they're working on or questions they're interested in. Then there's an annual conference; this was the second one.

Stuff's less polished than at an academic or business-focused conference, so the things I took from it are a little more abstract than a list of papers. Here's some good stuff:

Instant Feedback Gadgets

Nancy Dougherty demoed an EMG smile sensor attached to a string of blinky LEDs. When she smiled, the lights blinked. She mentioned she'd post instructions on her blog at theengineeress.com soon.

Lumoback is a posture sensor and feedback device. It's a comfortable band you wear around your waist that buzzes you when you slouch. This is the sort of thing I love, because it feels like you'd start to get a visceral sense of when you're slouching and automatically correct it. After a while, you wouldn't have to think about it at all, your posture would just be better. www.lumoback.com

In the same vein, I chatted with Eric Boyd, inventor of some neat biofeedback devices like the NorthPaw, which buzzes north until you eventually get a sense of where north is. What I didn't know before is that he's selling them. sensebridge.net

Butterfleye (butterfleyeproject.com) is a pulse meter for swimmers. I love the inventor's goals: frictionless and glance-able. 

Other Tools That Work

Quantified Mind (quantified-mind.com) is a platform for testing mental functions. Nick Winter talked about his experiments trying 11 different interventions to improve his cognitive skills. (creatine and piracetam+choline worked well. butter actually made him much worse. interesting, given the QS community's interest in butter as a mental enhancement.) Yoni Donner gave a talk about the platform and their goals. They've got tests for processing speed, executive function, attention, inhibition, context switching, working memory, learning, motor skill, and visual processing. I love this; the idea that there's a battery of tests out there that we can take anytime that might actually repeatably measure cognitive skills is exciting. The downside is that it's hard to convince people (even me) to take tests for 10 minutes. They may be working on something about this, but even if not, it's super cool stuff.

Project Life Slice is a short script that takes a screenshot and a photo of you every hour. So simple, but smart: gives you a sense of when you're working and what you're doing. wanderingstan.com/lifeslice

Other Neat Ideas

Matthew Keener talked about different brain areas that help make up our concept of the "self". Sure it's a simplification (we're dealing with brains after all) but identifying about 9 areas that really matter (and how they matter) is very interesting to me. So what can we do with this? (Besides FMRI tests?)

Kevin Kelly (www.kk.org) became the first person I've ever met to have reported actually trying the Uberman sleep schedule (20 minute nap every 4 hours, no sleep at night) successfully. He said he did it for two months but eventually gave it up because you really couldn't miss even one nap or you'd crash hard.

Larry Smarr reported on a long series of self-tracking to understand health problems. I've heard of omega-3 to omega-6 ratio, but he also pointed out Complex Reactive Protein (CRP) as an inflammation marker.

Robin Barooah talked about his relationship with coffee; stopping coffee made him more productive (though he felt less productive), but starting it again helped his mood. Indeed, coffee can ward off depression. What struck me about his talk is how this data not only gave him things to act on, but it helped him reflect on portions of his life and meant a lot to him. (oh, he's made a cool meditation tracking iPhone app too.)

Ubicomp 2012: some cool things

Ubicomp 2012 just ended, right here in Pittsburgh. I was a student volunteer and had a great time. As I guess is the norm at conferences, I'm a bit overwhelmed by all the information coming at me at once, so here's an attempt to sift through it a little bit by summarizing some talks/papers/posters that I liked. I'm in Week 3 of my PhD program, so this is not going to be super focused.

An Ultra-Low-Power Human Body Motion Sensor Using Static Electric Field Sensing by Gabe Cohn et al. Currently to track motion, we use accelerometers. Their device lets you wear one sensor on your wrist, it can detect when any part of your body moves, and the power usage is about 1-10% of an accelerometer's.

A Spark Of Activity: Exploring Informative Art As Visualization For Physical Activity by Chloe Fan, Jodi Forlizzi, and Anind Dey. So your Fitbit (pedometer) counts steps, right, but it only gives the data back to you in graph form. That's cool, but somehow it's more fun (and indeed, motivating) if it's in a little bit poppier form. She's developed some visualizations and found that people do prefer more abstract visualizations for display/fun purposes. (Graphs are still better if you're looking for concrete numbers.)

Lullaby: A Capture and Access System for Understanding the Sleep Environment by Matt Kay et al. Put this box in your room while you sleep, it'll tell you if there are any disturbances or anything that might be hurting your sleep. You can't tell what's wrong when you're sleeping. Difficult task, and well executed. Also, there are privacy concerns when there's a camera in your bedroom! (They address this.)

RubberBand: Augmenting Teachers' Awareness of Spatially Isolated Children on Kindergarten Field Trips by Hyukjae Jang et al. A system that'll alert a teacher if kids go wandering off. Solves a real problem, and does it in a clever way: clusters the kids based on proximity, then detects if any cluster is getting too far from the other kids, not from the teachers.

Providing eco-driving feedback to corporate car drivers: what impact does a smartphone application have on their fuel efficiency? by Johannes Tulusan, Thorsten Staake, and Elgar Fleisch. They gave drivers an iPhone app to mount on their dashboard that would give real-time feedback so they can learn to drive more efficiently. Cool not so much for the gas-saving effect (3%) as for the idea that they maintain their skills even after they take the phone app away. I'd love to know how they're driving a month or a year later.

SpiroSmart: Using a Microphone to Measure Lung Function on a Mobile Phone by Eric Larson, Mayank Goel, et al. Got lung problems? Need to measure lung function? Toss your $2000 home spirometer, use a smart phone.

MoodMeter: Counting Smiles in the Wild by Mohammed Hoque, Javier Hernandez, et al. They set up cameras and big screens around MIT that would detect who's smiling and who's not. Cool way to measure happiness of different places (in a sense), interesting interactions (everyone would try to make it see them as a smile or not), nice face recognition. What's this good for? As is, just seems neat, but it makes me think of a few other ideas. What if you had reminders to smile in your house? Making a smile causes you to feel happier... how far does this effect go, and is it worth trying to do it repeatedly? Or do we get into annoying cheesy "smile!" dystopias? Also, does the number or percent of smiles actually tell you anything useful about an area?

Enhancing the "Second Hand" Retail Experience with Digital Object Memories, by Martin de Jode et al. They put RFID tags and QR codes on stuff in Oxfam (second hand) shops in the UK, so you can hear the original owner's story behind something you buy. Cool. They had 50% more sales, but they couldn't attribute that to the tags. It makes the world more magical: you could find a secret story in any nook and cranny. Imagine you buy a second-hand fridge, find a QR code in the drawer, and the original owner left a message about a big party they had where they stored beer there. Or a recipe of their favorite thing to make, or a log of when people repaired this dang thing. Also, I'd love to know long-term how this affects sales; I'd imagine it'd make people more likely to buy and sell things. Increased use of second-hand shops is good for the environment, your wallet, etc.

Making Technology Homey: Finding Sources of Satisfaction and Meaning in Home Automation by Leila Takayama et al. They interviewed people who did home automation projects, from the basic to the extreme. People who made their own stuff had more connection to it than people who bought premade solutions. Sometimes other people think they're wasting time, but it brings them some meaning. One guy had a "Canyon Cam" on his vacation home so he could see this view he loved, one guy rigged up a system to take pictures of his cat when it got scared off the counter, one guy would turning off the lights in the whole house as a subtle signal to his daughter to go to sleep. These are surprisingly cool and surprisingly meaningful, and I guess the crucial insight is that people enjoyed them the most when they were connecting with their home, not controlling it.

Augmenting Gesture Recognition with Erlang-Cox Models To Identify Neurological Disorders in Premature Babies by Mingming Fan et al. Put accelerometers on babies, tell if they're doing Cramped Synchronized General Movements, which correlate with Cerebral Palsy. Instead of watching an hour of a baby moving, doctors can just watch 10% as much video to detect CSGMs for sure. Useful in the medical field (clinical trial going on now) and uses a cool variant of hidden Markov models.

Identifying Emotions Expressed by Mobile Users through 2D Surface and 3D Motion Gestures by Celine Coutrix and Nadine Mandran. This is not about general emotion detection, but rather about intentional actions people might use while expressing certain emotions. Neat study: when triggered once a day or on demand, users would do whatever gesture expresses what they're feeling, then rate how they felt on a PAD model (pleasure, arousal, dominance). This is interesting when creating apps that take emotional state into account (e.g. "shake the phone angrily to restart if it freezes"; I don't know whether this would be good or bad, but you get the idea.)

What Next, Ubicomp? Celebrating an Intellectual Disappearing Act by Gregory Abowd. Okay, everyone was talking about this the whole conference, so I've got to mention it. His point, as I understand it, is: "Ubicomp" used to just mean anything with small computers. Now that field is so big, the Ubicomp conference can't continue to be just anything with small computers. Imagine having a "Personal Computing" conference nowadays; it's too broad. Maybe new conferences need to form for subfields or something. Discuss.

Demos, Posters, etc.

Touche by Ivan Poupyrev, Chris Harrison, Munehiko Sato. I feel like I've heard about this, but it's still cool. Make anything touch- and gesture-sensitive.

SenSprout: Inkjet-Printed Soil Moisture and Leaf Wetness Sensor, by Yoshihiro Kawahara et al. Print out some conductive ink and it, well, senses soil moisture and leaf wetness.

Design of a Context-Aware Signal Glove for Bicycle and Motorcycle Riders, by Anthony Carton. Of course I want this.

uSmell: A Gas Sensor System to Classify Odors in Natural, Uncontrolled Environments by Sen Hirano, Khai Truong, and Gillian Hayes. I've never seen a smell sensor before. This allows lots of possibilities.

Big talks:

The keynote by Steve Cousins, CEO of Willow Garage, was cool. A "state of personal robotics". Saw how finding a beer is easy but opening the fridge is hard, folding a towel is easy but finding the corners is hard, letting go of things at the right times is hard, etc.

The talk before the conference by Jun Rekimoto and colleagues was cool too. FlyingBuddy2, a drone you can control with your mind; glass that you can turn transparent or opaque; a drawer system that can tell what's in each drawer; armband that programmatically activates your fingers, a smile sensor before you can open your fridge; a really cute potted plant on wheels that drives around; a fork that makes different noises based on conductivity when you touch food to your mouth. Some of these things I'm not going to argue are super useful, but they're all cool.